ed
/
rwkv.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
rwkv.cpp/rwkv
History
Mathmagician8191 82c4ac78f4
Add support for the world tokenizer (#86) 
* Add support for the world tokenizer

* Move tokenizer logic to rwkv_tokenizer.py

* Added test for the tokenizer
 		2023-06-08 16:37:18 +05:00
..
prompt punish repetitions & break if END_OF_TEXT & decouple prompts from chat script (#37) 2023-04-30 18:50:05 +05:00
20B_tokenizer.json Add text generation and chat scripts 2023-04-02 15:03:31 +04:00
chat_with_bot.py Add support for the world tokenizer (#86) 2023-06-08 16:37:18 +05:00
convert_pytorch_to_ggml.py Various improvements & upgrade ggml (#75) 2023-05-27 16:02:24 +05:00
convert_pytorch_to_ggml.test.py Various improvements & upgrade ggml (#75) 2023-05-27 16:02:24 +05:00
generate_completions.py Add support for the world tokenizer (#86) 2023-06-08 16:37:18 +05:00
measure_pexplexity.py Add support for the world tokenizer (#86) 2023-06-08 16:37:18 +05:00
merge_lora_into_ggml.py Various improvements & upgrade ggml (#75) 2023-05-27 16:02:24 +05:00
quantize.py Various improvements & upgrade ggml (#75) 2023-05-27 16:02:24 +05:00
requirements.txt Add text generation and chat scripts 2023-04-02 15:03:31 +04:00
rwkv_cpp_model.py Fix pytorch storage warnings, fixes #80 (#88) 2023-06-03 15:09:51 +05:00
rwkv_cpp_shared_library.py File parsing and memory usage optimization (#74) 2023-05-31 16:31:19 +05:00
rwkv_tokenizer.py Add support for the world tokenizer (#86) 2023-06-08 16:37:18 +05:00
rwkv_tokenizer_test.py Add support for the world tokenizer (#86) 2023-06-08 16:37:18 +05:00
rwkv_vocab_v20230424.txt Add support for the world tokenizer (#86) 2023-06-08 16:37:18 +05:00
sampling.py Add text generation and chat scripts 2023-04-02 15:03:31 +04:00