2 Commits

Author	SHA1	Message	Date
Mathmagician8191	b88ae59604	Fix bug in world tokenizer (#93)	2023-06-11 11:46:54 +05:00
Mathmagician8191	82c4ac78f4	Add support for the world tokenizer (#86) ... * Add support for the world tokenizer * Move tokenizer logic to rwkv_tokenizer.py * Added test for the tokenizer	2023-06-08 16:37:18 +05:00