ed
/
rwkv.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

suggestions

This commit is contained in:
hypnopump 2023-04-03 08:25:54 +02:00
parent a64aaa81ec
commit 6f3fb01913
No known key found for this signature in database
GPG Key ID: 5E31BA0612BC5C17
3 changed files with 14 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
README.md
View File

@ -31,14 +31,14 @@ cmake --build . --config Release
If everything went OK, `bin\Release\rwkv.dll` file should appear.
### 2. Download an RWKV model from [Hugging Face](https://huggingface.co/BlinkDL) like [this one](https://huggingface.co/BlinkDL/rwkv-4-pile-169m/blob/main/RWKV-4b-Pile-171M-20230202-7922.pth) and convert it into `ggml` format
### 2. Download an RWKV model from [Hugging Face](https://huggingface.co/BlinkDL) like [this one](https://huggingface.co/BlinkDL/rwkv-4-pile-169m/blob/main/RWKV-4-Pile-169M-20220807-8023.pth) and convert it into `ggml` format
**Requirements**: Python 3.x with [PyTorch](https://pytorch.org/get-started/locally/).
```commandline
# Windows
python rwkv\convert_rwkv_to_ggml.py C:\RWKV-4b-Pile-171M-20230202-7922.pth C:\rwkv.cpp-171M.bin float32
python rwkv\convert_rwkv_to_ggml.py C:\RWKV-4b-Pile-169M-20220807-8023.pth C:\rwkv.cpp-169M.bin float32
# Linux/MacOS
python rwkv/convert_pytorch_to_ggml.py ~/Downloads/RWKV-4b-Pile-171M-20230202-7922.pth ~/Downloads/rwkv.cpp-171M.bin float32
python rwkv/convert_pytorch_to_ggml.py ~/Downloads/RWKV-4b-Pile-169M-20220807-8023.pth ~/Downloads/rwkv.cpp-169M.bin float32
```
#### 2.1. Optionally, quantize the model
@ -47,9 +47,9 @@ To convert the model into INT4 quantized format, run:
```commandline
# Windows
python rwkv\quantize.py C:\rwkv.cpp-171M.bin C:\rwkv.cpp-171M-Q4_1.bin 3
python rwkv\quantize.py C:\rwkv.cpp-169M.bin C:\rwkv.cpp-169M-Q4_1.bin 3
# Linux / MacOS
python rwkv/quantize.py ~/Downloads/rwkv.cpp-171M.bin ~/Downloads/rwkv.cpp-171M-Q4_1.bin 3
python rwkv/quantize.py ~/Downloads/rwkv.cpp-169M.bin ~/Downloads/rwkv.cpp-169M-Q4_1.bin 3
```
Pass `2` for `Q4_0` format (smaller size, lower quality), `3` for `Q4_1` format (larger size, higher quality).
@ -64,18 +64,18 @@ To generate some text, run:
```commandline
# Windows
python rwkv\generate_completions.py C:\rwkv.cpp-171M-Q4_1.bin
python rwkv\generate_completions.py C:\rwkv.cpp-169M-Q4_1.bin
# Linux / MacOS
python rwkv/generate_completions.py ~/Downloads/rwkv.cpp-171M-Q4_1.bin
python rwkv/generate_completions.py ~/Downloads/rwkv.cpp-169M-Q4_1.bin
```
To chat with a bot, run:
```commandline
# Windows
python rwkv\chat_with_bot.py C:\rwkv.cpp-171M-Q4_1.bin
python rwkv\chat_with_bot.py C:\rwkv.cpp-169M-Q4_1.bin
# Linux / MacOS
python rwkv/chat_with_bot.py ~/Downloads/rwkv.cpp-171M-Q4_1.bin
python rwkv/chat_with_bot.py ~/Downloads/rwkv.cpp-169M-Q4_1.bin
```
Edit [generate_completions.py](rwkv%2Fgenerate_completions.py) or [chat_with_bot.py](rwkv%2Fchat_with_bot.py) to change prompts and sampling settings.
@ -88,7 +88,7 @@ Example of using `rwkv.cpp` in your custom Python script:
import rwkv_cpp_model
import rwkv_cpp_shared_library
# change by model paths used above (quantized or full weights)
# change to model paths used above (quantized or full weights)
model_path = r'C:\rwkv.cpp-169M.bin'

4
rwkv/chat_with_bot.py
View File

@ -3,11 +3,11 @@
import os
import sys
import argparse
import pathlib
import sampling
import tokenizers
import rwkv_cpp_model
import rwkv_cpp_shared_library
from pathlib import Path
# ======================================== Script settings ========================================
@ -38,7 +38,7 @@ args = parser.parse_args()
assert prompt != '', 'Prompt must not be empty'
print('Loading 20B tokenizer')
tokenizer_path = Path(os.path.abspath(__file__)).parent / '20B_tokenizer.json'
tokenizer_path = pathlib.Path(os.path.abspath(__file__)).parent / '20B_tokenizer.json'
tokenizer = tokenizers.Tokenizer.from_file(str(tokenizer_path))
library = rwkv_cpp_shared_library.load_rwkv_shared_library()

4
rwkv/generate_completions.py
View File

@ -2,12 +2,12 @@
import argparse
import os
import pathlib
import time
import sampling
import tokenizers
import rwkv_cpp_model
import rwkv_cpp_shared_library
from pathlib import Path
# ======================================== Script settings ========================================
@ -36,7 +36,7 @@ args = parser.parse_args()
assert prompt != '', 'Prompt must not be empty'
print('Loading 20B tokenizer')
tokenizer_path = Path(os.path.abspath(__file__)).parent / '20B_tokenizer.json'
tokenizer_path = pathlib.Path(os.path.abspath(__file__)).parent / '20B_tokenizer.json'
tokenizer = tokenizers.Tokenizer.from_file(str(tokenizer_path))
library = rwkv_cpp_shared_library.load_rwkv_shared_library()