54 lines
1.3 KiB
Markdown
54 lines
1.3 KiB
Markdown
# rwkv.cpp file format
|
|
|
|
This format is used by `rwkv.cpp` to store RWKV model checkpoints.
|
|
|
|
Preferred file extension: `.bin`
|
|
|
|
Specification in C-like pseudocode:
|
|
|
|
```
|
|
RWKVModelFile {
|
|
// All ints and floats are in machine byte order.
|
|
// Magic is "ggml" string bytes.
|
|
int32 magic = 0x67676d66;
|
|
int32 version = 100;
|
|
int32 n_vocab;
|
|
int32 n_embed;
|
|
int32 n_layer;
|
|
// Data type of most of the parameters. See "Data types" below for possible values.
|
|
int32 data_type;
|
|
// Read until EOF.
|
|
Parameter[] parameters;
|
|
}
|
|
|
|
Parameter {
|
|
int32 dim_count;
|
|
int32 key_length;
|
|
// Data type of the parameter. See "Data types" below for possible values.
|
|
int32 data_type;
|
|
// Compared to PyTorch's parameter.shape, dimension order is reversed here!
|
|
int32[dim_count] shape;
|
|
// Keys are like "emb.weight", "block.0.ln1.weight".
|
|
uint8[key_length] key_utf8;
|
|
// Length of the data array depends on parameter data type:
|
|
// - FP32: 4 * element_count
|
|
// - FP16: 2 * element_count
|
|
// - QX_Y (quantized): element_count / QKX_Y * sizeof(block_qx_y)
|
|
// See ggml.c for values of QK and block sizes of specific formats.
|
|
byte[] data;
|
|
}
|
|
```
|
|
|
|
## Data types
|
|
|
|
- 0: `FP32`
|
|
- 1: `FP16`
|
|
- 2: `Q4_0`
|
|
- 3: `Q4_1`
|
|
- 4: *unused*
|
|
- 5: `Q4_2`
|
|
- 6: *unused*
|
|
- 7: `Q5_0`
|
|
- 8: `Q5_1`
|
|
- 9: `Q8_0`
|