ChrisMcCormick
/

decoderstack-d24

Model card Files Files and versions

decoderstack-d24 / tokenizer

677 kB

Ctrl+K

Ctrl+K

1 contributor

History: 3 commits

ChrisMcCormick's picture

tokenizer: keep the pre-fix token_bytes so published bpb numbers stay reproducible

4aad0ae verified 4 days ago

token_bytes.pt

133 kB
xet

tokenizer: regenerate token_bytes.pt with upstream 2ce972a raw-bytes fix (191/32768 ids corrected) 4 days ago
token_bytes_legacy.pt

133 kB
xet

tokenizer: keep the pre-fix token_bytes so published bpb numbers stay reproducible 4 days ago
tokenizer.pkl
Detected Pickle imports (1)
- "tiktoken.core.Encoding"
How to fix it?
412 kB
xet

Add model card, meta.json, logs, tokenizer, training source, and nanochat converter 4 days ago