decoderstack-d24 / tokenizer

Commit History

tokenizer: keep the pre-fix token_bytes so published bpb numbers stay reproducible
4aad0ae
verified

ChrisMcCormick commited on

tokenizer: regenerate token_bytes.pt with upstream 2ce972a raw-bytes fix (191/32768 ids corrected)
47b3fec
verified

ChrisMcCormick commited on

Add model card, meta.json, logs, tokenizer, training source, and nanochat converter
11f07f9
verified

ChrisMcCormick commited on