Updated open-sci-ref baselines. Re-training without dropout. Re-training on DCLM, FineWeb-Edu, Nemotron, HPLT-2, Pile. Further ref datasets included.
AI & ML interests
Researching and building foundation models with improved generalization and reasoning. LAION & friends spin-off for open-sourcing foundation models with strong generalization and reasoning , including datasets necessary for their creation, to serve as common open, reproducible grounds for further research experiments.
Recent Activity
View all activity
models 102
open-sci/open-sci-ref-v0.02-1.7b-slimpajama-300B-4096
2B • Updated • 10
open-sci/open-sci-ref-v0.02-1.7b-pile-300B-4096
2B • Updated • 8
open-sci/open-sci-ref-v0.02-1.7b-hplt-2.0-300B-4096
2B • Updated • 14
open-sci/open-sci-ref-v0.02-1.3b-pile-300B-4096
1B • Updated • 13
open-sci/open-sci-ref-v0.02-1.3b-hplt-2.0-300B-4096
1B • Updated • 13
open-sci/open-sci-ref-v0.02-1.3b-fineweb-edu-1.4t-300B-4096
1B • Updated • 14
open-sci/open-sci-ref-v0.02-1.3b-slimpajama-300B-4096
1B • Updated • 13
open-sci/open-sci-ref-v0.02-1.7b-dclm-300B-4096-longsft_16k
Feature Extraction • 2B • Updated • 15
open-sci/open-sci-ref-v0.02-1.7b-fineweb-edu-1.4t-300B-4096-longsft_16k
Feature Extraction • 2B • Updated • 39 • 1
open-sci/open-sci-ref-v0.02-0.4b-commoncorpus-300B-4096
0.4B • Updated • 13