Sure, let me know what kind of advice are you looking for.
Though, a general advice, when it comes about llms is to never trust one benchmark. Test as much as possible.
👉 check void.1
appvoid
appvoid
AI & ML interests
singularity through byte-level tokens and small language models - apps and models on dotlabs
Recent Activity
liked a dataset about 16 hours ago
AxiomicLabs/FactBase-DCLM updated a Space about 17 hours ago
dotlabs/cognirank repliedto their post about 18 hours ago
New leaderboard coming to the small language model community!
The leaderboard measures cognitive skills, which is more challenging and beyond language modeling and instruction-following. This is a first step towards more everyday useful and general small models.
https://huggingface.co/spaces/dotlabs/cognirankOrganizations
replied to their post about 18 hours ago
posted an update 3 days ago
Post
7,471
New leaderboard coming to the small language model community!
The leaderboard measures cognitive skills, which is more challenging and beyond language modeling and instruction-following. This is a first step towards more everyday useful and general small models.
dotlabs/cognirank
The leaderboard measures cognitive skills, which is more challenging and beyond language modeling and instruction-following. This is a first step towards more everyday useful and general small models.
dotlabs/cognirank
- 2 replies
replied to their post 14 days ago
Sooner than you might think
replied to their post 14 days ago
check klm-1.5
posted an update 15 days ago
Post
4,005
You know, I can't believe we are actually doing it, literally advancing the frontier at the edge! We've come a long way since the first models on open slm leaderboard, and, as many things that change, performance is one of the main things that has exploded.
7 years ago, the best we had was GPT-2. Today, we got 90m models beating smollm2! I can't barely see what a future with increasingly powerful small models will look like.
All I can say is that the best is yet to come. And will be agentic.
7 years ago, the best we had was GPT-2. Today, we got 90m models beating smollm2! I can't barely see what a future with increasingly powerful small models will look like.
All I can say is that the best is yet to come. And will be agentic.
- 11 replies
reacted to Datdanboi25's post with 🚀 17 days ago
Post
3,648
THE SLM FRONTIER ADVANCES!
bench-labs/cagliostro-v3 just hit an Intelligence Index of 26.13 on the AxiomicLabs/Open_SLM_Leaderboard a 146M-param model trained completely from scratch on a single consumer GPU. That's 2nd place overall, and as far as I can tell, the most capable SLM trained on consumer hardware to date. Beating SmolLM-135m on 1/8th of the data is just silly levels of efficiency.
Big congrats to the @BenchLabs team and specifically @TobiasLogic !
bench-labs/cagliostro-v3 just hit an Intelligence Index of 26.13 on the AxiomicLabs/Open_SLM_Leaderboard a 146M-param model trained completely from scratch on a single consumer GPU. That's 2nd place overall, and as far as I can tell, the most capable SLM trained on consumer hardware to date. Beating SmolLM-135m on 1/8th of the data is just silly levels of efficiency.
Big congrats to the @BenchLabs team and specifically @TobiasLogic !
- 6 replies
replied to their post 24 days ago
@appvoid do you actually have a B200?
uh ofc not
replied to Banaxi-Tech's post 25 days ago
more models is fine, but focus on usability and you should be fine
posted an update about 1 month ago
Post
153
How much should OpenAI offer for a name like Dot Labs?
- 1 reply
posted an update about 1 month ago
Post
176
Reality has not limits if you know how to shape it, one step at a time.
- 1 reply
reacted to TobiasLogic's post with 🚀 about 1 month ago
Post
3,621
We’ve been cooking something new at Bench Labs.
Introducing Cagliostro-v3, our new 146M parameter language model trained completely from scratch.
The run isn’t even finished yet.
At the current checkpoint:
• 146M parameters
• 72.7B / 75B tokens trained
• 26.27 Open SLM Index
• 43.80 ArithMark-3
• Trained on a single RTX 5090
• ~90K to 103K tokens/sec during training
• ~9 days for the full run
• Apache 2.0
For some context, SmolLM2-135M scores 27.13 on the same Index after being trained on roughly 2 trillion tokens.
Cagliostro-v3 is currently at 26.27 with only ~72.7B.
That’s around 27x fewer training tokens.
The model also currently Hold the number 3rd spot for ArithMark-3, scoring 43.80
This wasn’t achieved by just throwing more tokens at the model. A huge part of v3 has been figuring out architecture, data mixture, and training dynamics at this scale.
The model uses a custom 30-layer decoder architecture with grouped-query attention and cross-head subspace attenuation, SwiGLU, RMSNorm, RoPE, tied embeddings, and a warmup-stable-decay training schedule.
During cooldown we also substantially shifted the data mixture toward higher-quality synthetic textbook and mathematics data, with the mathematics share increasing from 10% to 28%.
And everything is open.
The repository contains the training history with checkpoints pushed roughly every 30 minutes, so you can inspect how the model evolved throughout training rather than only seeing the final weights.
This is still a pre-final checkpoint. We have roughly 2.3B tokens left and the learning-rate cooldown is still running.
So 26.27 isn’t the final number.
Really excited to see where the last part of the run lands.
Cagliostro-v3:
bench-labs/cagliostro-v3
Built by Bench Labs.
Open SLM Leaderboard:
AxiomicLabs/Open_SLM_Leaderboard
Introducing Cagliostro-v3, our new 146M parameter language model trained completely from scratch.
The run isn’t even finished yet.
At the current checkpoint:
• 146M parameters
• 72.7B / 75B tokens trained
• 26.27 Open SLM Index
• 43.80 ArithMark-3
• Trained on a single RTX 5090
• ~90K to 103K tokens/sec during training
• ~9 days for the full run
• Apache 2.0
For some context, SmolLM2-135M scores 27.13 on the same Index after being trained on roughly 2 trillion tokens.
Cagliostro-v3 is currently at 26.27 with only ~72.7B.
That’s around 27x fewer training tokens.
The model also currently Hold the number 3rd spot for ArithMark-3, scoring 43.80
This wasn’t achieved by just throwing more tokens at the model. A huge part of v3 has been figuring out architecture, data mixture, and training dynamics at this scale.
The model uses a custom 30-layer decoder architecture with grouped-query attention and cross-head subspace attenuation, SwiGLU, RMSNorm, RoPE, tied embeddings, and a warmup-stable-decay training schedule.
During cooldown we also substantially shifted the data mixture toward higher-quality synthetic textbook and mathematics data, with the mathematics share increasing from 10% to 28%.
And everything is open.
The repository contains the training history with checkpoints pushed roughly every 30 minutes, so you can inspect how the model evolved throughout training rather than only seeing the final weights.
This is still a pre-final checkpoint. We have roughly 2.3B tokens left and the learning-rate cooldown is still running.
So 26.27 isn’t the final number.
Really excited to see where the last part of the run lands.
Cagliostro-v3:
bench-labs/cagliostro-v3
Built by Bench Labs.
Open SLM Leaderboard:
AxiomicLabs/Open_SLM_Leaderboard
- 8 replies
posted an update about 1 month ago
Post
89
How old were you when you discovered good AI agents favor the Strategy Design Pattern over any other?
- 1 reply
posted an update about 2 months ago
Post
135
LLMS for edge devices?
What if we go the reliable route instead of the speed route?
What if we make it run on sbcs with few megabytes available?
That's the idea for the next model.
Keep in tune.
What if we go the reliable route instead of the speed route?
What if we make it run on sbcs with few megabytes available?
That's the idea for the next model.
Keep in tune.
congrats!
posted an update about 2 months ago
Post
145
We trained a 10.9M byte-level recurrent Transformer on L3 and L6. (Loop 3 and Loop 6)
Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.
Same weights. More compute. Better predictions.
This is a new architecture for effective compute after several steps beyond original training!
We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.
One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!
I believe looped-transformers are the future of edge computing and this is a first step towards it.
Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f
appvoid/bet-10m
Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.
Same weights. More compute. Better predictions.
This is a new architecture for effective compute after several steps beyond original training!
We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.
One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!
I believe looped-transformers are the future of edge computing and this is a first step towards it.
Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f
appvoid/bet-10m
reacted to KlondikeDev's post with ❤️ about 2 months ago
Post
2,177
The SLM Consortium has begun work on a safety dataset for Small Language Models, with the creation of the dataset being headed by @wayneworkman2012
The dataset will focus on refusals and redirects surrounding dangerous or extreme sexual content, designed to be shaped sized appropriately for SLMs, without significantly lowering benchmark performance.
More info will be out soon!
slmconsortium
The dataset will focus on refusals and redirects surrounding dangerous or extreme sexual content, designed to be shaped sized appropriately for SLMs, without significantly lowering benchmark performance.
More info will be out soon!
- 3 replies
replied to their post 2 months ago
Well I don’t care if it’s AGI or not I just find it super helpful in real practice
It would be for me if my credits wouldn't drain even light mode 🥲
replied to their post 2 months ago
it is not AGI
For me it is not, but it's pretty close for me though