savoji/qwen3.5-0.8b-tablevqa-lora
Updated ā¢ 29
š Open to Work
Sajjad Pakdamansavoji
savoji
AI & ML interests
VLM, VLA, LLM, Robotics, Computer Vision
Recent Activity
repliedto their post about 2 hours ago
SO-Arena scored 21 System One models, and the most accurate one ranks only seventh on reliability.
System One models pick from fixed options in one forward pass, so teams use them for routing, tool choice and verification. A leaderboard score does not say whether the pick survives shuffled options or a missing right answer. SO-Arena (2,919 tasks, 11 domains) measures that separately.
š What's new
ā¢ Reliability R = sqrt(S*N). S is stability over 12 option reorderings. N = 2AUC-1 on spotting that the correct option was removed
ā¢ JevAny-27B is the most reliable System One model yet ranks eleventh in capability
ā¢ 14 of 18 models reach S above 0.8, but N runs from near chance to about 0.8
š The numbers
ā¢ GPT-6 Sol (82.3) and Luna (79.7) lead. The best System One models, GLiDE and KnowLine-31B, score 70.2
ā¢ Open-weight System One models answer within 0.3 s per task on one H100
ā¢ Knowledge, sentiment, retrieval: comparable to Luna, with little gain from a bigger backbone or thinking
ā¢ Math: thinking lifts StartLux Auto from 50.8 to 92.5. In intent, safety, tools and web, the larger backbone helps far more
ā ļø Caveats
ā¢ The backbone gain mixes size with Qwen3.5 vs Qwen3.8. Thinking is tested on one model
ā¢ Two closed System Two references only, and no independent replication yet
Before adopting a model, check per-domain scores, then reorder options and drop the correct one on your own task. The MIT repo ships the scoring and reliability tooling.
https://huggingface.co/papers/2611.15777
https://github.com/holdrain/SO-Arena
Do you test decision models for order sensitivity before putting them in an agent?
š Read it elsewhere
ā¢ The deep dive on Medium: https://medium.com/p/e923d2f58e85
ā¢ The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/so-arena-finds-the-most-accurate
ā¢ The thread on X: https://x.com/saavoji/status/2121665100981444943
ā¢ Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#so-arena-finds posted an update about 22 hours ago
SO-Arena scored 21 System One models, and the most accurate one ranks only seventh on reliability.
System One models pick from fixed options in one forward pass, so teams use them for routing, tool choice and verification. A leaderboard score does not say whether the pick survives shuffled options or a missing right answer. SO-Arena (2,919 tasks, 11 domains) measures that separately.
š What's new
ā¢ Reliability R = sqrt(S*N). S is stability over 12 option reorderings. N = 2AUC-1 on spotting that the correct option was removed
ā¢ JevAny-27B is the most reliable System One model yet ranks eleventh in capability
ā¢ 14 of 18 models reach S above 0.8, but N runs from near chance to about 0.8
š The numbers
ā¢ GPT-6 Sol (82.3) and Luna (79.7) lead. The best System One models, GLiDE and KnowLine-31B, score 70.2
ā¢ Open-weight System One models answer within 0.3 s per task on one H100
ā¢ Knowledge, sentiment, retrieval: comparable to Luna, with little gain from a bigger backbone or thinking
ā¢ Math: thinking lifts StartLux Auto from 50.8 to 92.5. In intent, safety, tools and web, the larger backbone helps far more
ā ļø Caveats
ā¢ The backbone gain mixes size with Qwen3.5 vs Qwen3.8. Thinking is tested on one model
ā¢ Two closed System Two references only, and no independent replication yet
Before adopting a model, check per-domain scores, then reorder options and drop the correct one on your own task. The MIT repo ships the scoring and reliability tooling.
https://huggingface.co/papers/2611.15777
https://github.com/holdrain/SO-Arena
Do you test decision models for order sensitivity before putting them in an agent?
š Read it elsewhere
ā¢ The deep dive on Medium: https://medium.com/p/e923d2f58e85
ā¢ The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/so-arena-finds-the-most-accurate
ā¢ The thread on X: https://x.com/saavoji/status/2121665100981444943
ā¢ Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#so-arena-finds posted an update 2 days ago
Chrome's modern-web-guidance repo gained 1,875 GitHub stars in one day (4,476 total, 252 forks, Apache-2.0), and it was only created on 2026-01-27.
Its bet: an agent's stale web habits are a delivery problem, not a training problem. The README says agents default to older patterns "because LLM training data contains vast amounts of legacy code." So instead of retraining, a skill's CLI returns curated guides into the agent's context window when a task needs them.
š What's in it
ā¢ 250 web features: 140 CSS and layout, 74 JavaScript and APIs, 36 HTML and DOM
ā¢ Each guide has a retrievable ID, such as animate-to-from-top-layer
ā¢ Evals prune what models already know, to save tokens
ā¢ Stewarded by the Chrome, Edge, Firefox and MDN teams
š ļø Try it, no install
```
npx modern-web-guidance@latest search "animate a dialog modal backdrop"
npx modern-web-guidance@latest retrieve "animate-to-from-top-layer"
npx modern-web-guidance@latest legacy-scan .
```
ā ļø Caveats
ā¢ No benchmark numbers in the README; it links an Eval Dashboard, but the gain is unquantified
ā¢ Stars show interest, not adoption or quality
ā¢ Guides cost context tokens and rely on the agent calling the skill at the right moment
ā¢ Web platform only; this is the authoring repo, the shipped guides live in GoogleChrome/modern-web-guidance
https://github.com/GoogleChrome/modern-web-guidance-src
https://goo.gle/modern-web-guidance
If you run legacy-scan on your own codebase, would you trust runtime guidance over a model upgrade to keep your agent current?
š Read it elsewhere
ā¢ The deep dive on Medium: https://medium.com/p/fa2703c6324b
ā¢ The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/chromes-modern-web-guidance-treats
ā¢ The thread on X: https://x.com/saavoji/status/2121292924419985453
ā¢ Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#chrome-treatsOrganizations
None yet