Jeva: System One Models from First Principles
Paper โข 2610.18773 โข Published โข 470
๐ค Open to Collab
Aliasgar Khimani
NovusEdge
AI & ML interests
None yet
Recent Activity
upvoted a paper 14 days ago
Jeva: System One Models from First Principles upvoted a paper 15 days ago
Thinking Inertia: LLMs Keep Thinking When Told Not To liked a model about 1 month ago
google/paligemma-3b-pt-224Organizations
upvoted a paper 14 days ago
upvoted a paper 15 days ago
liked a
model about 1 month ago
liked a
dataset about 1 month ago
updated a
dataset about 1 month ago
published a
dataset about 1 month ago
updated a
model about 1 month ago
published a
model about 1 month ago
replied to ginigen-ai's post 4 months ago
@RiverRider Just checked out the repo. The semiotic divergence angle is kinda interesting ngl, tracking where meaning splits across communities rather than pure confidence calibration. How are you seeing it perform on domain-specific jargon vs everyday terms?
reacted to ginigen-ai's post with ๐ฅ 4 months ago
Post
10,549
๐ง Does your LLM know when it's about to be wrong?
Most leaderboards measure accuracy. We measure metacognition โ whether a model catches its own errors. Benchmark + leaderboard + adapters, all open. ๐
The surprise: even a K-AI #1 model (JGOS-31B-Citizen) is the strongest on multiple-choice traps (trap_rate 0.005 โ ~2 misses in 400) yet blind to its own free-form mistakes (self-confidence AUROC = 0.5, pure random). A tiny base-frozen adapter recovers that signal.
Two independent axes (never compared across a row): โ trap_rate โ does it fall for tempting trap options? (lower = stronger) โก adapter gain ฮ โ how much a lightweight adapter catches errors the model itself misses. (higher = more adapter value)
What's open: ๐ 300+100 trap problems (each with a hidden trap + TICOS type) ๐ 24-model leaderboard ๐งฉ 11 per-model adapters โ adapters, NOT fine-tunes (base stays frozen; the adapter just reads the hidden state โ P(wrong))
Submit any HF model โ auto-scored daily at 09:00 KST and added to the board.
๐ Leaderboard โ ginigen-ai/Metacognition-Leaderboard-Space
๐ Benchmark โ ginigen-ai/Metacognition-Bench
๐งฉ Adapters โ https://huggingface.co/collections/FINAL-Bench/metacognition-adapters-6a42c032e6beb803dd032961
๐ Article โ https://huggingface.co/blog/ginigen-ai/metacognition
Benchmark by ginigen-ai ยท Adapters by FINAL-Bench (Darwin/Chimera platform + AETHER metacognition tech).
Most leaderboards measure accuracy. We measure metacognition โ whether a model catches its own errors. Benchmark + leaderboard + adapters, all open. ๐
The surprise: even a K-AI #1 model (JGOS-31B-Citizen) is the strongest on multiple-choice traps (trap_rate 0.005 โ ~2 misses in 400) yet blind to its own free-form mistakes (self-confidence AUROC = 0.5, pure random). A tiny base-frozen adapter recovers that signal.
Two independent axes (never compared across a row): โ trap_rate โ does it fall for tempting trap options? (lower = stronger) โก adapter gain ฮ โ how much a lightweight adapter catches errors the model itself misses. (higher = more adapter value)
What's open: ๐ 300+100 trap problems (each with a hidden trap + TICOS type) ๐ 24-model leaderboard ๐งฉ 11 per-model adapters โ adapters, NOT fine-tunes (base stays frozen; the adapter just reads the hidden state โ P(wrong))
Submit any HF model โ auto-scored daily at 09:00 KST and added to the board.
๐ Leaderboard โ ginigen-ai/Metacognition-Leaderboard-Space
๐ Benchmark โ ginigen-ai/Metacognition-Bench
๐งฉ Adapters โ https://huggingface.co/collections/FINAL-Bench/metacognition-adapters-6a42c032e6beb803dd032961
๐ Article โ https://huggingface.co/blog/ginigen-ai/metacognition
Benchmark by ginigen-ai ยท Adapters by FINAL-Bench (Darwin/Chimera platform + AETHER metacognition tech).
- 11 replies
replied to ginigen-ai's post 4 months ago
sweet! how about we take it over on twitter: https://x.com/0kaliasgar :))
replied to ginigen-ai's post 4 months ago
We're building something complementing to this @ https://engrammic.ai
You down to have a chat sometime about metacognition and externalized epistemics? ๐
