chaoliangUNSW (chaoliang yan)

posted an update 7 days ago

Post

171

Jev-style decisions, now at 2B: Jev-Style-2B-Decision-v3

State and typed questions in (pick one, yes/no, score); a calibrated probability for every option out, in one pass.

Try it in your browser:
chaoliangUNSW/jev-style-2b

• JevBench v1.4.1, 231 public items (self-run with the official v1.4.1 harness, not an official board entry): 73.6% (170/231), the highest public accuracy among the Qwen3.5-2B-family systems on the board (decider-2b 71.0%, open-jev-zefan-2b 64.5%; the lead over decider-2b is inside the 95% CI, 67.6–78.9%). 42 of the 82 board systems score higher, almost all 4B or larger. +9.5 points over our 0.8B v3. Jev 1.13 is well ahead at 86.6%.
• tweet_topic, zero-shot (n = 1,693): 82.2% accuracy [80.4, 84.0], above Jev's 79.3%; macro-F1 is below Jev (0.678 vs 0.694).
• fin_topic, zero-shot (n = 4,117): 61.1%, +14.4 points over our 0.8B v3 (Jev: 67.0%).
• 25,600 tokens per call, no option cap, nothing truncated.
• Read once, then ask: on an Apple M1 Max (GGUF F16), the first question about a 24,501-token input took 16.2 s and a further question about the same state 0.17 s (shared machine, indicative).

Contamination checks against the training pool: 0 hits for the JevBench items, and no exact or near-duplicate overlap with the tweet_topic and fin_topic test sets. Needs the shipped runtimes (PyTorch, llama.cpp with the bundled scorer, MLX); trained on a reduced data pool (60M tokens).

Models (Apache-2.0; some training data has restrictive or unclear terms and includes OpenAI GPT and Anthropic Claude outputs, see "Training data and licences" on the main card):
chaoliangUNSW/Jev-Style-2B-Decision-v3
chaoliangUNSW/Jev-Style-2B-Decision-v3-GGUF
chaoliangUNSW/Jev-Style-2B-Decision-v3-MLX
All v3 builds: chaoliangUNSW/jev-style-decision-v3-08b-2b-6ab87f32380cbd8c03b608b9

Not affiliated with TypeSafe or Jev.

1 reply

·

replied to their post 9 days ago

Thanks, this was a fair catch. I've rewritten the post around it: the headline is now the out-of-teacher results, and the typed number sits next to a teacher-noise reference.

On your question: an exact ceiling isn't possible with what the dataset releases. The test split ships only the mean of the three teacher samples plus a label_agreement flag, so the card's 73.5% can't be recomputed on the 400 test states. What I can add:

One draw from the teacher's mean distribution matches the gold argmax 65.9% of the time on the test split.
Split by teacher margin (gap between its top two probabilities), v3 vs Laya typed on the same rows:

teacher margin	n	v3	Laya typed	diff (95% CI)
< 0.1 (near-ties)	315	51.7	51.7	+0.0 [−4.4, +4.5]
0.1–0.3	555	69.7	66.1	+3.6 [+0.2, +7.3]
≥ 0.3	1,130	91.4	88.7	+2.7 [+0.8, +4.6]

So v3 isn't gaining by fitting the teacher's coin-flips. That still can't separate task skill from the teacher's consistent biases, since both models trained on its labels. Only an independent gold can, which is why the post now leads with Banking77, tweet_topic and JevBench. The gap to Jev (72.7, zero-shot) shouldn't be read as skill, and the post no longer frames it that way.

The same table and reference are going on the model card.

reacted to their post with 🔥 9 days ago

Post

182

Jev-style decisions on your laptop: Jev-Style-0.8B-Decision-v3

Give it any text and a question; it returns a calibrated probability for every option in one forward pass: pick one, yes/no, or a score. No text generation.

On data it never trained on:
• Banking77, all 77 intents in one pass (400 test rows): 68.2% vs 49.2% for the best official Laya checkpoint (+19.0, paired 95% CI +14.0 to +24.0)
• tweet_topic, zero-shot: 75.5% vs 63.2% for English Laya, within 4 points of Jev (79.3%), and better calibrated than Jev (ECE 0.027 vs 0.063)
• JevBench v1.4.1, 231 public items, zero-shot: 64.1%, ahead of every Qwen3.5-0.8B-based system on the board; ahead of Laya (58.4%) but inside the CI

Jev is still ahead on all three, and decider-2b scores higher on JevBench (71.0%). v3's case is size: 0.53 GB in 4-bit, runs on CPU.

Typed decisions (in-domain): 79.2% vs 76.6% for Laya typed, both trained on the same train split (paired CI +1.0 to +4.2). Read this as agreement with the dataset's teacher labels, not decision skill:
• teacher-noise reference: one draw from the teacher's own distribution matches the gold label 65.9% of the time (test split); teacher self-agreement is 73.5% (dataset card)
• on the teacher's near-ties, v3 and Laya typed score the same (51.7% vs 51.7%); v3's lead comes from cases where the teacher is clear
• Jev's 72.7% there is zero-shot, so not a like-for-like comparison

Also: up to 25,600 tokens per call (98.3% on real 24K-token documents); ~44 ms per short decision on an M1 Max (llama.cpp); one read of a document answers every question in one call; ahead of Laya multilingual in 51 of 51 languages.

pip install "jev-style[torch]" · https://github.com/lawrence3699/jev-style

Try it in your browser:
chaoliangUNSW/jev-style-v3

Model:
chaoliangUNSW/Jev-Style-0.8B-Decision-v3

Apache-2.0. Not affiliated with TypeSafe, Jev or Laya. Thanks to @dipankarsarkar for pushing on the typed-decisions framing.

3 replies

·

posted an update 9 days ago

Post

182

Jev-style decisions on your laptop: Jev-Style-0.8B-Decision-v3

Give it any text and a question; it returns a calibrated probability for every option in one forward pass: pick one, yes/no, or a score. No text generation.

On data it never trained on:
• Banking77, all 77 intents in one pass (400 test rows): 68.2% vs 49.2% for the best official Laya checkpoint (+19.0, paired 95% CI +14.0 to +24.0)
• tweet_topic, zero-shot: 75.5% vs 63.2% for English Laya, within 4 points of Jev (79.3%), and better calibrated than Jev (ECE 0.027 vs 0.063)
• JevBench v1.4.1, 231 public items, zero-shot: 64.1%, ahead of every Qwen3.5-0.8B-based system on the board; ahead of Laya (58.4%) but inside the CI

Jev is still ahead on all three, and decider-2b scores higher on JevBench (71.0%). v3's case is size: 0.53 GB in 4-bit, runs on CPU.

Typed decisions (in-domain): 79.2% vs 76.6% for Laya typed, both trained on the same train split (paired CI +1.0 to +4.2). Read this as agreement with the dataset's teacher labels, not decision skill:
• teacher-noise reference: one draw from the teacher's own distribution matches the gold label 65.9% of the time (test split); teacher self-agreement is 73.5% (dataset card)
• on the teacher's near-ties, v3 and Laya typed score the same (51.7% vs 51.7%); v3's lead comes from cases where the teacher is clear
• Jev's 72.7% there is zero-shot, so not a like-for-like comparison

Also: up to 25,600 tokens per call (98.3% on real 24K-token documents); ~44 ms per short decision on an M1 Max (llama.cpp); one read of a document answers every question in one call; ahead of Laya multilingual in 51 of 51 languages.

pip install "jev-style[torch]" · https://github.com/lawrence3699/jev-style

Try it in your browser:
chaoliangUNSW/jev-style-v3

Model:
chaoliangUNSW/Jev-Style-0.8B-Decision-v3

Apache-2.0. Not affiliated with TypeSafe, Jev or Laya. Thanks to @dipankarsarkar for pushing on the typed-decisions framing.

3 replies

·

chaoliang yan PRO

AI & ML interests

Recent Activity

Organizations

chaoliang yan PRO

AI & ML interests

Recent Activity

Organizations

chaoliangUNSW's activity