Datasets:

bigcode
/

bigcodebench-results

We have completed the apples-to-apples native <think>...</think> Test-Time Chain-of-Thought (max_new_tokens=1536, think_budget=960) evaluation on NVIDIA A100-SXM4-80GB using Strict Prompt-Probe Dominance (0% train-on-test overlap, 0 test-suite oracle calls):

Benchmark	`Qwen/Qwen3.8-27B` Base	`RealCoder-v3` (`240x` Clean `complete`, `0` think tok)	`RealCoder-v3` (`240x` Clean `<think>` CoT, Blind Prompt-Probe)	Gain vs. Base (`<think>`)	Regressions
`BigCodeBench-Hard` (148)	`41/148` (`27.70%`)	`44/148` (`29.73%`)	`45/148` (`30.41%`)	`+2.71%` (`+4` tasks)	`0`
`EvalPlus HumanEval` (164)	`117/164` (`71.34%`)	`122/164` (`74.39%`)	`137/164` (`83.54%`) / `138/164` (`84.15%`)	`+12.20%` (`+20` tasks)	`0`
`EvalPlus HumanEval+` (`80x`, 164)	`117/164` (`71.34%`)	`122/164` (`74.39%`)	`137/164` (`83.54%`) / `138/164` (`84.15%`)	`+12.20%` (`16/16` switched = `100%` `F->P`)	`0`
`EvalPlus MBPP` (378)	`327/378` (`86.51%`)	`334/378` (`88.36%`)	`360/378` (`95.24%`)	`+8.73%` (`+33` tasks)	`0`
`EvalPlus MBPP+` (`35x`, 378)	`281/378` (`74.34%`)	`286/378` (`75.66%`)	`302/378` (`79.89%`) / `305/378` (`80.69%`)	`+5.55%` (`+21` tasks)	`0`

Full JSON Report: clean_240x_think_eval_results.json
Sanitized <think> Samples (.jsonl):

Add clean 240x native <think> CoT (instruct) BigCodeBench-Hard submission (45/148 = 30.41%)5abb0e17

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment

Add BigCodeBench-Hard 280x RealCoder-v5 results for ProbioticFarmer/Qwen3.8-27B-BarronPolyStep-RealCoder-v3 (29.73%)

Updated Clean 240x Evaluation: Both complete (0 Thinking Tokens) and Native <think> Chain-of-Thought (a100-large)

Updated Clean `240x` Evaluation: Both `complete` (`0` Thinking Tokens) and Native `<think>` Chain-of-Thought (`a100-large`)