RedMod/recursive-v4-flash-rejected-samples

RTS × deepseek-v4-flash — REJECTED traces

25,210 rejected agentic attempts from deepseek/deepseek-v4-flash on the Recursive Task Synthesis corpus. The accepted counterpart is RedMod/recursive-v4-flash.

Every row is an attempt that ran and was graded, and the task's own tests returned 0. Harness failures (failed_pre_checks — the model never got a scored shot) are deliberately EXCLUDED, so nothing here misattributes a plumbing bug to the model.

Why these are worth keeping

They are near-misses, not garbage. Measured over a 1,544-attempt sample:

signal	rate
garbled / non-Latin tokens	0.8%
empty assistant text	0.2%
looping (same tool call ≥5×)	10.9%
median duplicate-tool-call fraction	0.06
median turns	26

The model typically does the substantive work and then misses part of the deliverable spec. Categorising 25,007 rejected attempts by what the grader objected to:

failure mode	share
a required artifact was absent	60.9%
shortcut / forbidden modification detected	19.0%
placeholder (TODO, stub) left in output	13.8%
artifact present but wrong shape	3.1%
value mismatch	2.1%

Per-check fail rates show the same gradient — surface compliance is easy, exactness is not:

check	fail rate
required_evidence (did you leave something)	32.7%
intermediate_artifact	56.3%
no_shortcut	72.2%
final_semantics	79.0%

Partial credit — the binary reward hides most of the signal

reward is 0 for every row here, but the graders run several independent test functions per task and these attempts pass most of them. Across 194,615 individual tests in 24,604 rejected attempts:

individual tests run	194,615
tests PASSED	113,057 (58.1%)
tests FAILED	81,558 (41.9%)
median per-attempt pass rate	50.0%
failed exactly ONE test	4,574 attempts (18.6%)

Most common shapes: 2/4 (4,151), 1/4 (4,139), 3/4 (2,848), 0/4 (2,096).

So a ~12% task-level accept rate coexists with a ~58% test-level pass rate, and nearly one rejected attempt in five missed by a single check. Use the tests_* columns to filter for near-misses rather than treating every row as an equal failure.

Schema

One row per attempt (a task retried twice contributes two rows; dedupe on prompt_id for task-level counts).

column
prompt_id, task_id, dataset_index	identity; task_id joins to the corpus
skip_bucket	always failed_task here
try, harness, model	which attempt, which harness ran it
prompt, docker_image, cwd, difficulty, category	task metadata
messages	the full trajectory as a native list of structs — {role, content, reasoning_content, tool_calls[], tool_call_id, name}; tool_calls is itself a list of {id, type, name, arguments} (arguments stays a JSON string, since its shape is per-tool)
turns	native list of {turn, reasoning, n_tool_calls}
tools	the harness tool schema (JSON string)
n_turns, submitted, reasons	episode outcome
reward, test_sh_returncode, test_sh_tail	grader output — test_sh_tail names the failing checks
tests_total, tests_passed, tests_failed, tests_pass_pct	per-TEST partial credit parsed from the grader output (null when unparseable)
env_cost, token_cost	real spend for this attempt
prompt_hint	scaffolding appended to the prompt, stripped from messages

Run configuration

max_turns 100, max_tries 1, temperature 0.5, top_p 0.95, reasoning_effort high, harnesses hermes / opencode / pi (no terminus2), Daytona sandboxes, verifier agentic (the task's own tests/test.sh).

prompt_hint is set on these rows (the "hint-v3" self-critical + enumerate + verify-as-stated instruction). It is stripped from messages so the trajectory reads as the unmodified task — check the column before mixing with unhinted data.

Caveats

• reward is MISSING on ~3% of rows: test.sh never wrote /logs/verifier/reward.txt (usually a grader that died early). Treated as a failure, but it is not a clean 0.
• ~4% of the underlying corpus is unsolvable regardless of model (the instruction cites rewrite_*.json docs the image never ships); those tasks were filtered out of this run's dataset, but related noise may remain.