Precise Debugging Benchmarking (PDB)

⭐ Neurips'26: Evaluations & Datasets

📄 Paper  ·  💻 Code  ·  🌐 Project page  ·  🏆 Leaderboard

PDB is an automatic pipeline that turns any coding dataset into a debugging benchmark with fine-grained metrics. Beyond binary unit-test scores, PDB evaluates a debugger with edit-level precision (did the model touch only the lines it had to?) and bug-level recall (did it fix every fault?). This rewards targeted fixes and penalizes the regeneration behavior frontier LLMs often fall back on.

Frontier models like GPT-5.1-Codex and DeepSeek-V3.2-Thinking top unit-test leaderboards (>76%) but score at or below 45% on precision: they pass tests by rewriting, not repairing. PDB makes that gap measurable.

Released datasets

PDB-Single, PDB-Single-Full and PDB-Multi are derived from BigCodeBench and LiveCodeBench; PDB-Wild adds SWE-smith repositories. All are built by the PDB pipeline and evaluated with precision / recall / unit-test pass rate.

Citation

@article{chai2026pdb,
  title={Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?},
  author={Chai, Miaosen and Zhu, Wang Bill and Wang, Shangshang and Liu, Yejia and Bian, Song and Dong, Honghua and Neiswanger, Willie and Jia, Robin},
  journal={arXiv preprint arXiv:2604.17338},
  year={2026}
}

Contact

Questions / submissions: wangzhu@usc.edu, miaosenc@usc.edu.