README.md

metadata

configs:
  - config_name: default
    data_files:
      - split: train
        path: data/*.jsonl.gz
tags:
  - arxiv
  - ocr
  - chandra
  - chandra-ocr-2
  - markdown
  - html
  - images
  - hf-jobs
  - uv-script

arXiv OCR with Chandra OCR 2

This output bundle stores OCR results for arXiv PDFs using datalab-to/chandra-ocr-2.

Summary

• Output dataset: nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-v2-20260416
• Output bucket: hf://buckets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-v2-20260416
• Source paper IDs in input list: 1
• Processed IDs recorded in state/processed_ids.txt: 1
• Successes: 1
• Partial successes: 0
• Errors: 0
• Next shard index: 1
• Updated at: 2026-04-16T09:15:30.914138+00:00

Files

• data/part-*.jsonl.gz: OCR result shards, one JSON object per paper
• state/processed_ids.txt: completed paper IDs used for resume
• state/summary.json: aggregate counters and bookkeeping
• <paper_id>/<paper_id>.md, .html, _metadata.json: optional per-paper outputs when --write-paper-files is enabled
• <paper_id>/images/*: extracted image assets when --include-images is enabled

Each paper record includes:

• num_pages: total number of pages in the source PDF
• num_pages_processed: number of pages actually sent to OCR
• pdf_exceeds_page_limit: whether the PDF had more pages than the configured OCR cap
• max_pages_per_paper: configured OCR page cap for the run
• pages_with_images: number of OCR pages that produced extracted images
• image_file_count: total number of extracted image files for the paper
• paper_output_prefix: root folder for the optional per-paper files

Load the results

from datasets import load_dataset

dataset = load_dataset("nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-v2-20260416", data_files="data/*.jsonl.gz", split="train")
print(dataset[0]["paper_id"])
print(dataset[0]["markdown"][:1000])

Job config

• Prompt type: ocr_layout
• Page batch size: 16
• Max output tokens: 12384
• Max model length: 18000
• GPU memory utilization: 0.85
• Minimum arXiv request interval: 3.1 seconds
• Max pages per paper sent to OCR: 200
• Bucket backend: hf-cli
• Paginate output: False
• Include headers/footers: False
• Include images: True
• Write paper files: True
• Image URL base: https://huggingface.co/buckets/nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-v2-20260416/resolve

Reproduction

hf jobs uv run --flavor l4x1 --image vllm/vllm-openai:v0.17.0 \
  -s HF_TOKEN --timeout 2d \
  ./chandra2-arxiv-ocr.py --output-dataset nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-v2-20260416 \
  --output-bucket nielsr/arxiv-chandra-ocr-2-include-images-demo-2604-08626-spacing-fix-v2-20260416 \
  --paper-ids-url https://.../hf_missing_paper_ids.txt