|
Download README.md from nutrientdocs/document-classification-benchmark: direct link, hf CLI and curl.
- Browser
- Download file 3.75 kB
-
https://huggingface.co/datasets/nutrientdocs/document-classification-benchmark/resolve/main/README.md
- Command line
-
hf download hf://datasets/nutrientdocs/document-classification-benchmark/README.md
-
curl -L -o README.md https://huggingface.co/datasets/nutrientdocs/document-classification-benchmark/resolve/main/README.md
3.75 kB
| license: other | |
| license_name: nutrient-eval-restricted | |
| pretty_name: Document Classification Benchmark (open-vocab, zero-shot) | |
| task_categories: | |
| - zero-shot-image-classification | |
| - image-classification | |
| language: | |
| - en | |
| tags: | |
| - document-ai | |
| - open-vocabulary | |
| - zero-shot-image-classification | |
| - document-image-classification | |
| size_categories: | |
| - 1K<n<10K | |
| dataset_info: | |
| features: | |
| - name: image | |
| dtype: image | |
| - name: label | |
| dtype: string | |
| - name: candidate_labels | |
| list: string | |
| - name: source | |
| dtype: string | |
| - name: license_tag | |
| dtype: string | |
| - name: doc_id | |
| dtype: string | |
| - name: track | |
| dtype: string | |
| splits: | |
| - name: test | |
| num_bytes: 453775818 | |
| num_examples: 1274 | |
| download_size: 453674933 | |
| dataset_size: 453775818 | |
| configs: | |
| - config_name: default | |
| data_files: | |
| - split: test | |
| path: data/test-* | |
| # Document Classification Benchmark (open-vocab, zero-shot) | |
| **Given a document image and an arbitrary set of text labels, which one is right?** A held-out, zero-shot, | |
| *open-vocabulary* evaluation for document-type classification — labels are supplied at inference, not baked | |
| into a head. Test split only; not for training. Every image is drawn from a **permissively-licensed, | |
| redistributable** source. | |
| Powers the | |
| [document-classification-leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard) | |
| and evaluates [`document-classification-v2`](https://huggingface.co/nutrientdocs/document-classification-v2) | |
| (commercial) and the open-weight | |
| [`document-classification-v1`](https://huggingface.co/nutrientdocs/document-classification-v1); try it live in | |
| the [demo](https://huggingface.co/spaces/nutrientdocs/document-classification-demo). | |
| ## Tracks (macro-F1) | |
| - **DocLayNet** — document page categories (financial reports, scientific articles, manuals, patents, …). | |
| - **Forms** — form vs non-form and fine form sub-types. | |
| - **OOD** — document *types absent from training* (invoices, handwriting, charts, tables) — the open-vocab | |
| stress test. | |
| - **OOV** — an evaluation *protocol* (not extra rows): score the same images against never-seen synonym | |
| paraphrases of each label ("invoice" → "bill") to measure concept-matching vs memorized wording. | |
| ## Contents & sources | |
| 897 test images (`track` column selects the slice). All sources are redistributable: | |
| | Source | Track | License | | |
| |---|---|---| | |
| | DocLayNet (`ds4sd/DocLayNet`) | doclaynet | CDLA-Permissive-1.0 | | |
| | synthetic IRS forms (public-domain templates + faked fields) | forms | public-domain | | |
| | mixed permissive HF sources (invoice / handwriting / chart / table) | ood | per upstream source | | |
| > **Tobacco3482 is deliberately excluded.** Its research-only license is not redistributable, so it is not | |
| > part of this public benchmark (it is used only in internal evaluation). | |
| ```python | |
| from datasets import load_dataset | |
| ds = load_dataset("nutrientdocs/document-classification-benchmark", split="test") | |
| ``` | |
| ## Schema | |
| | field | meaning | | |
| |---|---| | |
| | `image` | the document image (RGB) | | |
| | `label` | gold document-type label | | |
| | `candidate_labels` | the open-vocab candidate set for the row's track | | |
| | `source` | originating dataset | | |
| | `license_tag` | redistribution tier of the source | | |
| | `doc_id` | provenance id | | |
| | `track` | doclaynet · forms · ood | | |
| ## About the author | |
| <a href="https://nutrient.io/"> | |
| <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" /> | |
| </a> | |
| This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks. | |