Choose theme
AI Resources
ParseBench
ParseBench evaluates document parsing tools on tables, charts, content faithfulness, semantic formatting, and visual grounding.
The publisher provides annotated enterprise document pages, evaluation code, and reports for examining how a parser represents the original document. Use this as a first read, not a recommendation. Open the original project before trusting details like terms, limits, privacy, cost, setup, or safety.
What it is
A benchmark for parsed documents
ParseBench evaluates output from a selected parsing tool against annotated documents. It is a benchmark around parsers, rather than a document assistant.
Why it stands out
Separate views of parsing errors
The five dimensions separate table structure, chart values, text, meaningful formatting, and page locations instead of reducing the comparison to text similarity.
Availability
Code, dataset, and public reports
Python benchmark code is available on GitHub and PyPI, with the dataset on Hugging Face and a public comparison site. Running a pipeline also requires its selected parsing provider.
Why it matters
What makes it useful
A table can contain the right words while attaching a value to the wrong header. The dataset's table metric treats rows as records keyed by column headers: changing row or column order is different from dropping or transposing a header. This checks the association a reader needs when looking up a value.
What to know
Where it fits
For readers comparing parser runs, the documented compare command presents two parsing tools side by side. Per-case HTML reports let readers examine individual examples alongside the aggregate results, rather than rely only on a leaderboard position.
Notable points
What stands out
The current README says italic, underline, and highlight are evaluated as individual rules but excluded from the headline Semantic Formatting score. Improving one of those rules therefore need not improve that headline number. A separate open upstream report describes bold credit earned through heading promotion; its reported scorer behavior is not a verified claim that a parser understands formatting better.
Before using
What to review
API-based parsing pipelines use the selected provider's key. Evaluation is rule-based by default; the optional chart-normalization judge is a separate, disabled-by-default mode requiring an Anthropic key.
Open report #173 describes shared text IDs dropping content cases from an unfiltered evaluation. Its example distinguishes all-groups evaluation from a separately filtered content group, so reported coverage matters alongside a successful-run count.
Open report #170 describes different formatting results from different MinerU markdown post-processors and asks which renderer produced the published row. These are reporter findings, with reproduction and current fix adoption unverified here.
Check whether the annotated document mix, languages and scored tasks represent the PDFs you intend to process before extending a leaderboard result to your workflow. The benchmark's covered cases do not establish performance on every document type.
Reader fit
Who may find it relevant
Builders comparing document parsers or OCR output for search, extraction and document-reading workflows may find the separate metrics and per-case reports useful. Builders with another parser can register a provider and pipeline through the documented extensions interface. The extension guide says registration happens on import, so the package must be imported before running the CLI. This adds a parser to the existing scoring core without requiring a fork of that core.
Editorial note
Why LifeHubber lists it
We include ParseBench for builders who want to revisit how saved parser output was scored. Its documented re-evaluation path separates another scoring pass from another parser call, so an evaluator change can be examined without generating that output again. The publisher records parse_bench_version in run metadata and evaluation reports; its 1.0.0 changelog says runs made with 0.2.0 or earlier are not directly comparable. Keep that version with archived results when comparing scores.
Source links
Source materials
Reader note
Before relying on this entry
LifeHubber lists entries to help readers inspect AI projects, not to endorse them or prove they are safe, suitable, accurate, maintained, or right for a specific use. We do not verify every entry in depth. Before relying on anything listed, review the original materials, terms, privacy practices, limits, and risks that matter for your situation.
More in Datasets
Keep browsing this category
Explore more datasets.
Terminal-Bench 2.0
harbor-framework/terminal-bench-2
A terminal-agent benchmark for evaluating AI agents on hard containerized command-line tasks, with Harbor run commands, task-level registry pages, GitHub and Hugging Face materials, docs, and paper links.
UltraData-SFT-2605
openbmb/UltraData-SFT-2605
An OpenBMB supervised fine-tuning dataset with 15,036,178 thinking and non-thinking samples across math, code, knowledge, Chinese, instruction-following, and multilingual configurations, used in MiniCPM5-1B-SFT post-training.
olmOCR-bench
allenai/olmOCR-bench
A benchmark for evaluating how well OCR systems convert PDFs into useful markdown while preserving structure.
For project maintainers
Listed here? You can use the badge.
If you maintain a project with a current LifeHubber listing, you may add the optional “Listed on LifeHubber AI Resources” badge to its README, docs, or website. No introduction or permission request is needed.
