ParseBench: Enterprise Document Parsing Benchmark Takes Center Stage

ParseBench is a new, officially‑released benchmark for evaluating document‑parsing systems on real‑world enterprise PDFs. Curated by the LlamaIndex team, the dataset contains roughly 2,000 human‑verified pages from over 1,200 publicly available documents spanning insurance, finance, government and other domains. Each page is annotated with more than 169 K test rules across five capability dimensions—tables, charts, content faithfulness, semantic formatting, and visual grounding—providing fine‑grained diagnostics of where parsers succeed or fail.

The benchmark is organized into five JSONL splits (chart, table, text_content, text_formatting, layout), each delivering rule‑type specific payloads such as `chart_data_point`, `expected_markdown`, and layout bounding‑box annotations. Tags describe document difficulty, modality (PDF, JPG, PNG) and content type (e.g., `ocr`, `multicolumns`, `handwritting`). An accompanying evaluation suite on GitHub implements the published metrics: GTRM for tables, ChartDataPointMatch for charts, rule‑based scores for content faithfulness, semantic formatting checks, and element pass rate for visual grounding.

Because the annotations are produced via a two‑pass pipeline—auto‑labeling by a frontier vision‑language model followed by targeted human correction—the benchmark offers high‑quality ground truth while remaining scalable. Researchers can plug any PDF‑parsing pipeline into the evaluation code to obtain per‑dimension scores and compare against the community leaderboard. The dataset’s Apache‑2.0 license and openly available source documents make it straightforward to integrate into training or fine‑tuning workflows for AI agents that need reliable document understanding.

ParseBench’s multi‑modal, multi‑dimensional design positions it as a reference point for both academic research and production systems that must extract structured data (tables, charts) and preserve nuanced formatting (bold, superscript, LaTeX) from complex enterprise documents.