Scanned PDF Benchmark: How Did 10 Parsers Handle Graduate Entrance Exams?

🌏 中文版

The previous post discussed how to choose tools for the parsing layer: check licensing first, then corpus type, and benchmark scores last. Those were other people's benchmarks. This time I ran the tools myself on material I already had—scanned NTU graduate entrance exams.

Test Material

I used four exams. Every one was an image-only scanned PDF with no extractable text layer:

• Computer Science Algorithms, 2019: code mixed with formulas
• Information Management IT, 2024: multiple-choice questions mixing Chinese and English
• Information Management English, 2026: English reading comprehension
• Computer Science Mathematics, 2025: dense formulas and diagrams, including directed graphs and a heap tree

The set deliberately spans plain text, mixed-language text, code, formulas, and diagrams. The mathematics exam is the hardest; every tool performed noticeably worse on it.

The Result Up Front

Full Comparison

* Licensing caveats: Marker's model weights require payment above a revenue threshold—the $2M–$5M range is disputed; see the previous analysis. MinerU requires separate licensing and disclosure above $20M in monthly revenue. Docling uses a clean MIT license.

olmOCR, Chandra-OCR, and dots.ocr require a GPU or vLLM backend, so I could not test them locally on a CPU Mac.

The Litmus Test: Question 3, the Min-heap

Docling produced completely garbled text for this question, making it the clearest test of tool quality.

Firecrawl / MinerU 3.4 (perfect reconstruction):

3. (10%) Draw the final min-heap tree after the following operations:
insert 7, insert 4, insert 3, insert 1, delete min, insert 9,
insert 2, insert 5, delete min, delete min.

Docling (garbled):

'In oor  t  se Dmt e t ote  ie  s  ote  te f t e t t t t rt ts
delete min, delete min.

Marker v2 and RapidOCR both recognized the text correctly. Marker additionally preserved the LaTeX formatting.

Mathematical Formulas: The VLM Divide

The 2025 computer science mathematics exam was the hardest document for every tool. Firecrawl's VLM recovered complete LaTeX for matrices and vectors, including \begin{pmatrix}. MinerU produced 5,752 characters with occasional formula noise. Marker fell to 3,143 characters despite exceeding 7,000 on every other exam, while PaddleOCR returned only 1,865.

Conventional OCR tools such as RapidOCR and PaddleOCR flatten formulas into plain text—$a_n$ becomes an, and matrices disappear. They remain entirely adequate for plain prose and are 10 to 100 times faster.

Only Claude vision could actually “understand” diagrams such as directed graphs and heap trees. Every other tool merely recognized labels around the diagram without recovering its semantics.

Installation Pitfalls

MinerU: The Package Was Renamed, and the Old Name Leads to Dependency Hell

The magic-pdf package on PyPI, version 1.3.12, is obsolete. OCR model versions v4/v5 in its PDF-Extract-Kit-1.0 dependency do not match the v3 models expected by the code. Combined with conflicting transformers versions and incompatible tokenizer formats, this creates an unsatisfiable dependency chain.

The correct command is pip install -U "mineru[all]". The package is now named mineru; the current version is 3.4.5, and the official project has fully migrated.

Marker v2: The First Test Returned 0 Characters

This was a reading bug in my test code. result.markdown did contain 7,331 characters, including LaTeX. The model takes about 10 minutes to download on first use because of its surya-ocr dependency; installation itself is pip install marker-pdf.

PaddleOCR v3.7: The API Changed

It requires the separate paddlepaddle engine (pip install paddlepaddle). The API changed from ocr() to predict(), and the return structure changed as well.

Surya 0.22: Major API Changes

Use LayoutPredictor, not DetectionPredictor, together with RecognitionPredictor. Results moved from text_lines to blocks, which includes HTML output. I encountered intermittent connection errors; using Surya indirectly through Marker v2 was the most reliable path.

dots.ocr: Requires a vLLM Backend

The PyPI package is dots_ocr, installed from GitHub with pip install git+https://github.com/studio-dots-ai/dots.ocr.git. It requires a vLLM backend and cannot run in a CPU-only environment.

Tesseract: 0 Characters

Tesseract returned no characters for these scanned PDFs, possibly because of scan resolution or missing image preprocessing.

Why Did VLM-based Tools Win?

Firecrawl first uses pdf-inspector, written in Rust, to classify a PDF as scanned or text-based in 2ms and route it accordingly. Scanned PDFs go to a Vision Language Model rather than conventional OCR. Its output contains LaTeX such as $b_n$, $O(n)$, and \begin{pmatrix}, confirming the distinction: conventional OCR cannot generate this formatting.

Marker v2 embeds Surya OCR, which is VLM-based, in its pipeline and can therefore emit LaTeX. MinerU 3.4 uses UniMERNet for formula recognition, another model-inference approach.

This confirms an observation from the previous post: the boundary between pipeline systems and end-to-end VLMs is becoming less distinct.

Choosing by Scenario

A Two-stage Validation Strategy

After validating 61 past exams, this strategy offered the best trade-off:

Stage one, screening: run RapidOCR over all 61 documents in about five minutes. Compare them automatically against the question-bank JSON and flag mismatches.

Stage two, close inspection: run full MinerU parsing only on the 18 documents flagged during screening, then compare and correct each question.

The result: 50 documents passed directly and 10 required reconstruction—three contained the wrong subject, five came from the wrong year, and two failed the OCR comparison. All were repaired, covering 1,449 questions in total.

Mapping Back to the Three-layer Ladder

Returning to the framework from the first post in the series:

Every scanned exam belongs in the parsing layer; there is no shortcut. The point of the two-stage strategy is to use a cheap tool, RapidOCR, to decide which files need the full parsing stack rather than applying the most expensive method to every document.

Overall

The technology has converged to the point where every serious option is usable. The meaningful differences are speed, cost, and installation burden. Setup problems—MinerU's rename, Marker's model download, PaddleOCR's missing engine, and Surya's cross-version incompatibility—consumed more time than the quality gaps among the tools. Only RapidOCR and Docling worked immediately after pip install.

If you remember one thing, remember this: choose the strategy before the tool. The two-stage approach reduced processing for 61 exams from “30 minutes of MinerU for everything” to “five minutes of RapidOCR plus close inspection of 18 documents with MinerU.”

References

• mendableai/firecrawl — GitHub
• opendatalab/MinerU — GitHub
• datalab-to/marker — GitHub
• datalab-to/surya — GitHub
• docling-project/docling — GitHub
• PaddlePaddle/PaddleOCR — GitHub
• RapidAI/RapidOCR — GitHub
• allenai/olmocr — GitHub
• datalab-to/chandra — GitHub
• studio-dots-ai/dots.ocr — GitHub
• firecrawl/pdf-inspector — GitHub
• The Parsing Layer: When Structure Must Be Inferred—and Licensing Becomes the Real Selection Axis
• The Three-layer Ladder of Document Parsing: Conversion, Extraction, and Parsing
• The Deterministic Extraction Layer: Solve 80% of PDFs Without a Model