Source Library uses Gemini to read 400-year-old printed books — Latin, German, Dutch, French, Hebrew, Greek. We process over 300,000 pages. But how consistent is the OCR? If you photograph the same physical page twice and run both images through the same model, do you get the same text?
A digitization error gave us a natural experiment. The Early Modern Fiche collection — one of our largest sources — accidentally photographed the same page spreads multiple times during scanning. After our split-detection pipeline divided spreads into individual pages, the duplicates produced separate page records: same physical text, different source photographs. Before cleaning them up, we compared the OCR output character by character.
The result: 1,448 duplicate pairs across 201 books and six languages. To our knowledge, no published study has compared OCR outputs from different photographs of the same physical document. The standard approach in the literature compares OCR against human-transcribed ground truth, or compares multiple OCR engines on the same scan. Our dataset does something different: it measures how much variation the photograph itself introduces into the pipeline.
The Experiment
For each duplicate pair, both pages had been OCR’d by the same model (Gemini 3 Flash) with the same prompt. The only difference was the source image — two separate photographs of the same physical page, taken at different moments during the BPH digitization process.
We computed three metrics for each pair:
- Character-level similarity — Levenshtein edit distance on the stripped text (XML tags removed, whitespace normalized), expressed as 1 − (edits / max_length). This is equivalent to 1 − CER, where CER is the standard Character Error Rate used in OCR evaluation.
- Word-level Jaccard index — intersection over union of word sets, measuring vocabulary overlap regardless of word order.
- Raw similarity — edit distance on the full OCR output including XML markup tags, to detect structural differences in how the model formatted its response.
Neither output is “ground truth” — both are AI-generated. So our metric is inter-scan disagreement rate, not error rate. Where the two outputs agree, we can’t tell whether they’re both right or both wrong in the same way. Where they disagree, at least one is wrong. This makes our disagreement rate a lower bound on the actual error rate: the true CER is at least as high as what we measure, and likely higher.
Results
| Metric | Mean | Median | P5 | P95 |
|---|---|---|---|---|
| Character similarity (stripped) | 91.9% | 98.2% | 28.7% | 100.0% |
| Raw similarity (with XML) | 92.9% | 98.2% | — | — |
| Word Jaccard index | 88.3% | 96.3% | — | — |
The gap between mean (91.9%) and median (98.2%) tells the story: most pages are highly consistent, but a long tail of outliers pulls the average down. The P5 value of 28.7% reflects pages where the model classified the content differently between scans — one scan producing “blank” and the other a description of the physical page.
Distribution
| Similarity band | Pairs | Share |
|---|---|---|
| Exact match (100%) | 286 | 19.8% |
| Near match (>99%) | 235 | 16.2% |
| High similarity (>95%) | 615 | 42.5% |
| Moderate (>80%) | 180 | 12.4% |
| Low (≤80%) | 132 | 9.1% |
78.5% of pairs are above 95% character similarity. One in five pairs are exact character-for-character matches. The low-similarity tail (≤80%) consists almost entirely of blank pages, endpapers, and binding photographs where the model’s page-type classification varied between scans — one producing a terse “blank” tag, the other a physical description of the paper.
By Language
The dataset spans six major languages of early modern European print. Sample sizes vary because the EFM collection skews toward Latin, German, and Dutch theology.
| Language | Pairs | Mean | Median |
|---|---|---|---|
| Latin | 461 | 91.7% | 97.9% |
| German | 265 | 93.8% | 98.2% |
| Dutch | 246 | 91.8% | 98.1% |
| French | 244 | 92.6% | 98.8% |
| English | 127 | 92.0% | 97.9% |
| Italian | 48 | 88.1% | 98.6% |
| Greek | 15 | 65.7% | 91.2% |
| Hebrew | 4 | 52.3% | 52.3% |
The Western European languages cluster tightly: median similarity between 97.9% and 98.8%, with no statistically meaningful differences between Latin, German, Dutch, French, English, and Italian. This is notable because the texts span four centuries (1500s–1800s) and include blackletter/Fraktur (German), italic type (Latin), and early Roman type (French/Dutch).
Greek and Hebrew show lower consistency, though both have very small sample sizes (15 and 4 pairs). Greek early modern typography uses extensive ligatures and abbreviations that may amplify photographic variation. The Hebrew result should not be generalized from four data points.
Comparison with Published Work
The standard metric in OCR evaluation is the Character Error Rate (CER): the Levenshtein edit distance between OCR output and a human-transcribed ground truth, divided by the length of the ground truth. Our experiment doesn’t measure CER in this sense — we have no ground truth, only two AI-generated outputs. What we measure is inter-scan character disagreement, which sets a lower bound on the true CER.
The most directly comparable published result is Stoyanov et al. (2025), “Evaluating LLMs for Historical Document OCR” (arXiv:2510.06743), which tested Gemini 2.5 Pro on 18th-century Russian printed text and measured a CER of 3.36% with a coefficient of variation of 0.037 — the most stable of all LLMs tested. Our median inter-scan disagreement of 1.8% is lower than Stoyanov’s CER, which makes sense: inter-scan disagreement is a lower bound on error, not the full error.
Hiippala et al. (2025) found that LLMs consistently outperform traditional OCR engines (Tesseract, Kraken) on historical documents, with GPT-4o and Gemini achieving the lowest error rates (arXiv:2501.11623). Typical CER for neural OCR on historical text ranges from 3–10%; for incunabula and pre-1600 printing, 15%+ is common with older engines.
What appears to be novel about our dataset is the experimental design. Published OCR evaluations compare output against ground truth (accuracy), or compare multiple engines on the same image (inter-engine agreement). The closest methodological precedent is multiple-system voting, where several OCR engines process the same scan and disagreements are flagged for review. Our experiment varies the input instead of the engine: same model, same prompt, different photographs of the same physical page. This isolates the contribution of photographic variation to OCR inconsistency.
What the Outliers Tell Us
The most interesting findings are in the tails. The 132 low-similarity pairs (≤80%) break down into a few categories:
Blank page classification. The largest category. When two scans of a blank or near-blank page (endpaper, binding, flyleaf) are OCR’d, one scan might produce a minimal “blank” tag while the other produces a paragraph describing the physical characteristics of the paper. Both responses are arguably correct — but they have very low character overlap. This is a classification instability, not a reading error.
Marginal annotations. When a page has handwritten marginalia, slight differences in the photograph can cause the model to include or exclude them. The primary printed text matches, but the overall similarity drops because one output contains margin text the other doesn’t.
True reading differences. A small number of pairs show genuine disagreement on printed characters — typically in damaged or faded sections where the type is barely legible. These are the most informative outliers: they mark the boundary of the model’s visual capability on degraded historical printing.
What We Did About It
The duplicate pages were a data quality problem as well as an experiment. Duplicate source scans meant duplicate page records — readers saw the same text twice, page counts were inflated, and wasted API cost on redundant OCR and translation.
We built a detection pipeline using server-side MongoDB aggregation: fingerprint the first 400 characters of each page’s OCR text, group by fingerprint within each book, flag groups with two or more pages. This identified 1,568 duplicate pages across 205 books. Before removing them, we archived every duplicate to a duplicate_pages collection preserving the full page record, original page ID, and archive timestamp. Then we deleted the duplicates from the main collection, renumbered the remaining pages sequentially, and updated book-level page counts.
The archived duplicates are what made this experiment possible. By comparing each archived page’s OCR against the corresponding kept page, we got 1,448 natural A/B comparisons without needing to re-run any OCR.
Limitations
No ground truth. We measure disagreement, not error. When two outputs agree, they could both be wrong identically — for instance, both misreading the same damaged character the same way. Our 1.8% median disagreement is a floor, not a ceiling, for actual OCR error.
Single source collection. All duplicates come from the EFM/BPH digitization. This means all photographs were taken with the same equipment, under similar conditions. Comparing scans from different institutions (e.g., BPH vs. Bayerische Staatsbibliothek) would likely show higher variation.
Non-random sample. The duplicates are an accident of the scanning process, not a designed experiment. The distribution of page types, damage levels, and languages reflects whatever BPH happened to scan twice, not a controlled sample.
Small samples for some languages. Hebrew (4 pairs) and Greek (15 pairs) are too small for reliable language-level conclusions. The Western European languages (120+ pairs each) are more robust.
Conclusion
For the core task — reading 16th–18th century Latin, German, Dutch, French, English, and Italian printed text — Gemini 3 Flash is highly consistent across different photographs of the same page. The median inter-scan disagreement of 1.8% means that if you photograph the same printed page twice under similar conditions, 98.2% of characters will be identical in the OCR output. One in five pages produces an exact character-for-character match.
The main source of inconsistency is not character-level reading errors but page-level classification: whether the model treats a blank page as worth describing or not, whether it includes marginalia, whether it classifies an endpaper as a text page. These are judgment calls, not mistakes, and they account for most of the long tail below 80% similarity.
For a library processing hundreds of thousands of pages of historical text, this level of consistency is sufficient. The pipeline produces stable, reproducible transcriptions. And the duplicate scans — an accident of digitization — gave us a way to verify that claim at scale.
Technical details: Model: Gemini 3 Flash Preview, Standard OCR prompt v5.2026-02. 1,448 pairs across 201 books from the Early Modern Fiche collection. Metrics: Levenshtein edit distance (character-level), Jaccard index (word-level). All duplicate pages archived to duplicate_pages collection with full provenance. Detection and comparison scripts available on request.