What Makes a Good Scan?

Source Library holds roughly 17,000 rare books and over 100,000 extracted illustrations — emblems, woodcuts, engravings, manuscript figures, maps. The text in those books is recoverable: OCR fails, we re-OCR, eventually we get it right. The pictures are not. A faded engraving, a smeared woodcut, a microfilmed map — those are permanent losses, frozen in whatever quality the digitization captured. So a question we kept dodging: which of our scans are good enough, and which need to be re-sourced?

The honest answer used to be “we don’t really know.” A legacy script (audit-scan-quality.mjs) had assigned a quality score to about 4% of the visible library — pixel statistics over two sampled pages per book, condensed into a single 0–100 number. The number was easy to query. It was also wrong, often by 30 or 40 points, in ways we only noticed when we started spot-checking it. This note is about what happens when you do.

1. Why images, not text

A previous note on this blog (The Confident Hallucinator) made the case that OCR confidence and OCR agreement together give us a reliable signal for text quality. If a page is unreadable today, we can run it again next year with a better model and recover the text. Text is reversible.

Pictures are not. The reason is mechanical: every digitization pipeline ends with a JPEG (or a JP2) sitting on a server somewhere, and that file is all anyone will ever see of the underlying object. If the original photographer compressed it too hard, or scanned it from microfilm instead of from the page, or thresholded it to bitonal at 200 DPI, the loss is baked in. You can’t un-microfilm an engraving by running better software over it.

That makes scan quality the most consequential property of an image-bearing book, and the one we’ve been least equipped to talk about. So we started measuring.

2. What pixels can see

The first attempt was deterministic. For each of two sampled pages in a book, compute four numbers and combine them into a score:

• Color depth. Is the file color, grayscale, or bitonal black-and-white? Bonus points for color.
• Resolution. Megapixels. Bonus points for sharper.
• Dynamic range. The spread between the darkest and lightest pixel. Detects washed-out or faded scans.
• Contrast. Mean standard deviation across channels. Detects flat or uniform images.

This is roughly the right vocabulary. A high-resolution color photograph with strong dynamic range is almost always a great scan. A 1-megapixel bitonal page with a brightness range of 80 is almost always a terrible one. In the middle of the distribution — where most books actually live — pixel statistics work fine.

At the edges they fall apart.

3. What pixels miss

We pulled a stratified sample — eighteen books, spread across the score range — and looked at each one. Three failure modes were obvious within minutes.

Failure 1: a blank page scored 30/100

A book called Ganita Kaumudi, Part 1 had a quality score of 30. Page 157, one of the two pages the formula had sampled, looked like this:

The page is white. Not faded. Not light. White. But the formula handed it 25 points for being 14.8 megapixels and 5 more points for some residual scoring floor, and gave it a 30 — a score that put it in roughly the same bucket as a perfectly readable bitonal woodcut from 1515. The pixel statistics flagged that the image was degenerate (dynamic range 0, contrast 0) but the formula didn’t know what to do with the flag.

Failure 2: a beautiful map scored 74/100

A 19th-century fold-out map from the e-rara collection — Die Schlacht bei Austerlitz — scored 74. It is, plainly, an excellent scan: sharp color, every village name legible, the paper’s creases preserved as natural texture.

What dragged the number down was resolution — the map is only 1.3 megapixels. The formula penalized it 15 points for being small. But the map is also sharp enough that every line and label is crisp at native size; the resolution doesn’t hurt the image, it just isn’t enormous. Sharpness, not megapixel count, is what matters for a well-rendered illustration. The formula didn’t measure sharpness.

Failure 3: microfilm scored 84/100

The opposite failure: a 17-megapixel scan of George of Trebizond’s De Platonicae atque Aristotelicae Philosophiae Differentia earned 84 points and looked, statistically, like a high-quality archival capture. It wasn’t. The pixel count was real, but the underlying source was microfilm — what you see when you zoom in is the signature pepper noise and jagged character edges of a high-resolution scan of an old microfilm reel. The illustrations had been destroyed long before the JPEG existed.

Pixel statistics cannot detect this. A microfilm scan at 17 MP has the same megapixel count, the same dynamic range (the whites are still bright and the blacks are still dark), and the same contrast (high, because everything is one or the other) as a pristine archival photograph. The difference is in the texture — in what fills the spaces between the characters — and that’s a perceptual judgment, not a numeric one.

4. The flash-lite hallucination

So we did the obvious thing: asked an AI to look at the images. The right tool here is Gemini Flash — the same vision model we already use for OCR and illustration extraction. We ran the eleven sample pages through Flash with a blind prompt asking it to rate scan quality, and the results were a clear improvement. The microfilm scan dropped from 84 to 58 (“heavy bleed-through, low resolution, scan-of-scan artifacts”). The Austerlitz map climbed from 74 to 95 (“outstanding high-resolution scan with sharp detail”). The blank page got a clean 0.

Then we tried the cheaper sibling, Flash Lite, which costs roughly half as much per call. Because we’d be running this at corpus scale — 100,000 illustrations — a 50% cost difference matters. We expected slightly worse judgments but a similar shape of answer.

What we got was different. The same blank page that Flash had scored 0 came back from Flash Lite as:

scan_score: 95

classification: grayscale

readable: true

verdict: “This is an excellent, high-resolution scan with sharp text and clear contrast, making it perfect for both reading and OCR processing.”

There is no text in the image. There is nothing in the image. It is white. Flash Lite read the file’s metadata, decided what a high-resolution archival scan should look like, and wrote a confident description of one. The hallucination wasn’t subtle — it was a complete fabrication, marked “readable: true,” of content that did not exist.

That single failure mode is disqualifying for production use. We need this system specifically to catch blank pages, broken scans, scanner-bed artifacts — the cases at the bottom of the quality distribution. A model that rubber-stamps those as “excellent” doesn’t reduce error, it adds an extra layer of false confidence. Flash, at twice the price, doesn’t do that. We’ll pay the extra.

The pattern is the same one The Confident Hallucinator surfaced for OCR: consistency alone is a dangerous signal, and the cheaper model produces a more consistent kind of wrong. Vision tasks have the same shape.

5. A three-layer design

The synthesis is to use both signals together. Pixel statistics are cheap, deterministic, and reliable at the obvious cases — a dynamic range of zero is a blank page, every time. AI judgment is expensive, fuzzier, but able to make the perceptual distinctions that numbers can’t — bitonal-from-microfilm versus bitonal-from-woodcut, fragment-versus-full-page, scanner-bed-versus-page-content. Neither alone is sufficient. Both together cover the failure modes of either.

So the production design is three layers, embedded in the existing illustration-extraction pipeline:

We picked the extraction pipeline as the host because it’s already doing the work. The worker already downloads every page image, already calls Gemini vision, already writes one document per detected illustration. Augmenting the prompt is a 15-line change. Computing pixel statistics on the in-memory buffer is another 30 lines. We’re capturing technical-quality signals during work that’s already happening.

The result is two parallel signals per illustration. The existing gallery_quality field answers “is this image worth showing?” — an emblem with figures scores higher than a marbled endpaper. The new scan_quality field answers “how cleanly is this image digitized?” — orthogonal questions that we’d been muddling together. A famous Kircher diagram has a gallery_quality of 0.9 whether the scan is pristine or microfilmed. Now we can say so.

6. Bitonal clean vs. microfilm

The hardest distinction the system has to make is between a pristine bitonal scan of a woodcut — sharp, intentional, no information loss — and a bitonal scan from a microfilm reel, which looks superficially similar but has lost the original’s fine detail. Both are pure black-and-white. Both can be high-resolution. Pixel statistics see them as equivalent.

Two examples from the test set. On the left, a clean 1515 woodcut of Ramon Llull and his disciples at the shore; on the right, a microfilm scan of an early printed page from Steganographia. Both are bitonal. Both score similarly on pixel statistics (bimodality 0.52 vs 0.73, histogram entropy 4.82 vs 2.68 — close, but no firm threshold). Visually, only one of them is preserved.

Gemini handles this. In the validation set it correctly tagged the Steganographia scan as bitonal_microfilm with concerns “pepper_noise, scan_of_scan, low_resolution, compression_artifacts” — the exact technical vocabulary an archivist would use. The Llull woodcut was tagged bitonal_clean with concerns just “binding gutter shadow.” The distinction the pixel formula couldn’t make, the model made cleanly.

This is the single most useful field in the schema. A book whose illustrations are classified bitonal_microfilm or microfiche is a re-source candidate: somewhere out there is a better copy of those engravings, and we should track it down. A book whose illustrations are bitonal_clean at any resolution is fine. The classification cuts directly to action.

7. What it unlocks

Three uses immediately follow.

Best-copy duplicate picking. Source Library frequently holds two or three editions of the same work, sometimes from different providers. When dedupe runs, the question is which copy to keep canonical and which to hide. Until now the tiebreaker was OCR completeness — how much text was processed. That’s a text-availability heuristic, not a quality one. With per-illustration scan_quality, we can compare specific images across editions: of the two copies of Musaeum hermeticum, which one renders the Mercurius engraving sharply? Pick that edition. The other is the duplicate.

Concern flagging. An admin queue filtered by scan_quality.has_blank_pages, scan_quality.page_completeness_issues, and the high-curatorial-value / low-scan-quality combination (the worst-case: important illustrations badly digitized). These are actionable rows — each one corresponds to a specific intervention. Re-source. Re-archive. Manual inspection.

Provider quality baselines. Aggregate scan_class by image_source.provider. Some providers will turn out to be microfilm-heavy. Others will be color-photo-heavy. We’ll know which collections to prefer when multiple sources are available for a single work, and we’ll be able to ground future import decisions in data rather than habit.

8. What we’re still unsure about

Two things, mostly.

Granularity. The current design rates quality at the page level and inherits to each illustration on that page. But pages aren’t uniform — you can have a beautiful color frontispiece bound facing a faded text page, and they’ll get the same score. For high-value books we may want per-illustration Gemini calls; for everything else, page-level is fine. The right policy is probably tier-based, but we don’t know yet where the threshold should sit.

The microfilm threshold. Gemini occasionally classifies a clean modern bitonal scan as bitonal_microfilm when the source is, in fact, a sharp woodcut printed on yellowed paper. The model is being slightly conservative — treating any jagged edge as a microfilm signature. We could tune the prompt to be less aggressive, but the cost of getting it wrong in the other direction (silently labeling actual microfilm as “clean”) is much higher than a few false positives. For now we’re accepting the conservative behavior and reviewing the borderline cases manually.

The longer-term question is whether any of this is necessary. If we end up with a clean labeled corpus of 50,000 illustrations — pixel characteristics on one side, Gemini judgments on the other — that corpus is itself a small ML training set. A distilled classifier that runs on the deterministic features alone, calibrated against the Gemini labels, would let us re-score the entire library in minutes at no per-call cost. We’re not there yet. But it’s the kind of thing that becomes easy once you have the data, and that’s the most under-appreciated outcome of building a system like this. The schema and the corpus are the durable artifacts. The current scoring algorithm is just a placeholder.

The full design document (with schema, prompt, characteristics formulas, validation table, and the full eleven-sample spot-check) lives at .claude/docs/automated-image-quality-system.md in the repository. Phase 1 is tracked at GitHub issue #1810.