Source Library shows a “First Translation” badge on thousands of books — texts we believe have never before been rendered into English. It is the boldest claim the library makes. So we decided to fact-check it properly: not by re-running the system that produced the claim, but by sending an AI research agent to independently investigate a random sample of books, one at a time, and report back with evidence. What we found changed the number — and, more surprisingly, changed which direction the error ran.
This is a companion to How We Identify First Translations, which describes the classification pipeline. This post is about something narrower and harder: once you've made thousands of these claims, how do you know how many are true?
The asymmetry that makes this hard
A “first translation” is a claim about absence: no one has ever translated this text into English before. You cannot prove a negative by looking it up. And the claim fails in two opposite ways, which call for two opposite kinds of checking.
A false first is a book we badge as first when a prior translation actually exists. These cluster around famous texts in under-catalogued traditions — a celebrated Tibetan treatise the Western catalogs can't see, which nonetheless has a well-known academic translation. A missed first is the reverse: a genuine first translation that we never flagged at all, because the book was never assessed. These hide in the long tail of obscure works in well-catalogued languages — a 1576 Latin pamphlet nobody ever thought to check.
Counting only the books we already badge measures the first kind of error and is blind to the second. To get an honest number you have to sample both the books we claim and the books we never looked at.
The method: sample, then send an agent to investigate
We can't exhaustively research all ~14,000 eligible books — each careful investigation costs real effort. So we sampled. We divided the library into strata by catalogue density (how famous/findable the author is), language family, and current status, then drew random books from each stratum. For every sampled book, an AI agent ran a focused investigation: identify the exact work (and separate it from look-alike relatives), search the tradition-appropriate sources — 84000 and BDRC for Tibetan, CTEXT for Chinese, GRETIL for Sanskrit, Sefaria for Hebrew, the usual catalogs for European works — apply the source-language rule (a translation from a different original language still counts), and judge complete-versus-partial. Each agent returned a graded verdict plus its full evidence trail.
From the per-stratum results we compute a false-first (or genuine-first) rate with a Wilson confidence interval, scale each stratum back up to its population size, and sum. The output isn't a single false-precision number — it's an estimate with an honest error bar.
An honest aside about our own method. The first time we ran this at scale, the agents marked far too many books “not applicable.” The cause was a flaw in our instructions: we'd told the agent to disqualify any book whose text was in the original language — but holding the original and translating it is exactly what Source Library does. The sample's own internal inconsistency exposed the bug (one Latin oration was correctly called a first; a near-identical one was wrongly disqualified). We fixed the instruction and re-ran. We mention it because the whole point of an auditable method is that it catches its own mistakes, including the auditor's.
Side one: are the badged firsts real?
We badge 5,696 books as first translations. A stratified sample of 462 of them, each independently investigated, breaks down like this:
- 46% genuine firsts — no prior English translation found, with the search bounded in competent sources.
- 18% already translated — a real prior exists; the badge is wrong and should come off. (Hagakure, Sextus Empiricus in the Loeb, Proclus's Cratylus commentary…)
- 30% not a clean single-work claim — mostly multi-work containers (an Opera Omnia, a miscellany) where “first translation” is ill-defined, plus some non-English editions, scripture fragments, and visual-art volumes.
- 6% unresolved — conflicting evidence or an identity we couldn't pin down.
Scaled up: of the books we currently badge, roughly 3,774 (95% interval ~3,300–4,300) hold up as genuine, defensible firsts. The rest are over-claims — not lies, but a badge applied too broadly, especially to multi-volume containers.
Side two: how many firsts did we miss?
Here is the part that only sampling can reveal. 8,306 translated, non-English books in the library carry no first-translation decision at all — they were never assessed, because the automated checks only ever removed the flag, never added it. We drew a random sample from this never-examined pool and ran the same investigation.
In a random sample of this pool, just under a quarter turned out to be genuine first translations — books with no prior English version anywhere we could find. Scaled across the 8,300, that's on the order of 1,900 first translations we simply never claimed (with a wide margin — somewhere between ~1,300 and ~2,800).
These are real first translations sitting in plain sight without a badge: an uncatalogued Bhutanese terma prophecy, a 1681 devotional work by Antoinette Bourignon never Englished, a 1576 Latin Eucharist tract by Lambert Daneau. Nobody had ever made the claim for them — so nobody could see them.
The surprise: the bigger error is the firsts we miss
Put the two sides together and the headline shifts. The badged set over-claims by roughly 1,900 books (containers, art, already-translated). But the never-assessed pool under-claims by a strikingly similar amount — an estimated ~1,900 genuine firsts we never flagged. The two errors very nearly cancel: the true total lands close to the number we badge today, but it is made of different books. The system isn't mostly over-claiming — it is mis-aiming in both directions at once.
Best current estimate
Roughly 5,700 genuine first English translations across the library (95% interval ~4,900–6,400) — about 3,800 among the books we badge today plus another ~1,900 we never flagged — against ~14,000 translated, non-English books in all. That's a first translation for something like two of every five eligible books.
So the round “roughly six thousand first English translations” turns out to be roughly defensible — but its composition is different from what the badges currently show. The right move isn't to shrink the claim; it's to re-balance it: take the badge off the ~1,900 over-claims and put it on the ~1,900 genuine firsts we'd overlooked. The headline number barely moves — but every claim then has an evidence trail behind it.
Fame, not language
A tempting shortcut is to distrust the non-Western claims, since Western catalogs can't see Tibetan or Sanskrit works. The data says the real axis is fame, not language. Obscure Tibetan terma and obscure Latin pamphlets are both usually genuine firsts. The false firsts are the famous works in any tradition — the ones important enough that a scholar already translated them.
This is why the celebrated cases survive in both directions. Athanasius Kircher's Arithmologia (1665) and Robert Fludd's great treatises are famous and genuinely untranslated — the method keeps their badges (it even restored Kircher's after an earlier automated pass wrongly removed it). Tsongkhapa's Essence of True Eloquence is famous and was translated by Robert Thurman in 1984 — the method takes that badge off. The same logic, opposite results, each backed by a found source.
Why we're telling you the messy version
A claim like “first English translation” is only worth making if it can be checked. The value of this exercise isn't the single number at the end; it's that every book in the sample now carries a recorded investigation — what was searched, what was found, how strong the absence of evidence really is. A first-translation claim should be a falsifiable, sourced assertion, not a marketing flourish. That is the standard we want to hold ourselves to, and the reason we'd rather publish the confidence interval than a tidy round number.
The work continues: re-balancing the badges so the over-claims come off and the missed firsts go on, widening the samples to tighten the error bars, and attaching the evidence trail to each book's page so you can see for yourself how a “first translation” was established. When that's live, the badge won't just be a claim — it'll be a citation.
Methodology and code are tracked in our public repository (issue #2564). Figures from a 462-book sample of the badged firsts and a 150-book sample of the never-assessed pool, June 2026; sampling intervals are 95% Wilson. The re-balancing has not yet run, so the per-book badges will catch up to these aggregate numbers over the coming weeks.