We asked a frontier language model to quote the opening of Cornelius Drebbel’s Tractatus duo de Natura Elementorum (Hamburg, 1621), a rare Latin alchemical treatise we hold. The model produced 680 characters of plausible-looking scholastic Latin beginning “Cum omnia, quae sub Luna sunt…” The actual text begins “Cum discursus hic in manus tuas peruenerit, Amice Lector…” The model invented the opening with complete confidence. This is the puzzle that motivated a small empirical project: for a digital library of pre-modern and multilingual books, how many are genuinely new to the AI ecosystem, and how would we know?
The short answer, after a Bayesian survey of a random sample of 1,000 books drawn from our 16,871 OCR’d volumes: about 43% are confidently new to AI training (~7,240 books, 95% CI 6,715–7,778), and about 21% are confidently already in training (~3,559 books). The expected fraction in training overall is 39%. The full method, validation, and limitations are in a short preprint we’ve put up.
1. Two questions, not one
Asking “is this book in training” quietly conflates two different questions. The first is whether the AI has seen the specific scan, transcription, or printing we hold — the file. The second is whether the AI has encountered the work through some channel: a modern translation, a critical edition, a Wikipedia summary, a scholarly citation. For most pre-modern multilingual books, the answer to the second is what people actually want to know — whether the AI ecosystem already “has” the work in some form — and the answer to the first is approximately unanswerable with black-box probes.
We built our detector to answer the second question carefully and to refuse the first one honestly. A recent skeptical paper showed that the strongest membership-inference attacks on language models collapse under semantics-preserving paraphrase: their signal is overwhelmingly lexical. We don’t fight that. We design around it.
2. A prior, for free
We already maintain a multi-catalogue translation-verification system, described elsewhere on this blog. For each historical work we hold, it searches twelve catalogues (UNESCO Index Translationum, Loeb, Brill, Penguin, Cambridge, Open Library, Google Books, Internet Archive, OpenAlex’s 250-million scholarly records, the Library of Congress live catalogue, and the USTC) and assigns a disposition: confirmed_first (no prior English translation of the underlying work exists), translation_found (multiple English versions exist), and various intermediates.
That disposition turns out to be almost exactly what we need as a Bayesian prior for “has this work been propagated through derived editions an AI is likely to have seen.” A work with no English translation cannot have been encountered in English; a work with Loeb, Penguin, and Oxford translations almost certainly was. We convert the disposition, the translator/publisher list, the IA-scan presence, and the year of the oldest translation into a log-odds prior. On its own, this bibliographic prior agrees with the full Bayesian detector on 88% of books — the rare-book infrastructure we already had is doing most of the work.
3. The behavioural probe
To check whether the bibliographic prior is right, and to refine the answer for books in its uncertain middle, we ask the model directly. Two questions, batched twenty-eight at a time:
- Self-familiarity: Did you encounter this specific book in your training data — yes, no, or unsure? Distinguish knowing of a book from seeing its text.
- Verbatim opening: Quote one sentence from the opening in its original language, or write “unknown.”
We probe two models in parallel: Claude Haiku 4.5 and Gemini 3.1 Flash Lite Preview. The interesting empirical finding is that Haiku is unusually well-calibrated at the long-tail boundary — it says “no” with high reliability on books we know are obscure, and “unsure” rather than confabulating when it doesn’t know. Gemini 3.1 Flash Lite, by contrast, says “yes” to almost everything and produces confident plausible-fabricated text. (That’s the Drebbel story above: Gemini Flash Lite confidently invented Latin; Haiku said it didn’t know.) We treat Haiku’s response as the stronger likelihood signal and Gemini’s only as weak corroboration.
Combining the bibliographic prior with the behavioural probe via Bayes’ rule produces a posterior probability per book, with a credible interval from Monte Carlo. Total cost: about a hundredth of a US cent per book.
4. The catalogue numbers
Random sample of 1,000 books drawn uniformly from the 16,871 books in the library with at least one OCR’d page (the “probe-able” population — books for which any text exists for a model to potentially have seen):
| Posterior P(in training) | Sample % | Across 16,871 books |
|---|---|---|
| Confidently NOT (< 0.10) | 42.9% | ~7,240 books |
| Probably NOT (< 0.25) | 52.2% | ~8,811 books |
| Ambiguous (0.25 – 0.75) | 19.2% | ~3,238 books |
| Probably IN (> 0.75) | 28.6% | ~4,823 books |
| Confidently IN (> 0.90) | 21.1% | ~3,559 books |
| Expected fraction IN training | 39.0% | ~6,586 books |
95% bootstrap CIs in the preprint. The Bayesian posterior is bimodal: most books sit at one end of the spectrum, with only about 19% in the ambiguous middle.
A second analysis on the 10,911 books that have completed bibliographic verification shows just how strongly disposition predicts the answer. Books judged “no prior English translation exists” (n=6,306) receive a mean posterior of 0.10, and 77% are classified confidently new to AI. Books judged “multiple English translations exist” (n=2,866) receive a mean posterior of 0.83, and 69% are classified confidently in training. Almost everything we need to know about a work’s training-data status is in its bibliographic propagation history.
5. What this doesn’t measure
The honest qualifier: this method measures whether the AI recognises the work as a scholarly object — whether some channel of derived editions has reached it — not whether the model has seen our specific scan. We tested the distinction directly. For ten books our detector classified as “in training” and ten classified as “not in training,” we provided eighty characters of the actual OCR as a prefix and asked the model to continue. ROUGE-L of the continuation against the next 300 characters averaged 0.099 for “in training” books and 0.084 for “not in training” books — a ratio of 1.17×, well below any meaningful discrimination threshold. Even on books the model recognises, it cannot reproduce our specific OCR. Recognition is not file exposure, and we make no claim to the second.
We also discovered, in passing, a meaningful calibration regression in the smaller Gemini Flash Lite model relative to its bigger sibling Gemini 2.5 Flash. The Drebbel anecdote is reproducible: the smaller model confidently fabricates content for unfamiliar long-tail works where the larger model declines. We use Flash Lite in our enrichment pipeline (it’s cheaper) for tasks that ground the model in book text, but never for tasks that ask it to recall facts about works from training memory. The calibration of refusal at the long-tail boundary matters more for production reliability than raw capability.
6. Why it matters
For the library specifically, the headline finding is that roughly 7,000 books in our OCR’d corpus are confident additions to what frontier AI knows of the world. Some of these are first translations we’ve commissioned of texts that have never appeared in English. Some are obscure 17th-century treatises that haven’t been scanned anywhere outside our archive. Some are manuscripts with Vatican or Wolfenbüttel shelfmarks whose contents have never been digitised at all. These are the books our work moves into the AI ecosystem; they are what the library’s mission concretely produces.
For the broader question of AI training-data coverage of long-tail scholarly material, this is a small datapoint with a perhaps-unsurprising conclusion: frontier models cover the canon and miss the periphery, and the periphery is enormous — thousands of works per specialist collection. The unsurprising part is the rate. The surprising part is how cheap it is to measure: thirty hundredths of a US cent per book, no logprob access, no fine-tuned probes, just a calibrated language model answering a direct question well enough to be useful when fused with a good prior.
Read the preprint: Recognition vs.\ Exposure: A Bayesian Detector for Long-Tail Multilingual Book Membership in Frontier Language Models (PDF, 5 pages).