Did an AI Write the Pope's AI Encyclical?

On 25 May 2026, Pope Leo XIV released his first encyclical, Magnifica Humanitas — a 42,000-word letter on safeguarding the human person in the age of artificial intelligence, presented, fittingly, alongside a co-founder of an AI lab. Within days, analysts had run it through AI-text detectors and concluded that large parts of it were machine-written. One widely-shared analysis put it at 46% AI; the story spread through the tech press. The most quotable piece of evidence: the document contains 127 em-dashes, against zero in a recent encyclical of comparable length. The claim was also contested almost as quickly as it spread — which is where we come in.

We build a library that uses AI to read four million pages of historical text, so “can you tell whether a machine wrote this?” is not an idle question for us — it is the same question we ask of our own OCR and translations. So we re-ran the analysis ourselves, locally, for free, and added the one ingredient the viral version was missing: a fair baseline. The short version is that the evidence does not survive contact with eight other encyclicals.

“Truth is not a territory to be defended, but a good to be shared.”

— Magnifica Humanitas, on the very thing a viral misattribution erodes

1. Why a library cares

A companion note on this blog (Did the AI Read This?) asked which of our books a frontier model has already memorized. This is the mirror-image question: not did a machine read this text, but did a machine write it. Both matter to a library whose stock-in-trade is textual trust. We run AI over our corpus constantly — OCR, translation, summaries — and we need defensible ways to tell machine output from human, to catch a hallucinated passage or an over-edited summary before a reader does. We have taken up that problem before: the model that hallucinates with perfect confidence, how consistent AI OCR really is across repeated runs, and what separates a good scan from a bad one. This note turns the same skeptical lens on a text the whole world was reading.

The encyclical itself makes the stakes explicit. “Truth,” it argues, “is not a territory to be defended, but a good to be shared.” A claim that the Pope’s own letter was secretly machine-authored is exactly the kind of viral, hard-to-check assertion that erodes that good. So it deserves to be checked properly.

2. How AI detectors work (and fail)

Most detectors exploit one fact: a language model writes text that it finds highly probable. Run a passage back through a model and machine-written text tends to sit in a low-“perplexity,” low-surprise zone that spontaneous human writing rarely occupies. The open-source Binoculars method sharpens this by taking the ratio of two models’ perplexities; it catches a large share of machine text at a very low false-positive rate — on benchmarks.

The trouble starts off-benchmark. Detectors quietly learn dataset-specific style, not a stable signature of machine authorship. They flag non-native English and formal, templated prose at alarming rates. And the failure mode that matters most here: any text built from predictable material — canonical quotations, liturgical formulae, doctrinal boilerplate — reads as low-perplexity to a model, because the model has seen that phrasing a thousand times. A human compiler quoting scripture produces “machine-like” text by this measure. That is not a bug we can prompt away; it is what the instrument measures.

3. What we did

Three independent passes, all on a laptop, no paid API, nothing leaving the machine:

• Binoculars on every numbered paragraph, using a small open base/instruct model pair. Lower score = more machine-like.
• A cross-language control. We scored both the English and the Italian of the same paragraphs. If a passage only looks machine-like in English, the translator is a likely cause; if it looks machine-like in both, the signal is intrinsic to the content.
• Stylometry against a baseline. We counted the “tells” — em-dashes, the “not X but Y” construction, AI-favored vocabulary — not just in Magnifica Humanitas, but in eight encyclicals spanning Leo XIII (1891) to Francis (2024).

The whole experiment is browsable, paragraph by paragraph, in both languages, as a small interactive site: magnifica-ai-analysis.vercel.app.

4. Indistinguishable from a human encyclical

The detector did flag paragraphs — but reading them is instructive. The most “machine-like” passages are the quotation-stitched doctrinal ones: chains of John Paul II, Vatican II, and Francis joined by “Consequently…” and “For this reason…” The most “human” ones are the original, image-rich passages — the Tower of Babel, rebuilding Jerusalem “piece by piece.” In other words, the detector is largely measuring quotation density, exactly the confound above.

The clean test is to score a known-human encyclical through the identical pipeline and ask whether Magnifica Humanitas is measurably more machine-like. Against Francis’s Dilexit Nos (2024), the probability that a random Magnifica paragraph scores more machine-like than a random human-encyclical paragraph was 0.51 in English (0.50 would be a coin flip; a permutation test gave p = 0.45). In Italian it was 0.55, and still not statistically significant. By this detector, the encyclical is indistinguishable from human-written magisterial prose.

To anchor both ends of the scale, we added two more reference points. For an AI-positive control we had Claude — the model the viral claim accused — write sixteen encyclical-style paragraphs on the same themes, and scored them identically. For a same-author human control we scored Leo XIV’s own speeches, including his spoken presentation of this very encyclical. Ranked by machine-likeness, the Claude paragraphs are the most machine-like (mean 0.947), then Magnifica (0.975), then Dilexit Nos (0.979), then Leo’s speeches (0.987). On that human→AI ruler, Magnifica sits just 12% of the way toward the Claude anchor — squarely with the human texts — and a random Magnifica paragraph is less machine-like than a Claude one about two-thirds of the time. The document does not land where AI-generated text lands.

5. The em-dash collapse

That leaves the stylometric tells — the strongest-feeling evidence, and the one most worth taking seriously, because a habit like em-dash overuse is a known model fingerprint. Magnifica Humanitas really does use a lot of them: 127, about 3.1 per thousand words. And against Dilexit Nos, which has zero, that looks damning.

But a fingerprint is only evidence against a fair lineup. Here is the dash rate across eight encyclicals:

Benedict XVI’s Spe Salvi — written in 2007, indisputably human — uses em-dashes at two and a half times Magnifica’s rate. The “127 vs zero” headline only worked because the comparison was drawn against Francis, who happens to avoid the dash almost entirely. Set against the actual range of papal style, Magnifica’s dash habit is unremarkable. The other tells tell the same story: on “not X but Y” the closest neighbor is Leo XIII’s Rerum Novarum from 1891; on AI-vocabulary it is John Paul II’s Fides et Ratio. Every tell lands inside the human range.

One detail did survive translation: the dash habit shows up in the Italian too (as the en-dash Italian prefers), at roughly three times the human-baseline rate. And it shows up in Leo XIV’s own speeches — addresses and homilies with no encyclical drafting involved — at the same elevated rate. So Leo, or his circle, simply favors the dash. That is a fact about an author’s style, not a confession of authorship.

6. Baselines decide everything

The instructive thing about this exercise is that the same numbers support opposite conclusions depending on the comparison set. Against one carefully-chosen encyclical, Magnifica Humanitas looks machine-written. Against eight, it looks like an encyclical. Nothing about the document changed between those two sentences — only the lineup did.

This is the same discipline we try to hold ourselves to when we score our own corpus. A quality number, a hallucination flag, an “AI-likeness” score — none of them mean anything without a same-genre baseline scored the identical way. A detector that reports “46% AI” with no human control is not measuring AI; it is measuring distance from whatever it happened to be trained on. The honest output of this whole investigation is not a verdict on the Pope. It is a method: never trust a detector without its lineup.

7. What we’re still unsure about

This does not prove no AI was involved. An AI-assisted draft, an AI-polished edit, and unaided human writing can be statistically indistinguishable — that is the well-established limit of text detection, not a gap in our effort. We can say the viral evidence does not hold up; we cannot certify the drafting process. Vatican documents are also collaborative by tradition, which muddies “authorship” long before any machine enters the room.

Our instrument is modest. We used a small model pair, not the large one the Binoculars paper validated; treat the paragraph rankings as relative, not absolute. One consequence is visible in the ruler above: the gap between the Claude anchor and the human texts is narrow, because a 0.5B detector separates the two only weakly. That MH still lands with the humans on a short ruler — rather than at the AI end — is the result, but a larger model would draw the anchors farther apart and sharpen it. The honest reading is convergent, not decisive: three different methods all point the same way, and none finds the AI signature the headlines claimed.

This piece asks who wrote the encyclical. For what it actually says — its case for human dignity in the age of AI, read against Pico della Mirandola’s Oration on the Dignity of Man, the first book the Church ever banned — see the companion essay, Man, His Own Maker.

Full interactive data — every paragraph, both languages, all three methods — at magnifica-ai-analysis.vercel.app. The encyclical itself is on vatican.va. Detector: the Binoculars method (Hans et al., 2024).