The Rashi Problem

Source Library uses Gemini Flash to OCR historical texts in dozens of languages. We’ve measured its consistency on Latin and German (1.8% disagreement rate), tested it on hieroglyphs (it can’t) and cuneiform (mixed results). This week we found something more unsettling: a failure mode where the AI produces confident, plausible, contextually appropriate text that has almost no relationship to what’s actually on the page — and our quality metrics gave it a passing grade.

The script is Rashi. The book is the Zohar, Cremona 1558 edition — 548 pages of Kabbalistic commentary in Aramaic, printed in the semi-cursive Hebrew typeface that has been standard for rabbinic literature since the 15th century.

The Discovery

We were reviewing page 8 of the Zohar, a body text page in dense Rashi script. The OCR looked plausible at first glance — Hebrew and Aramaic words, Kabbalistic vocabulary, structured paragraphs. The English translation read like polished Zohar commentary.

Then we compared the OCR against Sefaria’s reference text of the same passage.

The v3 prompt produced garbled but recognizable text. You can see fragments of the real passage — “Israel when they went down to Egypt” is there, but כחתו should be נחתו and בנערה should be בצערא. Letter-level substitutions, the kind you’d expect from a model struggling with an unfamiliar typeface.

The v10 prompt — our newest, most sophisticated OCR prompt, with manuscript detection and calibrated warning thresholds — produced complete gibberish. It misidentified Rashi print as handwriting, triggered the manuscript processing mode, and generated repetitive modern Hebrew words (מכותיו... יתרון... מוסר...) in an incoherent loop. Not a single word from the actual page.

What Makes Rashi Special

Rashi script is not a language. It’s a typeface — the same 27 Hebrew letters rendered in a 15th-century Sephardic semi-cursive style. If you know square Hebrew, you know the same alphabet, but the glyphs look completely different. א (alef) in square Hebrew barely resembles its Rashi counterpart. ד (dalet) and ר (resh), already similar in square script, become nearly identical in Rashi.

This is why Arabic and Sanskrit work fine. Arabic Nastaliq and Devanagari are visually distinct scripts that Gemini has clearly seen in training data. But Rashi script occupies an uncanny valley: it maps to the same Unicode characters as square Hebrew, so the model “knows” Hebrew, but the visual forms are different enough to cause systematic misreadings. The model doesn’t know that it doesn’t know.

The only published study on Rashi OCR — Mahpod & Keller (2018) from Bar-Ilan University — achieved 99.84% letter accuracy using a specialized CNN+LSTM architecture trained on 3.3 million annotated characters from 170 different books. They found that the LSTM (which learns the Aramaic dialect) transfers perfectly between books, but the CNN (which reads the glyphs) needs per-book fine-tuning because every printing house shaped the letters differently. This is a fundamentally different approach from a general-purpose vision model trying to read everything zero-shot.

Rashi script has been used continuously in Jewish printing for over 500 years. The Talmud, Zohar, Midrash, halakhic codes, responsa literature — the core canon of rabbinic Judaism is printed in this typeface. We have 316 Hebrew and Aramaic books in Source Library. Many of them use Rashi script. The quality problem is not confined to one Zohar.

The Quality Metrics Blind Spot

This is the part that concerns us most. We ran semantic alignment scoring on the full 548-page Zohar. The system embeds the OCR and the English translation separately, then computes cosine similarity. Pages where OCR and translation diverge semantically get flagged.

Page 8 — the page we manually verified as significantly garbled — scored 0.884. Not flagged. Comfortably above our Hebrew threshold of 0.80.

The reason is straightforward: semantic alignment measures whether the OCR and translation agree with each other. It does not measure whether either one agrees with what’s on the page. If the model recognizes that this is the Zohar on Parashat Shemot — and it clearly does, given the correct metadata tags — it can generate plausible Zohar content for both the OCR and the translation. The two outputs align beautifully because they were confabulated from the same source knowledge.

This is a known limitation of any internal-consistency metric. CER (Character Error Rate) requires external ground truth by definition. Our alignment scoring was designed for a different failure mode: catching cases where the OCR reads one thing and the translation says another (mistranslation, dropped paragraphs, wrong language). It was never designed to catch cases where both sides of the pipeline are wrong in the same direction. On Rashi script, that’s exactly what happens.

The Audit: What Works

We audited OCR quality across every non-Latin script in the library. The results draw a clear line.

The pattern is clear: Gemini handles genuinely foreign scripts well. Arabic looks nothing like Latin; Devanagari looks nothing like Latin; Chinese looks nothing like anything. The model either reads them or admits it can’t. Rashi script is dangerous precisely because it’s close enough to square Hebrew that the model thinks it’s reading when it’s actually guessing.

Detecting the Problem Without Human Ground Truth

Most OCR evaluation requires human-transcribed ground truth. For 40,000 pages of Aramaic Kabbalistic commentary, that’s not realistic. But there are several automated approaches that don’t require a human reader:

1. Reference text comparison

Many of our Rashi-script books are canonical texts with existing digital transcriptions. The Zohar is on Sefaria. The Talmud is on Sefaria. The Shulchan Aruch, Mishneh Torah, and hundreds of other works have been manually typed and proofread over decades — the Responsa Project at Bar-Ilan University alone has 200 million words.

The approach: embed each page’s OCR text and slide a window across the reference text, finding the best-matching chunk. If the cosine similarity between our OCR and the known-correct transcription falls below a threshold, the page is flagged. This is computationally cheap (one embedding per page, one per reference chunk) and gives a genuine external quality signal — unlike our current OCR-vs-translation alignment, which only measures internal consistency.

2. Repetition detection

The v10 hallucination had a distinctive pattern: the same words repeated in loops (מכותיו... יתרון... מוסר... מכותיו... יתרון...). Real text — even repetitive liturgical text — has a different distribution. A simple unique-word ratio (distinct words / total words) catches the worst hallucinations. The trick is calibrating the threshold: Hebrew religious texts legitimately repeat divine names and key terms, so the bar has to be set lower than for, say, Latin philosophical prose.

3. Multi-temperature consistency

We already have a probe tool that re-OCRs the same page at different temperatures (0, 0.3, 0.6) and measures how much the outputs diverge. When the model is actually reading text, low temperatures converge on the same output. When it’s hallucinating, different temperatures produce different hallucinations. This is expensive (3 additional OCR calls per page) but catches confabulation that reference comparison misses — for the books that don’t have reference texts.

4. Character frequency analysis

Every language has a characteristic letter frequency distribution. Zohar Aramaic should have a different distribution from modern Hebrew. If the OCR output’s character frequencies match modern Hebrew better than Aramaic, something is wrong. This is a lightweight heuristic — not definitive, but a useful first-pass filter over 40,000 pages.

Where We Need Human Ground Truth

Automated methods can flag problems. They cannot fix them. For that we need:

Prompt calibration samples. To improve the OCR prompt for Rashi script, we need a small set of pages (50–100) with verified transcriptions. These don’t need to be manually typed from scratch — Sefaria’s existing digital text, aligned to specific page images, gives us the ground truth. But someone who reads Hebrew needs to verify the alignment, because our page images don’t carry canonical reference coordinates (folio numbers, parasha markers) that would allow automatic mapping.

Script classification. We need to know which books actually use Rashi script vs. square Hebrew. Many books use both (Talmud pages have square Hebrew for the Mishnah, Rashi script for the commentary). A Hebrew reader can classify our 316 books in a few hours. Automated classification is unreliable — our v10 prompt already showed it confuses Rashi print with handwriting.

Post-correction validation. Even after improving the prompt, we’ll need spot-checks to verify the improvement is real. The Mahpod & Keller paper showed that per-book fine-tuning is essential because every printing house shaped the letters differently. A prompt that works on the 1558 Cremona Zohar might fail on the 1560 Mantua Zohar. Human validators keep the feedback loop honest.

What’s Next

We’re building reference text validation into the pipeline — starting with Sefaria as the reference corpus for Zohar, Talmud, and biblical commentaries. Pages that fall below a reference-match threshold will be flagged for review rather than served to readers as if they’re reliable.

We’re also testing explicit Rashi script instructions in the OCR prompt (“This text uses Rashi semi-cursive Hebrew script, not standard square Hebrew”) and evaluating whether DICTA’s Hebrew language models can clean up systematic character substitutions in post-processing.

In the meantime, the honest thing to do is label these pages. If the OCR on a Rashi-script page hasn’t been validated against a reference text, the reader should know. Confident-sounding hallucinations are worse than visible uncertainty.

The broader lesson: internal consistency metrics are necessary but not sufficient. A system that only checks whether its outputs agree with each other will miss the case where the whole pipeline is wrong in the same direction. For well-resourced languages with existing digital corpora, reference comparison is cheap and definitive. For everything else, we need humans in the loop — not to read every page, but to keep the quality signal grounded in reality.

Update: 25 March 2026

We tested explicit Rashi script instructions with Gemini 3 Flash Preview. When the prompt tells the model “This text is printed in Rashi script (semi-cursive Hebrew typeface), NOT standard square Hebrew — read the letterforms accordingly,” the output improves substantially. The model produces recognizable Aramaic text with correct vocabulary, proper abbreviation markers ("ס"א, קב"ה), and coherent paragraph structure. It’s no longer hallucinating.

But “better” is not “good.” Comparing the prompted output against Sefaria’s reference text still reveals word-level substitutions and dropped phrases. The model reads the general sense correctly but gets individual letters wrong — exactly the kind of errors you’d expect from a vision model that hasn’t been trained on this specific typeface. This is a fundamentally harder problem than the complete confabulation we saw before, and one that may require specialized fine-tuning or future model improvements to solve.

Our conclusion: Rashi-script books now display a quality warning banner in the reader. The warning appears on both the OCR and translation tabs, telling readers that current AI models struggle with this typeface and that the output should not be trusted. We’ve set metadata.scriptType: “Rashi” on the Cremona and Mantua Zohar editions, which also triggers rendering in Noto Rashi Hebrew — the correct display typeface.

We are waiting for future model generations with better Rashi support before re-running OCR on these books at scale.

The Zohar page discussed here is viewable in the reader (with Rashi font rendering and quality warning). The full Hebrew OCR quality audit is tracked in GitHub issue #350.