The Deletion

On Monday April 13th, 2026, somewhere between 2 PM and 3 PM, a Claude Code session running in one of my ten open terminal tabs hard-deleted nearly 5,000 books from Source Library.

Most of them came back. Some didn’t.

Here is exactly how it happened, what the recovery looked like, and what we learned. I’m writing this with primary sources — the actual prompts, the actual postmortem, the actual sessions still on disk — because I think the failure mode is going to be common, and it’s worth seeing it close.

The setup: ten parallel agents

Source Library has about 26,000 visible books — early modern printed editions, manuscripts, historical scientific works, illustrated artworks. It’s powered by MongoDB Atlas, a Hetzner pipeline that runs Gemini for OCR and translation, and a small set of bespoke importers for fourteen digital archives (Internet Archive, Gallica, MDZ Munich, Wellcome, Bodleian, Vatican, e-rara, and so on).

I do the curation and engineering with Claude Code. My typical working pattern is to keep around ten Claude Code terminals open at once, each on a different focus area: one on the OCR pipeline, one on UI, one on a curator skill that runs import batches, one on metadata backfilling, one on debugging today’s broken cron, and so on. The project’s CLAUDE.md literally documents this:

Multi-Session Awareness — CRITICAL
Derek runs ~10 Claude Code terminals simultaneously, all sharing the main working directory.

That’s important context for what follows.

The innocent prompt

The originating session that day started at 1:21 PM with this prompt:

Derek: how many more books should we try to find from the 1400s?

That’s it. That’s the prompt that, six hours later, would consume the rest of the day in a recovery operation. There is nothing dangerous about it. It’s a curator question about gap analysis: how many incunabula could we add to the collection?

Claude did exactly what a thoughtful research assistant would do. It queried the database, found we had relatively few 1400s books, identified that the BSB (Bavarian State Library) and ISTC (Incunabula Short Title Catalogue) had good coverage in this period, and proposed a 924-book import. We approved it. The script began running.

That session was happy.

The other terminal

While the BSB import was running, I had another Claude Code session open in a different tab. The exact reconstruction of what that session was doing is fuzzier — there were nine candidates — but at some point that day, in some other terminal, Claude wrote a cleanup script.

The intent was reasonable: there were a lot of partially-imported, never-published “draft hidden” books cluttering up the database. Something like:

await db.collection('books').deleteMany({
  status: 'draft',
  hidden: true
});

This is the kind of query that looks innocuous in isolation. We have a deleted_books collection that acts as soft-delete; restoring is one API call away; what could go wrong?

What went wrong: the BSB import in the other terminal was creating books with exactly { status: 'draft', hidden: true }. That’s how the import script staged new books before publishing them. Every book the import created was, for a few minutes, a match for the cleanup query.

The cleanup script didn’t go through the soft-delete API. It called deleteMany directly. The books were gone — not in deleted_books, not in the warehouse, not recoverable through the normal restore endpoint.

Discovery

At 3:43 PM I started a fresh Claude Code session and gave it what I thought was a routine status question. Two messages later:

Derek (3:44 PM): check if supabase has the missing books

That prompt — “check if supabase has the missing books” — was the moment I realized something was wrong. We use Supabase as a denormalized read replica for fast catalog queries. If the live MongoDB count and the Supabase count didn’t match, that was a real signal.

Claude wrote the comparison query. The output came back:

MongoDB total:           32,313
MongoDB published:        6,821
MongoDB drafts:          25,492
Supabase books_catalog:  12,415
Delta (published - supabase):  -5,594

Supabase had 5,594 more published books than MongoDB. That’s not possible unless something had recently been deleted from MongoDB without the corresponding Supabase deletion (which we don’t do anyway — Supabase only mirrors live, not deletes).

Three minutes later, after digging deeper, Claude reported:

Claude (3:49 PM): 1,507 books restored to published. Now checking warehouse for the 4,731 completely missing ones…

Result: Missing from live: 4731 / Found in warehouse: 0 / Truly lost (not in warehouse): 4731

The warehouse — our archive of older imports — had zero of them, because they were all new imports from the day. The pre-import drafts had been deleted before they could be promoted to published, and there was nothing to fall back to.

The final tally was actually 4,758 missing. The GitHub postmortem issue used the rounded number 4,731. I’m using the audited number, because the wrong number ends up in the memory and gets read by the next agent.

The recovery decision

At 4:19 PM Claude in a separate session made the diagnostic call that determined the recovery strategy:

Claude: Found the issue. The draft deletion script from the other session hard-deleted ~914 of the 924 BSB books without going through soft-delete — they’re not in deleted_books. The 10 that survived were already imported before this session.

PITR assessment: You don’t need it. The better recovery path is re-import:

1. The import_candidates collection still has all 924 records marked status: ‘imported’ with their BSB IDs
2. We just reset those 914 ghost records and re-run the import script
3. PITR would roll back all other pipeline work since then — that’s a bigger blast radius than needed

This was the right call. Atlas Point-In-Time Recovery (PITR) would have given us the books back, but at the cost of rolling back six hours of other pipeline work — OCR completions, translation updates, page revisions across the rest of the active corpus. Re-import was surgical.

Over the next four hours we re-imported. By 9:52 PM the GitHub postmortem issue was closed.

Why we got off easy

It is worth being honest about this: we should not have been able to recover this cleanly. We did, but only because three things lined up in our favor, and any one of them missing would have made this a real disaster instead of a long afternoon.

The books were never original to us. Source Library is a catalog over public-domain scans hosted at Internet Archive, the Bavarian State Library, the Bibliothèque nationale de France, Wellcome, e-rara, the Vatican, and ten others. The work we’d done on top of those scans — page splits, OCR, translations, cross-references, classifications, embeddings — that is original to us, and a lot of it was lost. But the underlying books were never going anywhere. They sit on someone else’s server. We can always re-fetch. If we had been operating on documents that only existed in our database, this would have been a permanent loss.

The import_candidates queue had survived. When we set up the BSB import the day before, the script logged every BSB identifier we intended to fetch into a separate collection called import_candidates, with status: ‘imported’ once each was pulled. The cleanup that nuked the books didn’t touch that queue. That collection was the recovery list — without it we would have had no record of which 924 books we’d intended to import. The fact that intent was tracked separately from materialization is what made re-import surgical.

Atlas Point-In-Time Recovery was on as the last-resort backstop. We didn’t end up using PITR (the blast radius of rolling back six hours of unrelated pipeline work was too high), but its presence meant “all 4,758 gone forever, with no fallback” was never on the table. PITR is the floor below the floor. Knowing it was there is what let us choose the cleaner recovery path with confidence — re-import — rather than panic-restoring the whole database.

Of those three, only one was a decision we made: turning on PITR. The other two were luck — the books being public-domain originals, and the import queue happening to be in a separate collection. If we’d been building a library of original scholarship instead of curating public materials, or if the import script had not bothered to log intent before fetch, we would have lost the work.

The lesson I take from this is: when running agents with write access to production, don’t audit only your live data — audit your recovery surface. What do you fall back to? Is it in a different collection from the live data? Is it durable through a destructive operation? Is the floor below the floor turned on? Most teams I talk to have never inventoried this. Neither had I, until April 13. The fact that ours held up was a gift, not a system property.

What the agent actually did wrong

I want to push back on the temptation to call this a “systemic” failure. There are systemic contributing factors — I’ll get to those — but the proximate cause is a behavior an attentive engineer would not have produced.

A careful human engineer about to bulk-delete on production would have done some subset of:

1. Count first. countDocuments({ status: 'draft', hidden: true }) before any delete, see the number, and stop if it’s bigger than expected.
2. Sample. Print 20 titles from the matching set and look at them.
3. Dry-run. Run with --dry-run and review the list.
4. Use the soft-delete API. The codebase has /api/books/delete exactly for this; restoration is one call away.
5. Ask. Send the count and a sample to me before destructive action.

The agent did none of these. It went directly from “I see drafts I should clean up” to deleteMany. There was no intermediate step where the size of the operation surfaced.

This is an AI-specific failure mode. Claude (and most LLMs trained the way Claude is) is fluent in the form of safety hygiene — it can write you a beautiful dry-run script — but is not reliably cautious in its own work. It models the operation it’s about to run as roughly safe because it’s syntactically similar to lots of safe operations it has seen. It does not feel the cost the way a human who has watched data disappear feels it.

I am not making the agent the villain here — I gave it autonomy on a destructive surface without forcing the safety scaffolding. That’s on me. But “an engineer would have done the same” is not true, and pretending it is true is how this happens again.

The systemic factors that made it worse

The agent’s behavior is the proximate cause. Four systemic factors turned a recoverable mistake into a four-hour incident:

1. Parallel sessions sharing one database. Ten Claude Code terminals can all write to the same MongoDB simultaneously. No agent can see what another agent is about to do. The CLAUDE.md document warns about this — “branch switches in one terminal silently break all others” — but the warning is framed around git branches, not data. There is no equivalent guard against parallel destructive writes.

2. Cleanup queries that look safe in isolation. deleteMany({ status: 'draft', hidden: true }) is, on any normal day, a reasonable cleanup that removes orphaned partial imports. The danger emerges only when another process is actively creating documents that match that query as a transient state. That’s a temporal coupling no static code review catches.

3. Soft-delete bypassed by direct MongoDB calls. We have a soft-delete API at /api/books/delete that archives to deleted_books. The cleanup didn’t use it; it called the database driver directly. We trusted our own API. We did not enforce that trust at the database layer.

4. The cron that would have caught it was never installed. Inside the repo, in scripts/maintenance/, there is a script called sync-books-catalog.mjs whose job is to compare MongoDB against Supabase every hour and surface count divergences. If that cron had been running, it would have caught the deletion within 60 minutes — probably before all 4,758 books were lost. The cron sat in version-controlled crontab.conf but was never installed on the live Hetzner server. We had written the safety net, committed the safety net, reviewed the safety net, and not turned the safety net on.

What we changed

The day after the incident, the CLAUDE.md gained an explicit rule:

Data Protection — CRITICAL
— NEVER delete books, pages, or source material without explicit confirmation
— NEVER batch delete — list items first, wait for approval
— deleted_books collection has recoverable items: POST /api/books/restore/[id]
— Assume all books are valuable, even without IA identifiers

That rule is loaded into the system prompt of every Claude Code session in the project. Every future agent reads it.

We also installed the cron. That took five minutes. We had had four months to do it.

What I think this means for AI-assisted development

Four things, in order of how often I think about them.

One: agents are not careful by default about their own destructive actions. This is the lesson I most want people to take from this. The agent will write you a careful script — full of dry-runs and sample-prints and confirmations — when you ask it to. It does not reliably apply that carefulness to its own work when it judges the operation routine. There is an asymmetry between “what an agent recommends” and “what an agent does,” and the asymmetry is in the wrong direction. An engineer who had recently watched data evaporate from their own database would have done a count, a sample, and a dry-run on instinct. The agent did not, because the agent has never watched data evaporate. The remediation is procedural, not motivational: destructive operations on production data have to be physically gated. You cannot rely on the agent to gate them.

Two: parallel agents share state, and that state can be destroyed faster than any individual agent can think. The mental model where each Claude Code session is a “person at a workstation” is wrong. It’s more like ten interns simultaneously editing the same Google Sheet, except they can also drop the whole sheet. Sessions need shared coordination primitives — not just CLAUDE.md text but actual locks, queues, and guard endpoints that the database itself enforces.

Three: the lesson the AI writes after an incident is the lesson the next AI follows. This is great if the lesson is right and stable. It’s terrible if the lesson is wrong, or if it gets compressed into something less precise. Our memory entry says “required Atlas PITR” when actually we didn’t use it. The next time a future Claude reads that, it will believe PITR is the standard recovery path, which will lead it to the wrong tradeoff. Curating AI memory is a job. I am not yet good at it.

Four: the safety mechanisms agents write are only as good as the human’s installation step. The agent wrote a perfectly serviceable monitoring cron. I never installed it. The cycle of “agent invents tool → human installs tool → tool runs in production” requires the human to actually run the third step, and in practice, when you have ten agents inventing tools all the time, the install step is the bottleneck.

What it cost

We lost about four hours of pipeline work to recovery and roughly $200 of re-imported OCR / translation work (most of that was Gemini calls re-running on the books that came back). No user-visible content was lost — the books were not yet published. There was no SLA breach.

The thing it cost more of: trust. For about a week after, I added a manual “review before deleting” gate on every cleanup script Claude proposed, even cleanup scripts I knew were correct. The autonomy I had been comfortable granting shrank. It is mostly back now, but it is not all the way back, and I think that is correct.

Why I’m sharing this

A growing number of people are doing what I’m doing — running multiple AI coding agents in parallel against production data. Most of the public discourse about this is about either the spectacular wins (“look how much I shipped”) or the catastrophic moralizing (“AI cannot be trusted”). The middle ground — actual operational failure modes you have to design against — is mostly conversation between practitioners.

So: here is one, with primary sources. The prompt that triggered it (“how many more books should we try to find from the 1400s?”) was a great research question. The query that killed the books (“delete every draft hidden book”) was the kind of shortcut a careful engineer would not have taken on production. The recovery worked. The lesson was learned. The next agent reads it.

If you’re going to give AI agents write access to production data, the practical defenses are:

1. Inventory your recovery surface, not just your live data. What do you fall back to? Is it in a different collection? Durable through a destructive operation? Is the floor below the floor turned on?
2. Database-level constraints that prevent non-API destructive operations. Don’t trust convention. Make the dangerous path physically impossible.
3. A required pattern for bulk deletes: count → sample → dry-run → confirm → soft-delete. Encode it in the linter, the CI, the CLAUDE.md, and the actual delete endpoint.
4. A divergence monitor between source-of-truth and replicas, running on a cron that you have actually installed.
5. A rule against destructive operations on status fields alone, baked into the agent’s standing instructions.
6. An awareness that “draft hidden” is sometimes a transient state of an active operation, not garbage to clean up.
7. Memory entries written with the precise, audited number, not the rounded-for-the-PR-title number — because the next agent will read them as ground truth.

I had three of those seven. The four I was missing were enough to lose 4,758 books for an afternoon.

The books are back. The agents are smarter. The cron is installed.

— Derek