What Does a Library of 3,400 Rare Books Look Like?

Source Library holds over 5,000 digitized rare books — alchemy, Hermetica, natural philosophy, Kabbalah, Chinese medicine, Sanskrit astrology, early modern theology. The collection grew organically over months of curation. What structure does it actually have? We embedded 3,424 book summaries with a neural language model, projected them into a shared vector space, and let a density-based clustering algorithm find the answer.

The algorithm found 48 raw clusters. After editorial review — merging redundant splits, renaming opaque labels, and organizing into macro-domains — we arrived at 34 curated clusters across seven intellectual traditions. Not the categories we assigned — the categories the books assigned themselves, refined by human judgment.

The Map

Each dot is a book. Position reflects semantic similarity — books near each other have similar content. Colors are grouped by macro-domain: warm reds for Western esotericism, purples for Christian traditions, blues for classical & Renaissance, greens for natural philosophy, ambers for Chinese traditions, teals for South Asian traditions. Hover to see the title, author, year, and cluster name. Use the search box to find specific books or clusters.

2D UMAP projection of 768-dimensional sentence embeddings. Spatial proximity ≈ semantic similarity. The projection preserves local neighborhoods but distorts global distances — clusters that appear far apart on screen may be closer in the original embedding space.

Method

We started with 10,083 non-hidden books in our MongoDB database. For each book, we extracted title, author, language, year, AI-generated summary, thematic tags, and the top 30 index terms (weighted by page frequency). Books without a substantial summary or at least some thematic tags were excluded, leaving 3,424 books with enough signal for meaningful embedding.

Each book’s metadata was concatenated into a structured text representation:

Title: Traite de l'Harmonie Universelle
Author: Marin Mersenne
Language: French
Year: 1636
Summary: A comprehensive treatise on universal harmony...
Themes: music theory, Pythagorean harmony, acoustics
Key terms: harmonie, consonance, intervalles, proportion...

We embedded these using all-mpnet-base-v2, a 768-dimensional sentence transformer trained on over 1 billion text pairs. This runs locally — no API calls, fully reproducible. The model produces normalized vectors where cosine similarity reflects semantic relatedness.

For clustering, we used UMAP (Uniform Manifold Approximation and Projection) to reduce the 768-dimensional embeddings to 10 dimensions, preserving local structure while making the space tractable for density estimation. Then HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) identified clusters of varying density without requiring a pre-specified number of clusters. Unlike k-means, HDBSCAN doesn’t force every point into a cluster — books that don’t fit any group cleanly are classified as noise.

Finally, we sent each cluster’s metadata (themes, terms, languages, sample titles) to Gemini Flash and asked it to generate a descriptive name, a one-sentence description, and tradition tags from a controlled vocabulary. The result is a taxonomy discovered from the data, not imposed on it.

From 48 Raw Clusters to 34 Curated Labels

Algorithmic clustering is a starting point, not an end. Reviewing the 48 raw clusters revealed four categories of issues:

• Artificial splits: HDBSCAN found multiple density peaks within what is really one intellectual tradition. Three separate Sanskrit astrology clusters, two Rosicrucian clusters, two grimoire clusters, two natural philosophy clusters. These were merged.
• Single-work clusters: The Wubei Zhi (66 books), Bencao Gangmu (60 books), and Hai Guo Tu Zhi (35 books) each formed their own cluster because Source Library holds many volumes of these multi-volume Chinese encyclopedias. These are real density peaks but represent a single work, not a category.
• Language artifacts: Celtic/Irish texts (29 books) and African/Indigenous studies (29 books) clustered by language and regional origin rather than by subject matter. These are genuine clusters in embedding space, but they reflect linguistic distance more than intellectual affinity.
• Facets masquerading as categories: “Thirty Years’ War Politics” is a historical period, not a subject. “Classical Political Economy” is real but tangential to the library’s focus. These were kept but noted.

After merging 10 groups of redundant clusters and renaming 24 others for clarity, we arrived at 34 curated clusters organized into seven macro-domains.

Seven Intellectual Traditions

The curated clusters organize into seven macro-domains. This isn’t a classification we designed — it’s what the embedding space reveals about the collection’s actual content, refined by editorial judgment.

Western Esotericism

The largest domain, spanning 8 clusters and over 750 books. Western Alchemy is the single biggest cluster (350 books) — Latin and German texts on transmutation, Paracelsian medicine, and spagyric chemistry. It’s arguably too broad, mixing 16th-century Paracelsians with 18th-century chrysopoeia. Nearby: Hermeticism & Theurgy (79), Grimoires & Ritual Magic (78, merged from two raw clusters), Rosicrucianism (63, merged from two), Mesmerism & New Thought (58, merged from animal magnetism and self-improvement), Christian Kabbalah (47), Freemasonry & Secret Societies (47), and Demonology & Witchcraft (74).

Christian Traditions

Six clusters, ~430 books. Continental Christian Mysticism (185) is the second-largest cluster overall — German and Latin texts from Bohme, Tauler, Eckhart, and their successors. Biblical Scholarship (85), Patristic & Eastern Christianity (68, merged from Syriac/Armenian and early apologetics), Swedenborgian Theology (18, a single-author cluster), Religious Persecution & Toleration (30), and Apocalypticism & Prophecy (24). The mystical tradition sits close to the Hermetic cluster in embedding space — correctly reflecting the historical intertwining of these traditions.

Classical & Renaissance

Three clusters. Classical Texts & Philology (187 books) is very broad — Aristotle, Plato, Cicero, Plotinus, plus philological editions. It could arguably be split by period or language. Renaissance Philosophy (110) captures the Ficino-Pico-Bruno axis. German & Dutch Mysticism (47) is a language-specific subset of late-medieval mysticism that the algorithm correctly separated from the broader Christian mysticism cluster.

Natural Philosophy & Science

Six clusters tracing the pre-disciplinary history of science. Astrology & Astronomy (121), Botany & Herbals (90), Natural Philosophy & Optics (72, merged from two raw clusters), Engineering & Mechanical Arts (78, merged from Renaissance and ancient engineering), Music Theory & Harmony (49), and Medical Philosophy (52). These clusters capture the era before physics, chemistry, and biology separated from natural philosophy, alchemy, and Pythagorean harmony.

Chinese Traditions

Four clusters, ~350 books. Chinese Religion & Cosmology (152) covers Buddhism, Daoism, and folk religion. Chinese Military & Strategic Texts (119, merged from Wubei Zhi, coastal defense, and Hai Guo Tu Zhi) — though note this merges multi-volume encyclopedias with independent strategic works. Chinese Medicine (86, merged from materia medica and medical anatomy), and Chinese Celestial & Terrestrial Lore (31). These clusters sit in a completely separate region of the embedding space — Chinese-language content clusters by linguistic distance as much as by subject.

South & Central Asian Traditions

Three clusters. Sanskrit Astrology & Astronomy (182, merged from three raw clusters covering Jyotisha, astronomical treatises, and divinatory texts), Hindu Philosophy & Indology (68), and Islamic Mysticism & Philosophy (41). As with the Chinese clusters, Sanskrit content forms its own island in embedding space — a genuine structural feature, but one driven partly by linguistic distance rather than pure subject matter.

Other

Four clusters that don’t fit neatly into the macro-domains above. Political & Moral Philosophy (123, merged from four small raw clusters including Thirty Years’ War politics, classical economics, legal treatises, and moral philosophy), African & Indigenous Studies (29, a language artifact), Celtic & Irish Traditions (29, also a language artifact), and Pseudo-Dionysius & Commentators (15, a very specific but genuine intellectual cluster around a single late-antique corpus).

The Noise: 564 Unclassifiable Books

HDBSCAN classified 564 books (16%) as noise — points that don’t belong to any cluster with sufficient density. These aren’t bad data. They’re the most interdisciplinary books in the collection: texts that draw on multiple traditions simultaneously and resist placement in any single cluster.

This is actually useful information. A faceted taxonomy that allows multiple tags per book would classify these texts naturally, where a single-label system forces an arbitrary choice.

By the Numbers

Language Distribution

All 34 Curated Clusters

Organized by macro-domain. “Raw” shows how many algorithmic clusters were merged. “Notes” flags known issues: language artifacts, single-work collections, overly broad groupings.

What This Means for the Library

Source Library currently uses 29 hand-coded categories, mostly focused on Western esotericism (Alchemy, Hermeticism, Kabbalah, Rosicrucianism). The clustering reveals that these categories cover only about a third of the collection. The remaining two-thirds — Chinese traditions, Sanskrit literature, natural philosophy, biblical scholarship, political economy — are invisible to the current taxonomy.

Worse, the current categories are flat and mutually exclusive. A book tagged “Alchemy” can’t also be tagged “Medicine” or “Natural Philosophy” — but the clustering shows that these categories overlap heavily. The “Medical Philosophy” cluster (52 books) sits at the intersection of medicine, natural philosophy, and Neoplatonism. “Christian Kabbalah” bridges three traditions. Forcing these into a single category loses information.

The next step is to replace the flat categories with a faceted taxonomy — orthogonal dimensions like tradition, period, language, and form that can be combined freely. A book on Paracelsian medicine could be tagged as Western Alchemy + Medicine + Early Modern + German, allowing it to appear in any of those facets without forcing a single classification.

The 34 curated clusters and their macro-domains are the starting point for designing that taxonomy. They represent what the collection actually contains — discovered from the data, refined by judgment, and honest about where the algorithm sees structure that isn’t really there.

Technical details: Embeddings: all-mpnet-base-v2 (768d, local). Dimensionality reduction: UMAP (n_neighbors=30, min_dist=0.3, metric=cosine). Clustering: HDBSCAN (min_cluster_size=15, min_samples=5, cluster_selection_method=eom). Cluster labeling: Gemini 2.0 Flash with structured JSON output. Visualization: Plotly.js with UMAP 2D projection. 48 raw clusters → 34 curated (10 merge groups, 24 renames). Pipeline code and data available on request.