Evil Martians · Ruby · LLM discoverability

How to get your documentation into LLM training data

Research note · last updated 2026-08-05 · every major fact links a primary source

The biggest lever: a library is learned mostly from its public code. Code corpora take GitHub files with no prose-quality filter at all, so the repo path skips the gate that stops most websites.

Do this

1. Put the docs source in a public GitHub repo. Code datasets such as The Stack ingest Markdown directly (~254 GB of it) with no quality classifier. This is the highest-value action on this page.

2. Use a permissive license, or no license. The Stack v2 and v3 keep files labeled permissive and no_license, and drop only non_permissive (GPL and similar, "all rights reserved"). v1 dropped unlicensed files; that rule is three dataset generations old.

3. Check that Software Heritage has archived the repo. The Stack v2 and v3 are built from the Software Heritage archive, and archival is a separate step that is neither guaranteed nor automatic. workos/authkit (3.3k stars, MIT) is absent from it. A permissive license on an unarchived repo buys nothing.

4. Invest in the README and a runnable examples/ directory. The README is usually the most-trained piece of text about a library.

5. Make your canonical snippet the one people copy. Being in a corpus and being reproducible from memory are different things, and duplication is what separates them. Resend's quickstart is copied into roughly 1,840 GitHub repositories, and models reproduce it verbatim.

6. Lead blog posts with what the reader will learn, in plain sentences that define the terms. Only the first ~512 tokens (~380 words) are scored, so the opening decides the whole document.

7. Earn links to deep pages. Common Crawl ranks domains by harmonic centrality, gives higher-ranked domains a bigger crawl budget, and never copies any domain in full. New, deep, weakly linked pages are the ones it misses.

8. Fix crawl hygiene. Ship a sitemap.xml, 301 moved URLs, expand thin pages, reduce click-depth from the homepage.

9. Measure instead of guessing. Run any URL or repo through /test for all four checks, or /check for the quality score alone.

Skip this

Blocking CCBot, ClaudeBot, or GPTBot in robots.txt, if you want inclusion. This is the one lever that is known to work on Anthropic's own crawler, in the direction of exclusion.

Private repos and copyleft licenses. Private content never enters code corpora, and non_permissive files are dropped by name.

Chasing stars. Star count is not an inclusion filter in any version of The Stack. It decides only whether a fork gets crawled (5 or more stars) and which copy survives deduplication.

Mirroring your docs verbatim into other repos. Near-duplicate detection (MinHash-LSH) collapses copies into one representative and removed about 40% of permissively licensed files in The Stack v2. Your sentences quoted inside someone else's different document survive as a distinct example.

Stripping boilerplate to raise your quality score. Measured below: it does not help and can hurt.

Opening with war stories, benchmarks, or announcements. Those genres score below the keep bar no matter how they are edited.

llms.txt. No major AI system uses it for training or inference.

Relying on the rendered docs site alone. Reference and configuration pages fail the quality filter almost without exception. Let the GitHub channel carry them.

The four channels

The web gate is the open FineWeb-Edu classifier, which keeps a document when its score rounds to 3 or higher (raw score ≥ 2.5). It was deliberately tuned toward school-level explanation and away from technical pages.

What we measured

Stars do not predict memorization

We probed 168 documentation repositories, 309 passages (give a model 55 words of a page and measure the longest exact continuation it produces, with a positive and a negative control). Nineteen repos produced at least one hit.

vercel/next.js (141k stars) and supabase/supabase (107k stars) produced nothing. getsurfboard/manual (115 stars) produced a 33-word exact run.

Independent copies predict it instead. In an earlier, denser study of 15 repositories (38 passages), copy count correlated at ρ = +0.70, p = 0.005, while stars gave ρ = +0.28 (p = 0.31, not significant). tailwindlabs/tailwindcss.com has no license, about 140 stars and 68 independent copies, and was the most memorized repo in that sample. Stars measure how many people liked a project. Copies measure how many times its sentences exist in the world, and a corpus counts the second one. Re-running those passages against a frontier model from a different lab reproduces the same ranking (ρ = +0.68, p = 0.0065), so this reflects what is on the internet and holds across labs.

A missing license does not suppress anything

Unlicensed repos were hit more often, at Fisher exact p = 0.079. We do not claim a missing license helps: prominence is confounded with license here. The supportable reading is that there is no evidence a missing license suppresses inclusion, which matches the documented v2/v3 policy. The copyleft and source-available groups are too small to test.

Rendered reference docs do not clear the quality bar

Common Crawl reached about a third of two sites, and quality played no part

It held 37% of evilmartians.com (221 of 600 pages) and 27% of docs.anycable.io (34 of 125). Crawled and missed pages scored the same on quality (1.52 vs 1.51, and 1.83 vs 1.83). The gap comes entirely from how the crawler samples links.

Rewriting the lead clears the bar; removing boilerplate does not

The blog post clears the bar once its opening explains what the reader will learn. The reference page never clears it, even fully rewritten. Genre sets the ceiling.

Failing the quality filter does not mean absent from training

Both pages reached training through the code channel. Supabase's docs live in supabase/supabase under Apache-2.0, and Resend's quickstart is the canonical snippet from its MIT-licensed resend-node repo. Supabase's Row Level Security guide sits in the same repo and showed no verbatim recall, because it is prose, and prose gets paraphrased instead of copied.

Check your own pages

Score the page rather than judging it by eye. The people who built these classifiers report that expert humans cannot predict them: in the DCLM keynote Ludwig Schmidt found hand-labelers could not tell the best classifier variants apart, and Ari Morcos reports that NLP graduate students could not predict DCLM's classifier above chance.

Limits

• These are open proxies, not any lab's filter. FineWeb-Edu and Common Crawl are documented and public. Anthropic, OpenAI and Google do not disclose theirs.
• Quality scores are single runs on the first ~512 tokens. Differences under 0.3 are noise.
• A negative memorization result is weak evidence. Models memorize a fraction of what they read, so an absence proves little. A positive result is strong: text cannot be reproduced unless it was seen.
• Copy counts come from GitHub code search, which sees one slice of duplication. rails/rails shows 2 GitHub copies yet produced a 30-word run, because Rails Guides are duplicated across the web instead.
• Rules are versioned. C4's curly-brace rule is from 2019 and modern filters are softer. The Stack v1 excluded unlicensed repos and v2/v3 include them. Any claim about licenses has to name the corpus version.
• Inclusion is not influence. Trained-on content is still deduplicated and weighted.
• Run memorization probes in an empty directory with tools off. Our first cross-model run scored a perfect 1.0 overlap because the agent found the probe's own answer file on disk. The same passage from an empty directory scored 0.0.
• There is no working public Stack-membership oracle. The "Am I in the Stack?" space answers "No" for every input, including torvalds and rails. Check collection and license separately instead.

Sources

• Common Crawl: Baack, FAccT'24 · Mozilla · CC index
• Web filters: C4 / T5 · Documenting C4 · RefinedWeb · FineWeb / FineWeb-Edu · classifier
• Code corpora: StarCoder2 / The Stack v2 · The Stack v1 · v2 · stack-v3-train · Software Heritage API
• Crawler policy: ClaudeBot · Google on llms.txt
• Practitioners: Penedo / FineWeb · Ben Allal / FineWeb-Edu · Schmidt / DCLM · Morcos / data curation · Scialom / Llama 3 · Karpathy

Last updated 2026-08-05. Every measured number here is reproducible from scripts/ and data/ in this repo.