How your content reaches a model

Checklist

Ordered by measured impact.

Training

1. Put the docs source in a public GitHub repo. highest value The only channel with no prose filter. How →
2. Use a permissive licence, or none. The Stack keeps permissive and no_license; drops non_permissive. Which →
3. Look the repo up in the Am-I-in-The-Stack index; ask Software Heritage to archive it. The index answers membership directly; archival you can trigger yourself. Why →
4. Write the README and a runnable examples/. Usually the most-trained text about a library.
5. Own the snippet people copy. Copies predict recall at ρ = +0.70. Stars predict nothing. Data →
6. Close issues with fix PRs that carry a regression test. That exact linkage is what SWE-bench-style pipelines mine into RL training tasks. The five filters →
7. Open posts with what the reader will learn. Only the first ~512 tokens are scored. What scores →
8. Earn links to deep pages. Common Crawl never copies a domain in full. How it samples →

Retrieval

1. Verify a crawler actually gets a 200. most often broken robots.txt and your WAF disagree. Why →
2. Serve markdown: Accept: text/markdown or a .md twin. Saves the agent parsing your theme. How →
3. Ship sitemap.xml, 301 moved URLs, cut click depth. Crawl budget →
4. Decide crawler by crawler. Nine user agents, three jobs, different costs. Table →
5. Measure. /test for all checks, /check for the quality score alone.

Skip

• Blocking CCBot, ClaudeBot, GPTBot if you want inclusion. Blocking is the one lever proven to work, and it works toward exclusion.
• Private repos and copyleft. Never collected, or dropped by name.
• Chasing stars. No Stack version filters on them; only forks need 5+.
• Mirroring your docs into other repos. MinHash collapses copies into one.
• Stripping boilerplate. Measured: 2.43 → 2.48, and worse for reference pages. Chart →
• Opening with war stories, benchmarks or announcements. Genre caps the score.
• llms.txt. No major AI system uses it. More →
• Relying on the rendered docs site alone. 0 of 118 pages cleared the filter.

Two systems

Training makes you the default answer when nobody looks anything up. Retrieval keeps that answer current.

Training: collected, then filtered

• Public and crawlable clears gate 1 only.
• Web pipelines discard around 90% of what they collect (RefinedWeb).
• Your four surfaces meet different gates.

Common Crawl never copies a domain in full

• Ranks domains by harmonic centrality and budgets crawling by rank.
• Misses new, deep, weakly linked pages.

Crawled and missed pages score the same. The difference is links, and links can be earned.

The filter rewards school-style explanation

• FineWeb-Edu keeps a document scoring 2.5 or higher (classifier).
• Deliberately tuned toward school-level explanation and away from technical pages.
• Scores the first ~512 tokens only.

The code channel has no prose filter

• Licence check and duplicate check. Nothing else.
• A page scoring 1.6 on the web filter is collected without complaint from a public repo.
• The Stack v1 held about 254 GB of Markdown.

1. The repo is public. Private content never enters a code corpus.
2. The corpus collected it. The Stack v2 came from the Software Heritage archive, where archival is neither automatic nor star-driven: workos/authkit has 3.3k stars, an MIT licence, and no record there. v3 crawled GitHub directly, cutoff 2025-08-07; a repo public after that date waits for the next snapshot, whatever its licence. The official Am I in The Stack? index answers membership repo by repo. Opting out works: the rspec and bridgetownrb orgs asked out and are gone org-wide.
3. Only non-permissive licences are dropped. v2 and v3 keep permissive and no_license files: what goes is GPL and kin, all-rights-reserved, and source-available terms like BUSL, FSL and SSPL (StarCoder2 / The Stack v2). v1 dropped unlicensed files; that rule is three dataset generations old.

GitHub reports NOASSERTION for every licence it cannot match, so spdx_id cannot tell an Elastic-licensed repo from an unlicensed weekend project. Our check reads the LICENSE text, and the observed index gets the last word: v3 decides file by file, and its calls surprise both ways. karafka/wiki is in with an all-rights-reserved licence file; karafka/karafka, LGPL, is out; sidekiq/sidekiq, also LGPL, is in.

Copies predict recall. Stars predict nothing

• vercel/next.js (141k stars) and supabase/supabase (107k): no verbatim runs.
• getsurfboard/manual (115 stars): 33-word exact run.
• tailwindlabs/tailwindcss.com: no licence, ~140 stars, 68 independent copies, most memorized repo in that sample.
• MinHash-LSH removed about 40% of permissively licensed files in The Stack v2 as duplicates. Mirroring your own docs adds nothing.

Verbatim recall proves inclusion; silence proves nothing

• Method: give a model 55 words, measure the longest exact continuation, against a positive control (the MIT licence, must fire) and a fabricated passage (must not).
• A long run is strong evidence. Text cannot be reproduced unless it was seen.
• A null result is weak. Read it as "no proof found", never as "absent".
• A short run is a hint. Nine words of "returns 204 No Content on success" is phrasing thousands of API docs share.

Both failed the web filter and reached training through code: supabase/supabase under Apache-2.0, resend-node under MIT, copied into roughly 1,840 repositories. Supabase's Row Level Security guide sits in the same repo with no recall. Snippets get copied; prose gets paraphrased.

Five filters decide whether a repo becomes RL training tasks

Agentic RL trains on verifiable tasks mined from repos: an issue, the PR that fixed it, and a test that fails before the fix and passes after (SWE-bench). The mining is mechanical, so each step is a filter a repo passes or fails:

1. Enough stars to get scanned. SWE-bench Multilingual starts from the top-100 most-starred repos per language.
2. A test command a script can derive. Build and test commands are read from the contribution docs and the CI workflow files. A suite that only a maintainer knows how to run yields nothing.
3. Tests that build and finish. About 30% of candidate repos are discarded because they will not build locally, build too slowly, or run their tests too long.
4. A container-reproducible environment. Each task is rebuilt in layered Docker images (base, environment, instance).
5. Merged PRs that close an issue and modify tests. Each such PR is the raw material of one task. A repo whose fixes never reference an issue yields zero tasks, whatever the quality of its code.

We measured the top-starred open-source Rails products against these filters. The last one is where they fail: 6 of 17 merged even one issue-closing, test-touching PR in their last 50, and 5 of 17 have GitHub issues disabled (Discourse and OpenProject triage on their own forums; GitLab and Redmine are mirrors of development that happens elsewhere). The runtime filter bites too: Zammad's suite runs a 93-minute median. Ruby is one of SWE-bench Multilingual's 9 languages, and it is absent from Multi-SWE-bench and its Multi-SWE-RL training community, whose task pipeline is open to contributions.

Retrieval: what an agent fetches today

Nine crawlers, three jobs, one decision each

Published crawler lists: Anthropic, OpenAI, Google. All state their crawlers honour robots.txt. Our check reports the training crawlers, because that gate is the permanent one.

Content-Signal states the use, Disallow states the access

• One line, three signals, yes or no each: Content-Signal: search=yes, ai-input=yes, ai-train=no
• search: a search index, links and short excerpts. Excludes AI summaries.
• ai-input: the page fed to a model at answer time (RAG, grounding).
• ai-train: training or fine-tuning.
• A signal you leave out grants nothing and restricts nothing. Omission is not a no.
• It blocks no request. A page with ai-train=no still fetches, so we report the crawl as allowed and flag the declined use beside it.
• A no is an express reservation of rights under Article 4 of EU Directive 2019/790. It is aimed at people and courts rather than at fetch logic.
• Cloudflare published it in October 2025 and applies it to managed robots.txt across its network, so a site can be carrying one it never typed.
• No lab has published a policy of honouring it. We report it, and score it as neither a win nor a failure.

Sources: contentsignals.org, the IETF draft draft-romm-aipref-contentsignals, and the policy text served in developers.cloudflare.com/robots.txt.

Allowed and reachable are different facts

• robots.txt is a request. Your CDN, WAF or bot rule is the enforcement point.
• We fetch with a crawler user agent and report the status code that comes back.

Markdown lets an agent read your sentences

• Content negotiation: return text/markdown for Accept: text/markdown.
• Markdown twin: the same page at a .md URL.
• Neither is required by any lab. Both decide whether an agent reads your sentences or parses your theme.

A sitemap gets URLs discovered; centrality decides what gets crawled

• CCBot reads sitemaps. Common Crawl documents that its crawler supports the Sitemap Protocol and uses any sitemap announced in robots.txt. So the mechanism is real: a listed URL can be found without an inbound link.
• It does not decide what gets fetched. Common Crawl prioritises by harmonic centrality, a measure of how close a page sits to the structural core of the web. Being listed is not being chosen.
• We measured the effect and could not find one. Across the 93 resources on the scorecard, the 56 with a working sitemap have a median of 27 pages in Common Crawl and the 37 without have 6, but the distributions overlap heavily: a randomly chosen site with a sitemap beats one without only 60% of the time (Mann-Whitney, p ≈ 0.11), and the share with no coverage at all is the same either way (20% against 22%).
• Google's guidance is conditional too. Its documentation says a sitemap helps a large site, a site with poorly linked pages, or a new site with few external links, and that you may not need one for a site of "about 500 pages or fewer" whose pages are properly linked. That is about Search, not about corpora.
• So: cheap, documented, worth shipping, and not a lever. Click depth and inbound links are what centrality actually reads.
• Redirect moved URLs with 301, cut click depth, earn links to what matters.

llms.txt is widely adopted and used by nobody

• A curated index of your docs for LLMs. Broad adoption, and no major AI system uses it for training or inference.
• We report it as informational: present or absent, never as a win.

Your repo: discovery, evaluation, licences

An agent does three things with a library: discovers it, evaluates whether to trust it, suggests it. Install, configure, deploy and pay are separate problems that /test does not measure.

The crawler and corpus numbers above are measured. The rows below are mechanisms: how GitHub's own search and licence tooling behaves.

A licence answers two different questions

• Whether a corpus keeps your files, and whether anyone may use them. The two answers routinely disagree.
• The case people get backwards: no LICENSE file. The Stack v2 and v3 keep unlicensed code, while default copyright grants viewing and forking on GitHub and nothing more.

This describes what tooling and company policies do with each licence. It is not legal advice. The classification on /test reads the LICENSE text itself, for the NOASSERTION reason above.

Where you stand

The quality filter classifies genre. A reference page scoring 1.6 is written fine; it is aimed at a classifier tuned on school material. The code channel exists to carry it.

Check your own pages

Score the page; eyes mislead. In the DCLM keynote, Ludwig Schmidt found hand-labelers could not tell the best classifier variants apart, and Ari Morcos reports NLP graduate students could not predict DCLM's classifier above chance.

Glossary

Common Crawl

A non-profit that crawls the web and publishes the result; most open web corpora start from it. Samples by domain centrality, never copies a domain in full. Our check asks whether your URL is in the latest monthly index.

FineWeb and FineWeb-Edu

FineWeb is an open web corpus built from Common Crawl. FineWeb-Edu is the subset kept by an openly published educational-quality classifier. We score with it because it is the only filter in this pipeline anyone can run.

The Stack v2 and v3

Open code corpora keeping Markdown alongside source: v2 from the BigCode project, built from Software Heritage; v3 from Hugging Face's code team, a direct GitHub crawl with a 2025-08-07 cutoff. Licence policy: permissive and no_license files kept, non_permissive dropped, decided file by file. v1's stricter rule is what most advice still describes.

Software Heritage

A long-term archive of public source code, and the collection step for The Stack v2 (v3 crawls GitHub directly, using the SWH graph as one seed list). Archival runs through listers and explicit save requests, so a popular permissive repo can simply be absent.

SWE-bench family

Agentic coding benchmarks and RL training sets mined from GitHub: an issue, the PR that fixed it, and a test that fails before the fix and passes after make one verifiable task. SWE-bench is Python-only; SWE-bench Multilingual spans 9 languages including Ruby; Multi-SWE-bench covers 7 languages without Ruby, and its Multi-SWE-RL community collects RL training instances through an open pipeline.

CCBot

Common Crawl's crawler, operated by no AI lab. Blocking it in robots.txt removes you from future crawls and from the corpora built on them.

Memorization probe

Show a model the first 55 words of a passage, measure the longest exact continuation. Several passages, several models, two controls. If the controls misbehave, nothing in that run is interpretable.

Markdown twin

The same page served as Markdown at a parallel URL, usually .md. Distinct from content negotiation, which serves Markdown at the same URL on request.

Limits

• These are open proxies, not any lab's filter. Anthropic, OpenAI and Google do not disclose their pipelines.
• Quality scores are single runs on ~512 tokens. Differences under 0.3 are noise.
• A negative memorization result is weak evidence. A positive one is strong.
• Copy counts come from GitHub code search, one slice of duplication. rails/rails shows 2 copies yet produced a 30-word run, because Rails Guides are duplicated across the web instead.
• Rules are versioned. C4's curly-brace rule is from 2019; The Stack v1 and v2 disagree about unlicensed code. Any licence claim has to name the corpus version.
• Inclusion is not influence. Trained-on text is still deduplicated and weighted.
• Run probes in an empty directory with tools off. Our first cross-model run scored a perfect 1.0 because the agent found the probe's own answer file on disk.

Every number here is reproducible from scripts/ and data/ in the repo.