Here is the job this course does: it gets your documentation working as a discovery channel, not just a support cost. Three years ago that sentence would not have made sense. Today the second audience for your docs is the machines, and they never file a support ticket. I will cover how AI systems find and use docs, what makes a page get cited, the setup that makes docs readable to machines, the open versus locked decision and how to measure all of it.
Why documentation became a growth channel
Documentation became a growth channel because AI assistants now answer product questions by naming a specific tool and showing the exact steps, and the source they reach for is usually your docs, not your marketing page. That turns docs into a discovery surface rather than a support cost.
For most of the last two decades I watched teams treat docs as a cost center. Write enough to cut support tickets, then move on. Acquisition came from marketing pages, the blog and classic search.
That model is breaking. Ask an assistant how to test an LLM app for jailbreaks and it returns an answer, not ten links, and inside that answer it tends to name a tool and show how to run it. Your marketing page says you are the leading platform. Your docs say pip install yourtool and show the command. The model usually trusts the second one.
So the cost of locking docs changed. It used to be a little lost self-service and some long-tail search traffic. Now it can be your absence from the set of products an AI will even consider. You do not lose a ranking, you lose the mention.
- Docs are a discovery surface now, not only a support surface.
- AI names the source with concrete, runnable how-to, which is usually the docs and not the marketing page.
- The real cost of gating is being left out of the recommendation set, not a lost ranking.
How LLMs and coding tools find your docs
AI systems find your docs two ways: training, where a model crawls the public web once and bakes it in, and retrieval, where tools like AI Overviews, ChatGPT Search and Perplexity fetch live pages at query time and cite passages. Retrieval is the channel that is growing, and it rewards pages that are live and readable right now.
Training is the slow channel. Models crawl the web, and documentation is high-value fuel because it is structured and dense. Once it is in the weights it is baked in, which means anything already public has very likely been crawled already. Locking the door now does not recall what left.
Retrieval is the fast channel, often called RAG (retrieval-augmented generation). The system fetches live pages when you ask, pulls the relevant passage and writes an answer with citations. This rewards current, readable pages, so GEO (generative engine optimization) is ongoing work, not a one-time event.
One more mechanic matters: query fan-out. A complex question gets split into smaller sub-questions, and each one is answered by a different passage. That is why granular, well-labeled pages tend to beat one long page. Run the diagram to watch it happen.
- Training bakes content in, retrieval rewards live pages. Optimize for both, since retrieval is the part that is growing.
- Query fan-out splits a question into sub-questions, so one clean passage per likely sub-question tends to win.
- A machine-readable layer (an llms.txt index plus Markdown mirrors) is the current best practice.
What makes a page get cited or recommended
Four page types tend to earn citations: a named-concept page that owns one term, runnable proof like install commands and working code, capability pages an AI can use to confirm you do the thing, and a structured API and CLI reference. In my experience the docs that get recommended most build all four.
API here means application programming interface, and CLI means command-line interface. Tap each pattern below to see what it looks like in practice.
- Own the concept. Give each key term one page with a crisp definition up top.
- Show runnable code. It is what coding tools hand to developers.
- Make every capability findable as its own clearly titled passage.
- Publish real reference docs. They prove depth and capture exact-match terms.
The setup that makes docs AI-ready
The setup that makes docs AI-ready is a machine-readable layer: serve every page twice as a human HTML version and a clean Markdown mirror, publish an llms.txt index at the docs root that lists every page, structure each page for passage retrieval and keep it fresh. Most teams can ship the core of this in about a day.
Publish a machine-readable layer. Serve each page twice, the human HTML version and a clean Markdown mirror, and add an llms.txt index at the docs root that lists everything. Markdown strips nav, scripts and clutter, which leaves the model pure content that is cheaper to retrieve.
Structure for passage retrieval. Use descriptive headings that mirror how people ask, a strong definition at the top of each page, and a bias toward granular pages over long ones.
Add structured data and keep it fresh. Schema for technical articles, software and FAQs gives machines explicit signals. Retrieval favors current pages, so stale docs tend to decay in a way baked-in training data does not.
# Acme Scan, Docs ## Guides - [Quickstart](https://docs.acme.dev/quickstart.md) : install and first scan - [CI/CD](https://docs.acme.dev/guides/ci-cd.md) : run in your pipeline ## Reference - [CLI](https://docs.acme.dev/reference/cli.md) : every command and flag
If you advertise an llms.txt index and a Markdown layer, you have made promises to crawlers. Gate or remove individual pages but leave them listed in that index, and you create a field of broken or forbidden links crawlers keep hitting, which is worse than either extreme. If you go partial, govern the index and the access rules together.
- Ship llms.txt plus Markdown mirrors. It tends to be the single biggest AI-readiness lever.
- Write granular pages with descriptive headings and definition-first openings.
- Add schema and keep content fresh so you stay retrievable over time.
- Never orphan gated pages inside your crawler index.
Open, locked, or a deliberate middle
The open versus locked decision comes down to two questions per section: how much does this page drive discovery, and how genuinely sensitive is it. Content that drives discovery and is not truly secret stays open. Only genuinely proprietary, low-traffic content is a real candidate for gating.
The case for locking down is competitive intelligence. Open docs let rivals see your feature set, methodology and roadmap, and AI makes that easier to harvest. That concern is legitimate, so take it seriously.
Two counterarguments tend to win, though. First, locking docs hides you from the AI systems that recommend products to buyers, which is often a larger loss than the exposure you prevented. Second, most of this content is already public and already crawled, so locking now claws back very little secrecy.
So the honest answer is usually a deliberate middle. Plot your content below and watch where it lands.
high value, low risk
high value, high risk
low value, high risk
- Does it drive discovery? If yes, keep it open.
- Is it proprietary and low-traffic? If yes, it is a gating candidate.
- Already public means already crawled, so locking now recovers little secrecy.
Measuring the impact so it is not a guess
You can measure the open versus locked call instead of guessing it: split your docs referral traffic into classic organic search versus AI and LLM referrals, run log-file analysis to see which bots hit the docs, and spot-check whether assistants cite you. In most accounts I see, the AI referral line is the one that is moving.
Size the current value. Split docs referral traffic into classic organic search and AI or LLM referrals from tools like ChatGPT, Perplexity and AI Overviews. That second line is the GEO signal, and it tends to be the one trending up. Watch the trend, not just the snapshot.
Measure crawl behavior directly. Log-file analysis shows exactly which bots hit the docs and how often, which turns "we think LLMs read our docs" into a number you can defend.
Track citations by hand. Ask the major assistants your category's buyer questions on a schedule and note whether you are named, how accurately, and from which page. Put the three together and the decision makes itself.
Directional example for shape, not client data.
- Split referral traffic into organic and AI or LLM to size the discovery at stake.
- Run log-file analysis to prove crawler behavior with real numbers.
- Spot-check citations on a schedule: are you named, accurately, and from where.