llms.txt Generator & Validator — Plus AI Crawler robots.txt Rules
Create a spec-compliant llms.txt from a form, a URL list or your sitemap, validate an existing one, and generate robots.txt rules for AI crawlers.
# Example Co > Example Co is an appointment scheduling API for clinics. These docs cover setup, the REST API and webhooks. All API examples use the sandbox at https://sandbox.example.com. Prices are in USD. ## Docs - [Quick start](https://example.com/docs/quick-start.md): Install and send your first request - [API reference](https://example.com/docs/api.md): Every endpoint with request and response examples - [Authentication](https://example.com/docs/auth.md) ## Guides - [Webhooks](https://example.com/guides/webhooks.md): Event types, retries and signature checks ## Optional - [Changelog](https://example.com/changelog.md) - [Blog](https://example.com/blog)
6 links in 3 sections. Matches the llmstxt.org structure. Upload it to the root of your site so it is served at /llms.txt.
Runs entirely in your browser. Nothing is fetched from your site and nothing you paste is uploaded. Browse all free tools.
Found a problem, or something that could be better? We read every message and we fix things quickly.
This llms.txt generator builds the Markdown file that tells AI assistants and agents what your site is and which pages matter. The llms.txt proposal (llmstxt.org) asks for an H1 with the site name, a one-line summary in a blockquote, optional notes, then H2 sections that list links as "- [name](url): notes", with an "Optional" section for pages an agent can skip. Fill in the form or paste your sitemap or a list of URLs, and the file is written in that exact structure.
The validator checks an existing llms.txt line by line, and the third tab writes robots.txt rules for the AI crawlers that matter, separating model-training bots from AI search and user-triggered fetchers. Be realistic about both: llms.txt is a proposal that some AI tools read and many ignore, and robots.txt is a request that well-behaved crawlers honour, not a lock.
How to use
- Enter your site name and a one-sentence summary of what it is and who it serves.
- Paste your sitemap XML or a list of URLs and click Replace sections, or type links as Title | URL | notes, one per line.
- Rename sections, cut pages that would not help an AI answer questions, and move nice-to-have pages into Optional.
- Copy or download llms.txt and upload it to your site root so it loads at https://yourdomain/llms.txt.
- Open the AI crawler tab, pick a preset, adjust individual bots, and add the rules to your existing robots.txt.
Frequently asked questions
What is llms.txt?
llms.txt is a Markdown file at the root of a website, proposed by Jeremy Howard in 2024 at llmstxt.org, that gives language models a short, curated map of the site: a name, a summary and lists of the most useful pages, ideally with clean .md versions. It is meant to be read at inference time, when an assistant or agent needs your content, rather than by search crawlers.
Do ChatGPT, Claude or Google use llms.txt?
No major AI company has committed to llms.txt as a ranking or crawling signal, and Google has said it does not use it for Search. Its practical use today is with coding assistants, agents and documentation tools that fetch it when pointed at a site, which is why developer-docs sites adopted it first. It costs little to publish, but do not expect a traffic change from it alone.
What is the difference between llms.txt and llms-full.txt?
llms.txt is the index: a short list of links with notes. llms-full.txt is a convention used by several documentation platforms where the full text of the docs is concatenated into one Markdown file, so a tool can load everything in a single request. This tool generates llms.txt; keep llms-full.txt for docs sites where the whole corpus fits in a model's context.
How do I block GPTBot and other AI crawlers?
Add a robots.txt group with the crawler's user agent and Disallow: /, for example User-agent: GPTBot followed by Disallow: /. Several user agents can share one group by listing them on consecutive User-agent lines. Training crawlers from OpenAI (GPTBot), Anthropic (ClaudeBot), Google (the Google-Extended token), Apple (Applebot-Extended) and Common Crawl (CCBot) state that they honour these rules.
Will blocking AI crawlers remove my site from ChatGPT or Claude answers?
It depends on which bot you block. GPTBot and ClaudeBot collect training data; blocking them does not stop OAI-SearchBot or Claude-SearchBot from indexing pages for AI search answers and citations. The "block training, allow search" preset keeps you visible in AI search while opting out of training. User-triggered fetchers such as ChatGPT-User and Perplexity-User act for a person in real time, and OpenAI and Perplexity say robots.txt may not apply to them.
Does Google-Extended affect my Google rankings?
No. Google-Extended is a robots.txt token, not a separate crawler; Googlebot still does the fetching. Disallowing it tells Google not to use your content to train Gemini models or ground Gemini answers, and Google states it has no effect on inclusion or ranking in Google Search, including AI Overviews, which are controlled by Googlebot rules.
llms.txt format reference
• Line 1: "# Name". The H1 is the only required element and there should be exactly one.
• Then an optional "> summary" blockquote: the key facts an agent needs to understand everything below.
• Then optional free text (paragraphs or lists, but no headings) explaining how to use the links.
• Then any number of "## Section" headings, each followed by a Markdown list where every item starts with a link: "- [Page name](https://url): optional note".
• A section named "## Optional" holds links that can be skipped when a shorter context is needed. By convention it comes last.
The proposal also suggests serving a clean Markdown copy of important pages at the same URL with .md appended (page.html.md, or index.html.md for a folder). Linking those .md versions from llms.txt gives agents text without navigation and scripts.
AI crawlers by purpose
• Training: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google, token only), Applebot-Extended (Apple, token only), CCBot (Common Crawl), Meta-ExternalAgent (Meta), Bytespider (ByteDance), cohere-training-data-crawler, AI2Bot, Diffbot, Timpibot, omgili.
• AI search and answers: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot, Amazonbot, DuckAssistBot, YouBot.
• User-triggered fetches: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, MistralAI-User. These fetch a page because a person asked; OpenAI and Perplexity say robots.txt may not apply, while Anthropic says Claude-User follows it.
• Not on this list on purpose: Googlebot, Bingbot and Applebot. They power regular search (and Google's AI Overviews and Bing's Copilot answers), so blocking them removes you from search results.
Bytespider has been widely reported to ignore robots.txt. For bots that do not comply, use your CDN or firewall's bot rules, which can match on verified bot identity or IP ranges.
Which preset to choose
Allow all suits documentation, product and marketing sites that want maximum presence in AI answers. Block training, allow search suits publishers and creators who want to be cited and linked by ChatGPT search, Claude and Perplexity without contributing to model training. Block all AI suits paywalled, private or licensed content, at the cost of disappearing from AI search tools.
Changes are not retroactive: content already collected stays in existing datasets. Crawlers re-read robots.txt periodically, typically within a day, and AI search indexes update on their own schedules.
Related free tools
- Robots.txt Generator — Build a robots.txt with presets for allow-all, staging and WordPress, an AI crawlers section (block training, keep AI search), and warnings for the mistakes that de-index sites.
- Meta Tag Generator — Fill in a page's title, description and social card details; get a complete, correctly escaped <head> block with live length counters.
- Schema Markup Generator — Fill a form, get valid JSON-LD for Organization, LocalBusiness, Person, Article, Product, FAQ, Event or Breadcrumbs. Copy it into <head>.
- MCP Config Generator — Define an MCP server once and get the exact config file or command for Claude Desktop, Claude Code, Cursor, VS Code, Windsurf and Codex.