Docs Demo GitHub pip install geo-optimizer-skill

Generating llms.txt

geo llms auto-generates an /llms.txt file from your sitemap so AI crawlers can understand your site’s structure.

What is llms.txt?

llms.txt is a plain-text Markdown file placed at the root of your website (https://yoursite.com/llms.txt). It works like robots.txt but for AI language models: instead of telling crawlers what to block, it tells them what to read and how to understand your site.

Spec: llmstxt.org

A well-structured llms.txt helps AI crawlers:

  • Understand what your site is about before crawling individual pages
  • Discover your most important pages quickly
  • Categorize your content into logical sections
  • Extract a concise site summary for use in AI-generated answers

Why It Matters

AI crawlers (Perplexity, Claude, ChatGPT Search) use llms.txt as a trust and structure signal. Sites without it are treated as generic; sites with it are indexed more accurately. When a user asks an AI a question your site answers, having a well-structured llms.txt increases the probability of being cited.

Usage

# Minimal — auto-detects sitemap from robots.txt
geo llms --base-url https://yoursite.com

# Full options
geo llms \
  --base-url https://yoursite.com \
  --site-name "MySite" \
  --description "Free online calculators for finance and math" \
  --output ./public/llms.txt \
  --sitemap https://yoursite.com/sitemap.xml \
  --max-per-section 10 \
  --fetch-titles

All Flags

Flag Default Description
--base-url (required) Root URL of the site (e.g. https://yoursite.com)
--output ./llms.txt Output file path
--sitemap auto-detected Direct URL to the sitemap XML; skips auto-detection
--site-name derived from URL Human-readable name for the site
--description none Short description of the site (1–2 sentences)
--max-per-section 20 Max URLs per content section
--fetch-titles false Fetch each URL to extract its <title> tag (slower but richer output)
--check-drift false Check an existing llms.txt against the current sitemap instead of generating one; exits 1 on stale URLs
--llms-file fetched live Local llms.txt to check with --check-drift, instead of fetching {base-url}/llms.txt

Checking for drift

llms.txt is generated once and then trusted by AI agents as a map of the site, and nothing checks whether that map is still accurate as pages get renamed or removed. --check-drift compares the URLs listed in llms.txt against the site’s current sitemap (reusing the sitemap fetch geo llms already does for generation, so it costs no extra per-link HTTP request):

geo llms --base-url https://yoursite.com --check-drift
  • Stale URLs: listed in llms.txt, gone from the sitemap. These actively hurt citability: an agent following the link hits a 404 or redirect instead of the content it was told to expect. Presence of any stale URL exits with code 1, so this is safe to run in CI. The command also exits with code 1 when llms.txt, the --llms-file path or the sitemap cannot be read.
  • Missing URLs: in the sitemap, not yet in llms.txt. This means llms.txt is behind, not wrong. Regenerate it with geo llms --base-url ... --output ....

Sitemap Auto-Detection

When --sitemap is not specified, the script:

  1. Fetches https://yoursite.com/robots.txt
  2. Looks for a Sitemap: directive
  3. If found, uses that URL as the sitemap
  4. If not found, tries https://yoursite.com/sitemap.xml as a fallback
  5. If the sitemap is a sitemap index (contains multiple sitemaps), it fetches and merges all of them

If your sitemap is in a non-standard location, always pass --sitemap explicitly.

Structure of the Generated File

The script generates a structured Markdown file. Example output:

# MySite

> Free online calculators for finance, math, and health. Accurate, fast, no signup required.

## Finance Tools

- [Mortgage Calculator](https://yoursite.com/finance/mortgage): Calculate monthly payments for fixed and adjustable-rate mortgages
- [Compound Interest Calculator](https://yoursite.com/finance/compound-interest): Compute growth over time with compound interest
- [Loan Amortization](https://yoursite.com/finance/amortization): Full amortization schedule for any loan

## Math

- [Percentage Calculator](https://yoursite.com/math/percentage): Calculate percentages, increases, and decreases
- [Fraction Simplifier](https://yoursite.com/math/fractions): Reduce fractions to their simplest form

## Blog & Articles

- [What is GEO?](https://yoursite.com/blog/what-is-geo): Introduction to Generative Engine Optimization
- [robots.txt for AI Bots](https://yoursite.com/blog/robots-txt-ai): Complete guide to configuring AI crawlers

## Optional

- [About](https://yoursite.com/about)
- [Contact](https://yoursite.com/contact)
- [Privacy Policy](https://yoursite.com/privacy)

Structure breakdown:

  • # H1 — your site name
  • > blockquote — your site description (parsed by AI as the summary)
  • ## H2 sections — content categories (auto-detected from URL patterns)
  • - [Title](URL): description — links with optional descriptions
  • Links without descriptions go under ## Optional

Where to Put the File

Place llms.txt at the root of your web server so it’s accessible at https://yoursite.com/llms.txt.

Static Sites (Astro, Next.js, plain HTML)

# Astro
geo llms --base-url https://yoursite.com --output ./public/llms.txt

# Next.js (app router)
geo llms --base-url https://yoursite.com --output ./public/llms.txt

# Plain HTML
geo llms --base-url https://yoursite.com --output ./llms.txt

After running: deploy as you normally would. The file will be served at /llms.txt.

WordPress

# Generate the file locally
geo llms \
  --base-url https://yoursite.com \
  --sitemap https://yoursite.com/sitemap.xml \
  --output llms.txt

# Upload to WordPress root (where wp-config.php lives)
scp llms.txt user@yourserver.com:/var/www/html/llms.txt

Or use a plugin like “WP Robots Txt” to serve a custom llms.txt via code.

Generic Site (any server)

# Generate
geo llms --base-url https://yoursite.com --output llms.txt

# Upload via FTP/SCP to the web root
scp llms.txt user@yourserver.com:/var/www/yoursite.com/llms.txt

Verify It’s Live

curl https://yoursite.com/llms.txt

Expected: Markdown content starting with # YourSiteName.

If you get a 404, the file is not in the right location. Check your web root directory.

Example for Specific Frameworks

Astro — with custom sitemap

geo llms \
  --base-url https://yoursite.com \
  --site-name "MySite" \
  --description "Free tools and calculators" \
  --sitemap https://yoursite.com/sitemap-index.xml \
  --output ./public/llms.txt \
  --max-per-section 15 \
  --fetch-titles

Next.js — standard setup

geo llms \
  --base-url https://yoursite.com \
  --site-name "MySite" \
  --output ./public/llms.txt

WordPress — with Yoast SEO sitemap

geo llms \
  --base-url https://yoursite.com \
  --sitemap https://yoursite.com/sitemap_index.xml \
  --site-name "MySite" \
  --description "Your WordPress site description" \
  --output llms.txt

Generic site — no sitemap

If you have no sitemap, create a simple llms.txt manually following the structure above, or generate a sitemap first (many free tools exist online), then pass it with --sitemap.