Auto-Generate llms.txt in Astro Content Collections

✓ First-hand experience: Based on direct hands-on use. Built live on astro-content-lab.vercel.app, directly following the sitemap.xml work from the previous article.

I just finished writing a custom sitemap.xml generator that reads straight from Content Collections instead of relying on a plugin’s defaults. The instinct here is exactly the same problem, wearing a different name.

llms.txt looks a little like a Markdown sitemap, but it is not a crawler-control standard and it does not replace XML sitemaps or robots.txt. The proposal gives compatible LLM tools a concise index of a site’s useful material. Since I already built the collection-reading pattern for sitemap.xml, generating this export was mostly familiar code.

But before writing any code, the question worth asking honestly is: does this file actually do anything?

What the Proposal Actually Does

llms.txt is a community proposal, not a W3C or IETF web standard. It suggests a Markdown file at the site root that summarizes useful content for tools that choose to read it.

Google’s documentation is direct: Google Search does not use llms.txt, including for its generative AI features. Creating the file neither improves nor harms Google rankings or visibility.

For other systems, support depends on the product. I found no basis for promising more citations, better crawling, or faster inclusion. That uncertainty belongs in the article because “AI visibility file” is exactly the kind of phrase that turns a small experiment into fake certainty.

Read the llms.txt proposal for its intended format and Google’s AI search guidance for what Google actually uses.

I am treating it as a cheap, reversible experiment. Worth documenting. Not worth overselling.

Two Files, Two Different Jobs

llms.txt and llms-full.txt aren’t duplicates of each other. They serve different purposes, the same way a table of contents and the full book serve different readers.

llms.txt is the index: site name, a short description, and selected pages grouped by topic with links and summaries. A compatible tool can use it to find material relevant to a task.

llms-full.txt is an optional full-text export. It can become large, duplicate public content, and expose material you did not mean to aggregate. I include only published entries and would reconsider it before using the same pattern on a private, licensed, or very large collection.

Both get generated the same way: as Astro API routes, reading from the same getCollection() calls that already power the sitemap.

Building llms.txt

// src/pages/llms.txt.ts
import type { APIRoute } from 'astro'
import { getCollection } from 'astro:content'
import { SITE_NAME, SITE_URL, SITE_DESCRIPTION } from '../config'

export const GET: APIRoute = async () => {
  const posts = await getCollection('posts', ({ data }) => data.status === 'published')
  const reviews = await getCollection('reviews', ({ data }) => data.status === 'published')
  // ...compares, guides follow the same pattern

  const lines: string[] = []
  lines.push(`# ${SITE_NAME}`)
  lines.push('')
  lines.push(`> ${SITE_DESCRIPTION}`)
  lines.push('')

  lines.push('## Blog')
  for (const post of posts.sort((a, b) => b.data.publishedAt.getTime() - a.data.publishedAt.getTime())) {
    lines.push(`- [${post.data.title}](${SITE_URL}/blog/${post.id}/): ${post.data.description}`)
  }

  return new Response(lines.join('\n'), {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  })
}

This follows the llms.txt proposal’s expected format directly — an H1 with the site name, a blockquote summary, then H2 sections grouping content by type, each entry a Markdown link with a trailing description. Nothing exotic. The build output confirmed it instantly:

# Astro Content Lab

> A real content site built with Astro, MDX and Tailwind CSS.

## Blog

- [Hello World](https://astro-content-lab.vercel.app/blog/hello-world/): Every developer starts here. So do I.
- [Why I Moved From WordPress to Astro](...): After 10 years of building WordPress sites...

## Reviews

- [Hostinger Review 2026](...): Hostinger is one of the most popular budget hosting providers...

Sorted newest-first within each section, every entry pulled live from frontmatter. Publish a new review tomorrow, run the build, it shows up here without touching this file.

Building llms-full.txt

This one needed access to the actual markdown body, not just the frontmatter fields. Astro’s Content Collections expose this directly as entry.body — the raw markdown source string, no rendering pipeline required.

// src/pages/llms-full.txt.ts
import type { APIRoute } from 'astro'
import { getCollection } from 'astro:content'
import { SITE_NAME, SITE_URL, SITE_DESCRIPTION } from '../config'

export const GET: APIRoute = async () => {
  const publishedOnly = ({ data }) => data.status === 'published'
  const posts = await getCollection('posts', publishedOnly)
  const reviews = await getCollection('reviews', publishedOnly)
  const compares = await getCollection('compares', publishedOnly)
  const guides = await getCollection('guides', publishedOnly)

  const sections: string[] = []
  sections.push(`# ${SITE_NAME} — Full Content Export`)
  sections.push('')

  const allEntries = [
    ...posts.map((p) => ({ ...p, type: 'Blog', path: 'blog' })),
    ...reviews.map((r) => ({ ...r, type: 'Review', path: 'reviews' })),
    ...compares.map((c) => ({ ...c, type: 'Comparison', path: 'compares' })),
    ...guides.map((g) => ({ ...g, type: 'Guide', path: 'guides' })),
  ]

  for (const entry of allEntries) {
    sections.push(`## ${entry.data.title}`)
    sections.push(`**Type:** ${entry.type}`)
    sections.push(`**URL:** ${SITE_URL}/${entry.path}/${entry.id}/`)
    sections.push('')
    sections.push(entry.body ?? '')
    sections.push('---')
  }

  return new Response(sections.join('\n'), {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  })
}

entry.body is worth pausing on. Astro’s Content Collections expose the raw Markdown source, so there is no need to fetch rendered HTML and strip the tags. The published-only filter above is essential. Without it, a full-text export can expose drafts that were never meant to leave the repository.

The build output confirmed everything carried through correctly — code blocks, image alt text, headings, the works:

## Astro vs Next.js
**Type:** Comparison
**URL:** https://astro-content-lab.vercel.app/compares/astro-vs-nextjs/

Both are modern JavaScript frameworks. Both are popular. Both are well-maintained.
And they are built for fundamentally different purposes.

## How to Deploy Astro to Vercel
**Type:** Guide
**URL:** https://astro-content-lab.vercel.app/guides/how-to-deploy-astro/

Vercel is the easiest way to get an Astro site live. It connects to your GitHub repo,
builds automatically on every push, and gives you a live URL on the free tier.

Every content type, every field, intact.

Where This Connects to robots.txt

The previous article in this series covered building sitemap.xml and robots.txt by hand instead of relying on the official sitemap package. This is the natural next step in the same file — except llms.txt doesn’t actually belong in robots.txt the way the sitemap does.

A sitemap has a defined Sitemap directive in robots.txt. llms.txt does not. Adding a comment or invented directive would not grant permission, block a bot, or make a crawler support the proposal. I left robots.txt unchanged.

What’s Actually Running Now

astro-content-lab ships three machine-readable files, all generated the same way: as API routes reading live from Content Collections, none of them maintained by hand.

sitemap.xml — discoverable URLs with honest lastmod dates
llms.txt — a concise index for compatible tools
llms-full.txt — an optional full-content export

None of these required a plugin. None of them require remembering to update a static file after publishing new content. Publish a post, run the build, all three update.

Whether llms.txt becomes useful outside a small set of compatible tools is genuinely unclear. The useful lesson is not “add this file for GEO.” It is how to generate a clean text export from one content source without creating another document that quietly goes stale.

That distinction matters to me. I have installed enough WordPress SEO features because a dashboard turned the warning light red. I do not want to rebuild that habit in Astro with newer acronyms.

Part 4 starts with work that helps regardless of crawler fashion: clear answers, useful headings, and content written for a real reader.

Frequently Asked Questions

Does llms.txt improve Google or AI visibility?

Google explicitly says it does not use llms.txt for Search or its generative AI features, so the file neither improves nor harms Google visibility. The broader proposal is still experimental. Build it only for tools that choose to support it, not as a ranking or citation tactic.

What's the difference between llms.txt and llms-full.txt?

llms.txt is a short Markdown index — site name, description, and a linked list of pages with one-line summaries, meant for quick scanning. llms-full.txt contains the complete raw text of every page, meant for AI systems that need to ingest full content rather than summaries.

Why generate llms.txt with code instead of writing it by hand?

A hand-written file goes stale the moment you publish new content and forget to update it. Generating it from Content Collections at build time means it's always accurate — every new blog post, review, or guide appears automatically the next time the site builds.

Should llms.txt be referenced in robots.txt?

No standard robots.txt directive exists for llms.txt. Adding a comment or invented field does not make a crawler support the proposal, grant permission, or block access. Keep normal crawler controls in robots.txt and expose /llms.txt at the root only for compatible tools that choose to request it.

Is llms-full.txt part of an official web standard?

No. llms.txt is a community proposal, not a W3C or IETF standard, and llms-full.txt is an optional companion convention. Support varies by tool. Treat both files as experimental exports that may help a compatible consumer, not as crawler directives or replacements for normal HTML.