llms.txt in Next.js: What Reads It and What Does Not

llms.txt is a proposed markdown index at your domain root that lists the pages you want a language model to read. Nothing is obliged to fetch it. Google has not endorsed it, John Mueller has said no AI system he knows of uses it, and an Ahrefs log study of 137,210 domains found 97% of existing llms.txt files received zero requests in a month. I ship one anyway. The version live on this site today is a hand-written public/llms.txt, which is how it ended up recommending two articles that return 404. This post covers the proposal, the evidence, the route handler that replaces the static file, and the check that caught the dead links.

What the file actually proposes

The proposal is Jeremy Howard's, first published in September 2024 and maintained at llmstxt.org. The argument is narrow and reasonable: a model's context window cannot hold a whole site, and converting navigation-heavy HTML into clean text is lossy. So give it a curated map instead.

The format

Reading the spec on 6 September 2026, the shape is:

1. —An H1 with the site or project name. This is the only required part.
2. —A blockquote with a short summary.
3. —Optional paragraphs of context.
4. —H2 sections, each followed by a markdown list of links with optional descriptions.
5. —By convention, an Optional section for things a model can skip.

That is it. It is a sitemap written in markdown with editorial judgement applied. The judgement is the point — the file says these pages, in this order, instead of handing over every URL in the sitemap.

llms-full.txt

A companion convention concatenates whole pages into one large file. Perplexity's docs publish one at docs.perplexity.ai/llms-full.txt. I did not find llms-full.txt defined in the llmstxt.org spec text when I read it, so treat it as community practice rather than part of the standard.

What the evidence says

This is the part most posts on this topic skip.

Google

John Mueller has been consistent and blunt. In June 2025: "FWIW no AI system currently uses llms.txt". In January 2026, asked whether Google properties shipping the file counted as an endorsement: "I'm tempted to say something snarky since this has come up so often, but to be direct, no". More recently he called the file purely speculative, adding that the thing genuinely worth doing is not blocking agents at all.

The log study

Ahrefs published the strongest evidence I found: an analysis of 137,210 domains by Louise Linehan, June 2026. The headline numbers:

• —28% of those domains (38,360) publish a valid llms.txt.
• —97% of those files received zero requests during May 2026.
• —Of the 3% that got any traffic, the largest categories were SEO audit tools (21.7%) and unidentified bots (14.9%). GPTBot accounted for 4.51%, ClaudeBot 0.8%, OAI-SearchBot 0.74%.
• —No AI bot requested llms.txt on domains where the file did not exist. They do not probe for it.

That last point is what changed my thinking. A crawler that speculatively checks /llms.txt on every domain would make the file a discovery mechanism. A crawler that only fetches it when a human pastes the URL makes it a document you hand out.

The tracking study

Search Engine Land ran a smaller before-and-after: Ana Fernández tracked 10 sites across a 180-day window, published January 2026. Eight showed no measurable change after adding the file. The two that grew grew for other reasons — PR coverage in one case, 27 new downloadable templates in the other. Her own summary is the line worth keeping: llms.txt is "useful infrastructure, not a growth lever."

Publishing one is not reading one

Anthropic, Perplexity and a long list of developer-tool companies publish llms.txt files. That is a statement about their documentation, not about their retrieval pipeline. I could not find a claim from any major provider that production retrieval parses a third-party llms.txt. When a listicle says "Perplexity supports llms.txt" and the evidence offered is that Perplexity has one, the sentence has quietly swapped publishing for consuming.

Why I ship one anyway

Three honest reasons, none of them "it will get me cited":

It costs one route file. The generated version below is 37 lines. At that price I do not need it to work, I need it to not be wrong.

It is the URL I paste. When someone asks Claude or ChatGPT to look at my site, www.santoshpaudel.me/llms.txt is one fetch instead of a model crawling my blog index and guessing. This is the directed case the Ahrefs data describes, and it is a real case — it happens to me most weeks. The catch is that handing out a file only helps if the file is true, and mine was not.

It forces an editorial decision. Writing down which twenty pages matter is a useful exercise after cutting the published count from 267 posts to 103. That cut is documented in my content pruning scoring model and the 389-pages, 22-clicks teardown.

What I do not claim: that any of this moved a ranking, an impression or a citation. I have no data showing that, and I could not find anyone who does.

The Next.js implementation

What is live on this site today is a hand-written public/llms.txt. That is the wrong place, for a reason the dead-link section below makes concrete. Below is the replacement route and the two changes to existing files it depends on. Neither of those two files has the shape the route needs yet, so read this as the diff to apply, not as something already merged.

Delete the static file first

public/llms.txt and app/llms.txt/route.ts both claim the path /llms.txt. In the Pages Router, Next.js documents that collision as an error it tells you about. The App Router docs do not say which one wins, and production is a bad place to find out — so delete the static file in the same commit that adds the route, and there is nothing left to resolve.

The route handler

// app/llms.txt/route.ts
import { SITE_URL } from '@/lib/constants'
import { getBlogSlugs, getCaseStudySlugs } from '@/lib/slugs'
import { SERVICE_SLUGS } from '@/lib/service-slugs'

export const dynamic = 'force-static'
export const revalidate = 86400

const label = (slug: string) => slug.replace(/-/g, ' ')

export async function GET() {
  const [posts, studies] = await Promise.all([getBlogSlugs(), getCaseStudySlugs()])

  const body = [
    '# Santosh Paudel',
    '',
    '> Marketing strategist in Kathmandu. Build logs and worked arithmetic:',
    '> promo economics, content systems, and the Next.js stack this site runs on.',
    '',
    '## Services',
    ...SERVICE_SLUGS.map((s) => `- [${label(s)}](${SITE_URL}/services/${s})`),
    '',
    '## Case studies',
    ...studies.map((c) => `- [${label(c.slug)}](${SITE_URL}/case-studies/${c.slug})`),
    '',
    '## Blog',
    ...posts.map((p) => `- [${p.title}](${SITE_URL}/blog/${p.slug})`),
    '',
  ].join('\n')

  return new Response(body, {
    headers: {
      'content-type': 'text/plain; charset=utf-8',
      'x-robots-tag': 'noindex',
    },
  })
}

Two supporting changes, both required before that file type-checks.

First, lib/slugs.ts. It already runs the published-posts query for the sitemap and for the blog route's generateStaticParams, but it selects two columns and its return type says so, which means p.title above does not exist yet:

// lib/slugs.ts — as it stands
export async function getBlogSlugs(): Promise<{ slug: string; published_at: string | null }[]> {
  const { data } = await supabase
    .from("blog_posts")
    .select("slug, published_at")        // add title here
    .eq("published", true)

Widen the select to "slug, title, published_at" and widen the return type to match. The sitemap ignores the extra field.

Second, SERVICE_SLUGS. It is currently a local const at the top of app/sitemap.ts and is not exported, so @/lib/service-slugs does not resolve. Move the array into lib/service-slugs.ts and import it from both files. That list has already caused a problem once: four service pages were live but missing from the sitemap because the array was maintained by hand in exactly one place. There is a comment above it in app/sitemap.ts recording that.

Why sharing the source matters more than the format

A hand-written llms.txt is a second copy of your URL inventory, and second copies rot. Mine did. A generated file cannot list an unpublished post, because the query filters on published = true — the same filter the page itself uses, which is the whole reason that shared helper exists.

About the noindex header

Mueller suggested noindexing the file in July 2025, on the grounds that other sites might link to it and a user landing on a plain-text index is a bad search result. That is why x-robots-tag: noindex is on the response. It costs nothing, and I would rather the file not compete with the pages it points at.

The agent-readable surface, honestly

The column that matters is the last one. Four of these six have observable evidence of being consumed. One has evidence of being ignored at a 97% rate. That ordering should decide where your afternoon goes, which is also the argument in what schema AI agents actually parse.

Checking the file is not lying

A curated list of links is only worth publishing if the links resolve. Here is the check, no dependencies:

// scripts/check-llms-txt.ts
const SITE = process.argv[2] ?? 'https://www.santoshpaudel.me'

const txt = await fetch(`${SITE}/llms.txt`).then((r) => r.text())

// Catches markdown links and the bare paths my hand-written file used.
const raw = txt.match(/https?:\/\/[^\s)]+|(?<![\w:])\/[a-z0-9][a-z0-9/-]*/g) ?? []
const urls = [...new Set(raw.map((u) => new URL(u, SITE).href))]

const dead: string[] = []
for (const url of urls) {
  const res = await fetch(url, { redirect: 'follow' })
  if (!res.ok) dead.push(`${res.status}  ${url}`)
}

console.log(`${urls.length} links checked, ${dead.length} dead`)
dead.forEach((d) => console.log(d))
process.exit(dead.length ? 1 : 0)

The default host is the www one because that is what SITE_URL in lib/constants.ts is set to. Check the hostname your canonicals actually use — auditing the other one is how you get a clean report about a site nobody visits.

I ran this against the hand-written file on 6 September 2026. Result:

32 links checked, 3 dead
404  https://www.santoshpaudel.me/blog/personal-brand-framework
404  https://www.santoshpaudel.me/blog/digital-marketing-nepal-ai-playbook
999  https://www.linkedin.com/in/santoshpaudel7211/

The third one is a false positive worth knowing about: LinkedIn answers unauthenticated bot requests with HTTP 999, a non-standard status that means "no". The profile is fine. Every checker you write against real URLs will need an exception list, and LinkedIn is usually the first entry on it.

The other two are real. Both sat under a heading called Notable articles. Both were unpublished during the September prune that cut the published count from 267 posts to 103. The static file had no idea. Two of the five articles I was explicitly recommending to language models were 404s, and I only knew because I ran the eighteen-line script above.

That is the same failure mode as the 32 broken internal links that went live after that prune — 10 of them pointing at slugs the prune had unpublished, the other 22 at posts from a seed file that had never actually been run — and the /for route that returned a 404 while twenty persona pages linked into it. Link rot after a large content change is the normal outcome. If you have just cut or restructured a corpus and nothing has crawled it since, a content audit finds these before a crawler does.

FAQ

Does Google use llms.txt?

There is no evidence that it does, and Google has never announced support. Asked in January 2026 whether Google properties publishing the file amounted to an endorsement, John Mueller said no. In June 2025 he said no AI system he was aware of used llms.txt at all, and by mid-2026 he was still calling it purely speculative. He has also compared the file to the meta keywords tag, which is not a compliment.

Do I need llms.txt if I already have a sitemap?

Not for discovery — the sitemap does that job and search engines actually fetch it. The llms.txt file does something different: it says which pages matter and in what order. If you never hand the URL to anyone, it will most likely sit unread, which is what happened to 97% of the files in the Ahrefs sample.

Where do I put llms.txt in a Next.js app?

Either public/llms.txt for a static file, or app/llms.txt/route.ts for a generated one. Both answer the same path, so ship exactly one of them: delete the static file in the commit that adds the route. Use the route handler if you want the contents to track your real published set.

Should llms.txt be noindexed?

Mueller suggested it makes sense, since inbound links can get the file indexed and a user landing on a plain-text index is a poor result. One x-robots-tag: noindex response header handles it.

The short version

Ship it because it is cheap and because it is a good URL to paste, not because a crawler is waiting for it. Generate it from the same query as your sitemap so it cannot drift. Check the links on a schedule. Then spend the rest of the afternoon on the surface every one of these bots actually fetches, which is the page itself.

Just restructured your site and not sure what still resolves? A content audit checks the machine-readable surfaces alongside the pages, which is how the two dead links above turned up. Book a content audit or get in touch.