Skip to main content

llms.txt Missing or Misconfigured

Issue No: 186

Category: AI Crawlability & Discoverability

Issue type: Suggestion

Priority: STANDARD

Description​

The site is missing an llms.txt file at its root domain, or the file exists but is misconfigured — wrong format, broken links, file too large, or not accessible at the correct path. This file helps AI language models discover and understand a site's key content in a structured, LLM-friendly format.

How do we capture it​

  1. Send a GET request to {site_root}/llms.txt.
  2. Record the HTTP status code from the response.
  3. Check the Content-Type header to ensure it's plain text and not HTML/JSON.
  4. If the HTTP status is 200, parse the raw response body using a Markdown parser to validate its structure, checking for an H1 heading (#), a blockquote description (>), section headings (##), and links.
  5. Extract all listed URLs from the Markdown and send HEAD requests to each to validate they return a 200 OK status.
  6. Check the total byte size of the raw HTTP response body to ensure it does not exceed the 100KB threshold.
  7. Request the homepage HTML and parse the raw <head> tag looking for a exact match of <link type="text/plain" rel="llms" href="/llms.txt" />. (Note: Relies strictly on raw HTTP response, does not execute JS).

What to store​

FieldTypeComment
is_presentBOOLEANIndicates if the file exists and returns HTTP 200
http_statusINTEGERHTTP response code (200, 404, 500 etc.)
is_valid_markdownBOOLEANTrue if content is plain text Markdown (not HTML/JSON)
has_h1_titleBOOLEANTrue if the file starts with a # H1 heading
has_blockquote_descBOOLEANTrue if the file has a > blockquote description line
has_sectionsBOOLEANTrue if the file has at least one ## section
link_countINTEGERTotal number of links listed in the file
broken_link_countINTEGERNumber of listed URLs returning a non-200 status code
file_size_bytesINTEGERThe physical size of the file in bytes
is_too_largeBOOLEANTrue if the file exceeds the 100KB threshold limitation
has_discovery_tagBOOLEANTrue if <link rel="llms"> is found in the homepage <head>
has_full_txtBOOLEANTrue if llms-full.txt is also present at the site root

Condition for trigger​

The issue triggers based on the presence, status, and correct formatting of the llms.txt file.

  • Trigger Rule 1: File missing

    • Target Field: is_present
    • Evaluation Logic: is_present = FALSE
    • Severity: STANDARD
    • Diagnostic Message: "No llms.txt file was found at the root URL."

  • Trigger Rule 2: Server error

    • Target Field: http_status
    • Evaluation Logic: http_status >= 500
    • Severity: IMPORTANT
    • Diagnostic Message: "The server returned a 5xx error when fetching llms.txt."

  • Trigger Rule 3: Wrong format

    • Target Field: is_valid_markdown
    • Evaluation Logic: is_present = TRUE AND is_valid_markdown = FALSE
    • Severity: IMPORTANT
    • Diagnostic Message: "The llms.txt file is not formatted as plain Markdown (e.g., returned HTML/JSON)."

  • Trigger Rule 4: Missing H1 title

    • Target Field: has_h1_title
    • Evaluation Logic: has_h1_title = FALSE
    • Severity: STANDARD
    • Diagnostic Message: "The llms.txt file is missing a required '#' heading at the top."

  • Trigger Rule 5: Missing description

    • Target Field: has_blockquote_desc
    • Evaluation Logic: has_blockquote_desc = FALSE
    • Severity: STANDARD
    • Diagnostic Message: "The llms.txt file is missing a '>' blockquote introduction."

  • Trigger Rule 6: No sections

    • Target Field: has_sections
    • Evaluation Logic: has_sections = FALSE
    • Severity: STANDARD
    • Diagnostic Message: "The llms.txt file is missing '##' section headings."

  • Trigger Rule 7: Broken links

    • Target Field: broken_link_count
    • Evaluation Logic: broken_link_count > 0
    • Severity: IMPORTANT
    • Diagnostic Message: "The llms.txt file contains URLs that return 404 or other non-200 status codes."

  • Trigger Rule 8: File too large

    • Target Field: is_too_large
    • Evaluation Logic: is_too_large = TRUE
    • Severity: STANDARD
    • Diagnostic Message: "The llms.txt file exceeds the 100KB size limit."

  • Trigger Rule 9: No discovery tag

    • Target Field: has_discovery_tag
    • Evaluation Logic: has_discovery_tag = FALSE
    • Severity: STANDARD
    • Diagnostic Message: "The homepage <head> is missing the <link rel=\"llms\"> tag."

Sources​

Long description​

What is llms.txt?​

llms.txt is a plain-text Markdown file placed at the root of a website (e.g. https://example.com/llms.txt). It was proposed to help websites communicate their key content structure to AI language models in a clean, concise format. The motivation is pragmatic: LLM context windows are finite. When an AI model tries to ingest a full website — with HTML boilerplate, navigation menus, and thousands of pages — it cannot efficiently identify which content matters most. llms.txt solves this by providing a curated index: the most important pages, grouped by topic, with plain-language descriptions.

Relationship to Other Crawl Files​

Unlike robots.txt which is an official, enforced standard for search engine crawlers, or sitemap.xml for indexers, llms.txt is an index of key pages specifically designed for AI language models. It is a proposed standard with no enforcement mechanism, but it is increasingly adopted by technical sites and platforms.

The Specification Format​

The file must follow a strict Markdown structure:

  • File name must be exactly llms.txt.
  • Must be accessible at the root path.
  • Must return Content-Type: text/plain with HTTP 200 OK.
  • Must begin with a # H1 heading (site name).
  • Should have a > blockquote description on the second non-empty line.
  • Sections use ## H2 headings.
  • Links use standard Markdown: - [Label](URL): description.
  • Must NOT be HTML, JSON, or any other format.

Duplicate Content Risk with llms-full.txt​

If implementing llms-full.txt, there is a risk of duplicate content at scale if the full-content Markdown pages are publicly crawlable without a noindex directive, potentially suppressing rankings for original canonical pages. It should be appropriately handled with robots.txt or meta tags.

How to Fix​

  1. Create a plain text file named exactly llms.txt at the root of your domain.
  2. Structure the file using plain Markdown starting with a # H1 heading for the site name.
  3. Add a > blockquote description outlining what the site does.
  4. Add ## H2 headings for different sections (e.g., Product, Docs).
  5. Add links to key pages formatted as Markdown links (- [Title](URL): Description).
  6. Ensure the file is served with a 200 OK status and the Content-Type: text/plain header.
  7. Include a <link type="text/plain" rel="llms" href="/llms.txt" /> tag in the <head> of your homepage.
  8. Verify that there are no broken links in the file by confirming each linked URL returns a 200 OK status code.
  9. Keep the file size below 50KB to ensure fast processing and respect LLM limits.