Skip to main content

llms.txt file created at /llms.txt (AI model content manifest)

Issue No: 186

Category: GEO & AI Engine Signals

Issue type: Suggestion

Priority: IMPORTANT

Description​

Verifies the presence and correct delivery of an llms.txt file at the root directory (/llms.txt) of the website. This file serves as a machine-readable directory and content map optimized for discovery and citation by Large Language Models (LLMs) and generative search agents.

How do we capture it​

  1. The crawler constructs a target URL by appending /llms.txt to the verified root origin (e.g., https://example.com/llms.txt).
  2. The crawler issues an HTTP GET request using standard browser-imitation headers (specifically mimicking an AI crawler/user-agent or a standard user-agent) with a configured timeout (default: 10 seconds).
  3. The crawler follows any HTTP redirects (301, 302, 307, 308) up to a maximum of 5 hops, logging both the redirect paths and the final destination URL.
  4. If the final HTTP response status code is 200 OK, the crawler performs the following validation:
    • Inspects the Content-Type response header to verify it contains text/plain.
    • Downloads the response body bytes.
    • Decodes the payload using UTF-8 encoding.
    • Checks that the body is not empty and contains Markdown-formatted plain text.
  5. If the request results in a non-200 status code, a connection timeout, DNS resolution failure, or SSL error, the crawler registers the file as missing or inaccessible.

What to store​

FieldTypeComment
llms_txt_status_codeINTEGERThe final HTTP response status code returned by the server when requesting /llms.txt.
llms_txt_contentTEXTThe raw UTF-8 text contents of the /llms.txt file (truncated to 50KB in the database to prevent bloat).
llms_txt_final_urlVARCHAR(2048)The final destination URL from which the /llms.txt file was retrieved after resolving redirects.
llms_txt_has_valid_h1BOOLEANTrue if the retrieved Markdown content starts with exactly one H1 header (# Project Name), as required by the spec.

Condition for trigger​

The triggering evaluation uses the captured database fields to evaluate whether the site has successfully implemented and exposed the llms.txt specification.

  • Trigger Rule 1: Missing llms.txt File

    • Target Field: llms_txt_status_code
    • Evaluation Logic: llms_txt_status_code != 200
    • Severity: SUGGESTION
    • Diagnostic Message: "No llms.txt file was found at the root directory of your domain (/llms.txt). Creating an AI content manifest helps LLMs, AI agents, and Generative Search Engines accurately discover, summarize, and cite your site content."

  • Trigger Rule 2: Empty or Non-Conformant Content Structure

    • Target Field: llms_txt_has_valid_h1
    • Evaluation Logic: llms_txt_status_code = 200 AND llms_txt_has_valid_h1 = FALSE
    • Severity: WARNING
    • Diagnostic Message: "The /llms.txt file was found but does not conform to the standard Markdown format structure. The specification requires the file to start with an H1 heading representing the project or business name."

Sources​

Long description​

The llms.txt file is an emerging, community-driven standard designed to streamline how Large Language Models (LLMs), AI crawlers, and retrieval-augmented generation (RAG) agents index and understand website content. Traditional web search engines rely on complex XML sitemaps and raw HTML scraping. However, converting highly nested, interactive, and ad-laden HTML pages into clean context is computationally expensive and prone to parsing errors.

Because LLMs have strict context window limits, indexing a large site by pulling random HTML pages often introduces irrelevant noise, leading to hallucinations. The /llms.txt file solves this by providing a clean, flattened Markdown index at the root of a domain. It lists curated links to high-quality Markdown or clean HTML resources (such as product specs, documentation, or FAQs) along with brief context descriptions.

This crawler test checks for the existence of this file. Common implementation errors checked by the crawler include:

  • The Soft 404: Web servers redirecting the request to /llms.txt back to the homepage with a 200 OK code instead of throwing a clean 404 Not Found.
  • Incorrect MIME Types: Serving the file as text/html or application/octet-stream, which prevents AI agents from recognizing it as raw, structured plain text.
  • Blank or Placeholder Files: Serving a file that is empty or populated with auto-generated generic template code that lacks the mandatory # Site Title H1 marker.

How to Fix​

Create a raw plain-text file and save it as llms.txt. Format the file using standard Markdown syntax according to the specification:

  • Start with a single # [Project or Business Name] as the H1 header.
  • Follow with a blockquote > [Brief, precise summary of your site's scope and purpose].
  • Use H2 headers (##) to organize groups of high-priority page links.
  • Add descriptive link entries to highlight your key content, using the format - Page Title: A brief description of this resource for LLMs. Upload the file to the root directory of your web host so that it is publicly accessible at https://yourdomain.com/llms.txt. Configure your web server (Nginx, Apache, IIS, or CDN rules) to serve /llms.txt with the HTTP response header Content-Type: text/plain; charset=utf-8.