Skip to main content

robots.txt File Analysis

Issue No: 5

Category: Crawlability-and-indexation

Issue type: Issue

Priority: CRITICAL

Description​

A robots.txt file instructs web crawlers which parts of a site may or may not be accessed; misconfigurations can inadvertently block important pages from being crawled or indexed, impacting search visibility.

How do we capture it​

  1. Send an HTTP GET request to https://{site_domain}/robots.txt.
  2. Record the HTTP status code (200, 404, 500, etc.).
  3. Store the raw text body of the response.
  4. Parse all directives line-by-line: User-agent, Disallow, Allow, Sitemap, Crawl-delay.
  5. Record Last-Modified from the HTTP response headers if present.
  6. Compare against the site's known sitemap and crawl data to identify any blocked but important URLs.
  7. For each crawled HTML page, extract <link rel="canonical" href="..."> and evaluate whether that canonical URL's path is matched by any active Disallow rule. Flag if blocked.
  8. For each crawled HTML page, parse <meta name="robots" content="..."> and inspect the content attribute for: noindex, nofollow, max-snippet, max-image-preview, max-video-preview. Record all values.
  9. For each HTTP response (HTML and non-HTML), record the X-Robots-Tag response header value if present. For non-HTML resources (.pdf, .docx, .jpg, .png, .webp, etc.), perform an HTTP HEAD request and inspect the header.
  10. Combine per-page signals into a composite indexability verdict: HTTP 200 + not blocked by robots.txt + no noindex in meta tag + no noindex in X-Robots-Tag = indexable.

What to store​

FieldTypeComment
contenttextRaw unparsed robots.txt body as fetched
versionintegerIncremented each time a new fetch detects a change
fetched_attimestampWhen this robots.txt was last fetched
last_modifiedtimestampValue from HTTP Last-Modified response header
status_codeintegerHTTP status code returned (200, 404, 500, etc.)
statustextHuman-readable status label, e.g. "OK", "NOT_FOUND"
errortextError message if fetch or parse failed; null on success
is_activebooleanWhether this is the current active version for the site
user_agenttextTarget crawler: "*", "Googlebot", "Bingbot", etc.
directivetextRule type: "Allow", "Disallow", "Sitemap", "Crawl-delay"
pathtextURL path or pattern this rule applies to, e.g. /admin/
rule_orderintegerParse sequence order; used to resolve conflicting rules
http_statusintegerFinal HTTP status code after following redirects
is_blocked_by_robotsbooleanTrue if a matching Disallow rule exists for this page's path
canonical_urltextValue of <link rel="canonical" href="...">, null if absent
canonical_is_blockedbooleanTrue if the canonical target path is itself blocked by robots.txt
meta_robots_rawtextFull raw content attribute of <meta name="robots">, null if absent
has_meta_noindexbooleanTrue if noindex found in meta robots content
max_snippettextExtracted value, e.g. "-1", "150", null if absent
max_image_previewtextExtracted value: "none", "standard", "large", null if absent
max_video_previewtextExtracted value, e.g. "-1", "0", null if absent
x_robots_tag_valuetextRaw X-Robots-Tag response header value, null if absent
has_header_noindexbooleanTrue if noindex found in X-Robots-Tag header
is_indexablebooleanComposite: true only when all blocking signals are absent
checked_attimestampWhen the audit was performed
resource_urltextFull URL of the non-HTML resource
content_typetextMIME type from Content-Type header
is_in_sitemapbooleanWhether this resource URL appears in the XML sitemap

Condition for trigger​

These rules define when specific issues should be flagged during crawling.

  • Trigger Rule 1: Robots file missing
    • Target Field: status_code
    • Evaluation Logic: = 404
    • Severity: WARNING
    • Diagnostic Message: "robots.txt does not exist; crawlers will crawl everything by default — may expose unintended paths"
  • Trigger Rule 2: Server error reading robots
    • Target Field: status_code
    • Evaluation Logic: >= 500
    • Severity: CRITICAL
    • Diagnostic Message: "Server error when fetching robots.txt; crawlers may pause crawling the entire site"
  • Trigger Rule 3: Global disallow all
    • Target Field: directive
    • Evaluation Logic: directive = 'Disallow' AND path = '/' AND user_agent = '*'
    • Severity: CRITICAL
    • Diagnostic Message: "Entire site is blocked from all crawlers — nothing will be indexed"
  • Trigger Rule 4: Global Googlebot Block
    • Target Field: directive
    • Evaluation Logic: directive = 'Disallow' AND path = '/' AND user_agent = 'Googlebot'
    • Severity: CRITICAL
    • Diagnostic Message: "Entire site blocked specifically from Google"
  • Trigger Rule 5: Sitemap conflict
    • Target Field: directive
    • Evaluation Logic: directive = 'Disallow' AND path = sitemap_url
    • Severity: IMPORTANT
    • Diagnostic Message: "A URL in the XML sitemap is also Disallowed — a direct conflict"
  • Trigger Rule 6: Resource block conflict
    • Target Field: directive
    • Evaluation Logic: directive = 'Disallow' AND path matches CSS/JS
    • Severity: IMPORTANT
    • Diagnostic Message: "Render-blocking resources are disallowed; Google cannot render the page properly"
  • Trigger Rule 7: Sitemap missing
    • Target Field: content
    • Evaluation Logic: Sitemap string is missing
    • Severity: SUGGESTION
    • Diagnostic Message: "Sitemap not declared in robots.txt; crawlers must discover it independently"
  • Trigger Rule 8: Sensitive paths exposed in robots
    • Target Field: content
    • Evaluation Logic: Disallow string contains /admin, /private, or /backup
    • Severity: WARNING
    • Diagnostic Message: "Sensitive directory names are exposed in robots.txt, which is publicly visible"
  • Trigger Rule 9: Outdated robots file
    • Target Field: last_modified
    • Evaluation Logic: > 365 days
    • Severity: SUGGESTION
    • Diagnostic Message: "robots.txt has not been reviewed/updated in over a year"
  • Trigger Rule 10: New blocking rules discovered
    • Target Field: version
    • Evaluation Logic: version > 1 AND new disallows present
    • Severity: WARNING
    • Diagnostic Message: "New blocking rules detected compared to previously stored version — change alert"
  • Trigger Rule 11: Canonical is blocked
    • Target Field: canonical_is_blocked
    • Evaluation Logic: = true
    • Severity: CRITICAL
    • Diagnostic Message: "Canonical target is disallowed — Google cannot verify the canonical signal and may ignore it"
  • Trigger Rule 12: Broken critical page
    • Target Field: http_status
    • Evaluation Logic: != 200
    • Severity: CRITICAL
    • Diagnostic Message: "Page is unreachable or redirected; not indexable"
  • Trigger Rule 13: Meta noindex found
    • Target Field: has_meta_noindex
    • Evaluation Logic: = true on critical page
    • Severity: CRITICAL
    • Diagnostic Message: "meta robots noindex found on critical page — page excluded from index"
  • Trigger Rule 14: Header noindex found
    • Target Field: has_header_noindex
    • Evaluation Logic: = true
    • Severity: CRITICAL
    • Diagnostic Message: "X-Robots-Tag: noindex in HTTP response — overrides any on-page tag"
  • Trigger Rule 15: Warning on secondary noindex
    • Target Field: has_meta_noindex
    • Evaluation Logic: = true on non-critical page
    • Severity: IMPORTANT
    • Diagnostic Message: "Unintentional noindex on secondary content (blog, about, etc.)"
  • Trigger Rule 16: Missing X-Robots-Tag on private file
    • Target Field: x_robots_tag_value
    • Evaluation Logic: IS NULL AND content_type is pdf/image
    • Severity: WARNING
    • Diagnostic Message: "Private document or image may be indexed unintentionally"
  • Trigger Rule 17: Header noindex on sitemap resource
    • Target Field: is_in_sitemap
    • Evaluation Logic: = true AND has_header_noindex = true
    • Severity: WARNING
    • Diagnostic Message: "Sitemap declares the resource; header blocks it — contradictory signal"
  • Trigger Rule 18: Missing max-snippet
    • Target Field: max_snippet
    • Evaluation Logic: IS NULL
    • Severity: SUGGESTION
    • Diagnostic Message: "Google uses conservative default snippet length"
  • Trigger Rule 19: Missing large image preview
    • Target Field: max_image_preview
    • Evaluation Logic: != 'large'
    • Severity: SUGGESTION
    • Diagnostic Message: "Large image previews not enabled; affects Google Images and Discover eligibility"
  • Trigger Rule 20: Missing video preview
    • Target Field: max_video_preview
    • Evaluation Logic: IS NULL
    • Severity: SUGGESTION
    • Diagnostic Message: "Extended video preview clips not enabled"

Sources​

Long description​

A robots.txt file is a plain-text file placed at the root directory of a website (e.g., https://example.com/robots.txt) that follows the Robots Exclusion Protocol. It tells web crawlers — such as Googlebot, Bingbot, or any other automated bot — which URLs or sections of the site they are permitted to access and crawl.

The file is fetched by crawlers before they begin crawling a site. It is one of the first things any compliant web crawler checks. A robots.txt file consists of one or more "groups" of rules. Each group begins with one or more User-agent lines (identifying which crawlers the rules apply to), followed by Disallow and/or Allow directives.

Supported directives:

  • User-agent — Specifies the bot the following rules apply to. * means all crawlers.
  • Disallow — Instructs the crawler not to access the specified path. An empty value means "allow all."
  • Allow — Explicitly permits access to a path (overrides a broader Disallow).
  • Sitemap — Points crawlers to the XML sitemap location.
  • Crawl-delay — Asks crawlers to wait a specified number of seconds between requests (not supported by Google).

When both Allow and Disallow rules match a URL, the most specific rule wins. If two rules have equal specificity, Allow takes precedence over Disallow (per Google's implementation).

What robots.txt Does NOT Do:

  1. It is not a security tool. Any human or non-compliant bot can access blocked URLs directly. The file is publicly visible, and listing sensitive paths in it can actually expose those paths to bad actors.
  2. Blocked URLs can still appear in search results. If other pages link to a disallowed URL, Google can discover and index the URL (without a snippet) even if it cannot crawl it. To prevent a page from appearing in search results, use noindex meta tags or HTTP response headers instead.
  3. It does not prevent caching. CDNs or third-party caches may still serve blocked content.

A page being indexable requires all four of the following to be true simultaneously. Each layer is independently capable of blocking indexation:

  1. HTTP 200 — The page must be reachable. A 301/302 redirect chain must ultimately resolve to a 200.
  2. Not blocked by robots.txt — The page's path must not match any active Disallow rule for User-agent: * or User-agent: Googlebot.
  3. No <meta name="robots" content="noindex"> — An on-page noindex tag tells the indexer to skip the page after crawling it.
  4. No X-Robots-Tag: noindex in the HTTP response header — takes precedence over the on-page tag if both are present.

How to Fix​

  • Do not block the entire site: Remove any Disallow: / rule under User-agent: * on production sites. If the site is in development, restrict access at the server/firewall level instead.
  • Do not use robots.txt to hide sensitive content: Remove sensitive directory paths from robots.txt and protect them using proper authentication.
  • Allow CSS, JS, and render-critical resources: Ensure all resources needed for page rendering are accessible to search engine bots.
  • Resolve sitemap conflicts: Audit the XML sitemap against robots.txt rules. Any URL present in the sitemap should not be disallowed in robots.txt.
  • Declare your sitemap: Add a Sitemap: directive pointing to your XML sitemap inside robots.txt.
  • Use noindex instead of Disallow to remove pages from search results: Use a meta robots tag on the page itself or return an X-Robots-Tag: noindex HTTP response header. Do not rely on Disallow for this purpose.
  • Keep robots.txt updated: Review robots.txt whenever the site's URL structure changes significantly.
  • Ensure canonical targets are not blocked by robots.txt: Any canonical target that matches a Disallow rule must be unblocked.
  • Audit all critical pages for composite indexability: Run a full indexability check on all URLs listed in the XML sitemap checking for conflicting noindex directives or status codes.
  • Set X-Robots-Tag on non-HTML resources: At the server level, apply X-Robots-Tag: noindex, nofollow to private/sensitive generic files such as PDFs and max-image-preview:large to large, discoverable images.
  • Add snippet and preview directives: Add max-snippet, max-image-preview, and max-video-preview instructions to page headers globally to maximize rich results CTR.