Skip to main content

Crawl-delay directive set where server needs rate limiting

Issue No: 165

Category: Crawl Behaviour

Issue type: Suggestion

Priority: STANDARD

Description​

This audit verifies that a crawl-delay directive is declared when a site is explicitly marked as requiring crawler-side rate limiting. It helps reduce request burst pressure and supports stable crawl throughput.

How do we capture it​

  1. Read project-level crawl policy input for the target host to determine whether server-side protection requires slower bot request pacing. This must come from explicit configuration, not heuristic inference.
  2. For each audited host, request the robots file at /robots.txt using HTTP GET.
  3. Record HTTP status code, content type, response size, and response bytes for diagnostics.
  4. If response is successful and text-based, parse robots directives line-by-line according to robots grammar conventions:
    • Identify User-agent groups.
    • Capture Crawl-delay values within each group.
    • Preserve original token values for traceability.
  5. Normalize parsed values:
    • Trim whitespace.
    • Convert decimal tokens to numeric seconds when valid.
    • Reject negative, zero, or non-numeric values as invalid.
  6. Determine effective crawl-delay for the applicable bot policy by checking the most specific matching User-agent group first, then fallback to wildcard group when applicable.
  7. Mark compliance only when all required conditions are true:
    • Server rate-limiting requirement flag is true.
    • robots.txt is fetchable and parseable.
    • Effective crawl-delay exists and is a valid positive numeric value.
  8. Mark non-compliance when required flag is true and crawl-delay is missing, malformed, or non-positive.
  9. Record parser and fetch failures distinctly so operational issues are separable from policy issues.
  10. Constraint: this check is based on raw HTTP responses and robots text only; no browser rendering or JavaScript execution is used.

What to store​

FieldTypeComment
host TEXTTEXTAudited host/domain for robots crawl-delay validation.
rate_limit_required BOOLEANBOOLEANExplicit configuration flag indicating crawl-delay is required for this host.
robots_status_code INTEGERINTEGERHTTP status code returned by robots.txt request.
has_robots_file BOOLEANBOOLEANTrue when robots.txt is successfully retrieved.
effective_user_agent TEXTTEXTUser-agent scope used to resolve crawl-delay value.
crawl_delay_raw TEXTTEXTRaw crawl-delay token as parsed from robots.txt.
crawl_delay_seconds NUMERICNUMERICNormalized positive crawl-delay value in seconds when valid.
has_valid_crawl_delay BOOLEANBOOLEANTrue when effective crawl-delay exists and is valid positive numeric value.
crawl_delay_violation BOOLEANBOOLEANTrue when crawl-delay is required but missing or invalid.
parser_error_code TEXTTEXTParser or fetch error classification such as FETCH_FAILED or INVALID_VALUE.
checked_at TIMESTAMPTZTIMESTAMPTZTimestamp when crawl-delay audit completed.

Condition for trigger​

A trigger is raised only when the host is explicitly marked as requiring rate limiting and crawl-delay policy is absent or invalid.

  • Trigger Rule 1: Required Crawl-delay Missing

    • Target Field: crawl_delay_violation
    • Evaluation Logic: rate_limit_required = TRUE AND has_valid_crawl_delay = FALSE
    • Severity: SUGGESTION
    • Diagnostic Message: "This host is configured as rate-limit sensitive, but robots.txt does not provide a valid crawl-delay directive."

  • Trigger Rule 2: robots.txt Unavailable When Delay Required

    • Target Field: robots_status_code
    • Evaluation Logic: rate_limit_required = TRUE AND (has_robots_file = FALSE OR robots_status_code >= 400)
    • Severity: SUGGESTION
    • Diagnostic Message: "Rate-limit-sensitive host does not expose a usable robots.txt for crawl-delay policy declaration."

Sources​

Long description​

Rate-limiting policies exist to prevent crawler traffic spikes from saturating origin resources, especially on constrained infrastructure or burst-sensitive endpoints. The crawl-delay directive can communicate desired pacing to crawlers that support it, reducing concurrent request pressure and smoothing fetch distribution over time.

Common failure scenarios checked by the crawler:

  • Hosts marked as rate-limit sensitive with no crawl-delay directive in robots.txt.
  • Crawl-delay present but malformed or non-numeric.
  • Directive defined only in unrelated User-agent blocks and not effective for the evaluated bot policy.
  • robots.txt fetch failures preventing policy retrieval.
  • Zero or negative delay values that do not represent valid throttling.

Support for crawl-delay varies by crawler implementation, so this audit is advisory and policy-oriented. It is still useful for documenting intent, aligning crawl governance, and reducing avoidable load where supported.

How to Fix​

For hosts that require reduced crawl pressure, publish a valid robots.txt with an explicit positive crawl-delay value in the correct User-agent scope and ensure the file is consistently reachable. Align the declared value with infrastructure capacity and monitor server load after deployment to confirm stable crawl behavior.