Skip to main content

www vs non-www redirects correct

Issue No: 10

Category: Crawl Behaviour

Issue type: Issue

Priority: CRITICAL

Description​

Verifies that both hostname variants (www and non-www) resolve to a single canonical host through consistent redirects. This prevents duplicate crawl paths and index fragmentation caused by host-level URL duplication.

How do we capture it​

  1. Build two test URLs for the same root host and scheme scope:
    • Variant A: https://example.com/
    • Variant B: https://www.example.com/ Use the audited domain as the base and preserve HTTPS as the primary protocol for checks.
  2. Send independent HTTP GET requests for both variants with redirects disabled on first request (maxRedirects = 0) to capture raw status and Location header.
  3. If either response is a redirect (3xx), follow redirects manually hop-by-hop up to a fixed limit (for example, 10 hops), recording each intermediate URL, status code, and Location value.
  4. Normalize each observed URL in the chain for comparison:
    • Lowercase scheme and host.
    • Remove default ports (:80, :443).
    • Preserve path, query, and fragment as received.
    • Normalize trailing slash only for root-path canonical comparison.
  5. Determine final landing URL for each variant after redirect following completes or stops on non-redirect response.
  6. Evaluate canonical host consistency:
    • Pass condition requires both variant requests to end on the same final canonical host and equivalent canonical URL target.
    • Redirect loops, missing Location on 3xx, timeout, DNS failure, TLS failure, or hop-limit overflow are capture failures.
  7. Record chain diagnostics for each variant, including total hops and terminal status.
  8. Constraint: crawler logic is based only on HTTP responses/headers and final resolved URLs from requests. No browser rendering or JavaScript execution is involved.

What to store​

FieldTypeComment
host_redirect_check_target TEXTTEXTBase audited host used to generate www/non-www variant requests.
non_www_test_url TEXTTEXTFull non-www variant URL requested by the crawler.
www_test_url TEXTTEXTFull www variant URL requested by the crawler.
non_www_initial_status_code INTEGERINTEGERFirst HTTP status code returned for non-www variant before manual redirect follow.
www_initial_status_code INTEGERINTEGERFirst HTTP status code returned for www variant before manual redirect follow.
non_www_redirect_chain JSONBJSONBOrdered hop records for non-www request with URL, status, and location per hop.
www_redirect_chain JSONBJSONBOrdered hop records for www request with URL, status, and location per hop.
non_www_final_url TEXTTEXTFinal resolved URL for non-www variant after redirect traversal.
www_final_url TEXTTEXTFinal resolved URL for www variant after redirect traversal.
non_www_final_host TEXTTEXTFinal resolved hostname for non-www variant.
www_final_host TEXTTEXTFinal resolved hostname for www variant.
non_www_hop_count INTEGERINTEGERNumber of redirect hops followed for non-www variant.
www_hop_count INTEGERINTEGERNumber of redirect hops followed for www variant.
non_www_fetch_error TEXTTEXTNetwork/protocol error text for non-www request when fetch or chain resolution fails.
www_fetch_error TEXTTEXTNetwork/protocol error text for www request when fetch or chain resolution fails.
is_www_non_www_redirect_consistent BOOLEANBOOLEANTrue when both variants converge to the same canonical destination host/URL.
canonical_redirect_target_host TEXTTEXTCanonical host selected by redirect behavior when consistency check passes.
checked_at TIMESTAMPTZTIMESTAMPTZTimestamp when redirect consistency audit was executed.

Condition for trigger​

The crawler should raise this issue when host canonicalization between www and non-www is missing, inconsistent, or technically broken.

  • Trigger Rule 1: Variants Do Not Converge

    • Target Field: is_www_non_www_redirect_consistent
    • Evaluation Logic: is_www_non_www_redirect_consistent = FALSE
    • Severity: CRITICAL
    • Diagnostic Message: "www and non-www versions do not resolve to one canonical destination, causing duplicate URL hosts."

  • Trigger Rule 2: Non-www Variant Fetch Failed

    • Target Field: non_www_fetch_error
    • Evaluation Logic: non_www_fetch_error IS NOT NULL AND non_www_fetch_error <> ''
    • Severity: CRITICAL
    • Diagnostic Message: "The non-www variant could not be reliably resolved due to a network or protocol failure."

  • Trigger Rule 3: www Variant Fetch Failed

    • Target Field: www_fetch_error
    • Evaluation Logic: www_fetch_error IS NOT NULL AND www_fetch_error <> ''
    • Severity: CRITICAL
    • Diagnostic Message: "The www variant could not be reliably resolved due to a network or protocol failure."

Sources​

Long description​

Search engines treat hostnames as distinct URL spaces unless explicit canonicalization is enforced. If www and non-www versions are both reachable without deterministic consolidation, crawlers can discover duplicate pages under separate hosts, splitting crawl budget and link equity.

A robust host canonicalization check requires network-level validation of both variants because redirect intent must be proven through actual response status codes and Location headers, not assumptions. The crawler should inspect redirect chains end-to-end and confirm both entry points terminate at one canonical destination.

Common failure scenarios checked by the crawler:

  • One variant returns 200 OK while the other redirects, leaving both hosts indexable.
  • Both variants redirect, but to different final hosts or different canonical URL paths.
  • Redirect loops occur between hosts or within chain hops.
  • Redirect responses are malformed (for example, 3xx without valid Location).
  • Fetch failures prevent reliable determination of canonical host behavior.

How to Fix​

Choose one canonical host strategy (www or non-www) and enforce it consistently at the edge/web server layer for all URLs. Ensure every request to the non-canonical host returns a deterministic redirect to the equivalent canonical URL, with no loops and no split destinations. Validate behavior for both root and deep URLs, confirm consistent protocol handling, and re-test until both variants always converge to one canonical endpoint.