Skip to main content

Redirect Analysis & Validation

Issue No: 8

Category: Crawl Behaviour

Issue type: Issue

Priority: CRITICAL

Description​

Redirects automatically forward users and crawlers from one URL to another. Misconfigured redirects — including chains, loops, broken destinations, wrong status codes, and client-side redirects — waste crawl budget, dilute link equity, and degrade user experience.

How do we capture it​

  1. Crawl all known URLs: Seed the crawl queue from the sitemap and discovered internal links. Send HTTP GET requests to every URL with redirect-following disabled at the HTTP client layer, capturing the raw status code and Location header on the first response.
  2. Follow redirects hop by hop: For each 3xx response, resolve the Location header against the current URL per RFC 3986 rules, then issue a new GET to that resolved URL. Repeat up to a fixed maximum hop limit (e.g. 10). Normalize each hop URL before comparison:
    • Lowercase scheme and host.
    • Remove default ports (:80 for HTTP, :443 for HTTPS).
    • Normalize dot segments in path.
    • Preserve query string order as received.
  3. Detect redirect loops: After normalizing each hop URL, check whether it already appears earlier in the same chain. If a URL repeats, mark has_loop = true and stop traversal immediately.
  4. Detect redirect chains: Count total hops before a terminal non-3xx response. Flag has_redirect_chain = true when hop count exceeds one (A → B → C or longer).
  5. Stop traversal and record error when: the next Location URL is malformed, the request times out or a network error occurs, or the maximum hop limit is reached without a terminal response — store the reason in redirect_error_code (MALFORMED_LOCATION, TIMEOUT, MAX_HOPS_EXCEEDED).
  6. Check destination URL validity: Send a final GET to the terminal URL. Record final_status_code, Content-Type, and canonical tag.
  7. Check HTTP → HTTPS: For every URL discovered on http://, verify a redirect to https:// exists and that the HTTPS version returns 200 (not another redirect).
  8. Detect client-side redirects: Parse raw HTML of fetched pages for <meta http-equiv="refresh"> tags. JavaScript execution is not performed; only statically available meta-refresh tags are captured.
  9. Cross-reference with analytics/logs: Enrich hit_count per redirect rule from server logs or analytics. Identify high-traffic URLs involved in chains, loops, or broken destinations as highest priority.

What to store​

FieldTypeComment
site_idintegerWhich site this redirect rule belongs to
from_pathtextOrigin path or URL pattern that triggers the redirect
to_pathtextDestination path or full URL the redirect points to
status_codeintegerHTTP redirect status: 301, 302, 307, or 308
is_activebooleanWhether this redirect rule is currently active
is_regexbooleanWhether from_path is a regex pattern rather than a literal path
hit_countintegerNumber of times this redirect has been triggered
last_hit_attimestampTimestamp of the most recent hit on this redirect rule
created_attimestampWhen this redirect rule was created
updated_attimestampWhen this redirect rule was last modified
notetextInternal comment explaining why this redirect exists
origin_urltextThe original URL a user or crawler would request
final_urltextThe final destination URL after all hops are followed
hop_countintegerTotal number of hops from origin to final destination
has_loopbooleanWhether a redirect loop was detected in this chain
final_status_codeintegerHTTP status code of the final destination URL
chain_jsonjsonbFull ordered array of hop URLs and status codes
detected_attimestampWhen this chain was last observed during a crawl
is_resolvedbooleanWhether this chain has been acknowledged/fixed
chain_idintegerWhich chain this hop belongs to
hop_indexintegerPosition in the chain (0 = origin, 1 = first redirect, etc.)
urltextThe URL at this step in the redirect chain
hop_status_codeintegerHTTP status code returned at this hop
redirect_typetextClassification: 'http', 'meta_refresh', 'javascript'
response_time_msintegerTime in milliseconds to receive a response at this hop
redirect_error_codetextCrawler-side error: TIMEOUT, MALFORMED_LOCATION, or MAX_HOPS_EXCEEDED
checked_attimestamptzTimestamp when redirect validation was executed

Condition for trigger​

The following rules determine when a redirect issue is triggered based on the data queried from the database.

  • Trigger Rule 1: Redirect loop detected

    • Target Field: has_loop
    • Evaluation Logic: has_loop = true
    • Severity: CRITICAL
    • Diagnostic Message: "Redirect loop detected: This URL is unresolvable. Crawlers and users will receive an error."

  • Trigger Rule 2: Chain of 3 or more hops

    • Target Field: hop_count
    • Evaluation Logic: hop_count >= 3
    • Severity: WARNING
    • Diagnostic Message: "Redirect chain of 3+ hops detected. Excessive hops waste crawl budget and increase latency."

  • Trigger Rule 3: Chain of exactly 2 hops

    • Target Field: hop_count
    • Evaluation Logic: hop_count = 2
    • Severity: SUGGESTION
    • Diagnostic Message: "Redirect chain of exactly 2 hops detected. Consider flattening to a single direct redirect."

  • Trigger Rule 4: Max hop limit exceeded

    • Target Field: redirect_error_code
    • Evaluation Logic: redirect_error_code = 'MAX_HOPS_EXCEEDED'
    • Severity: WARNING
    • Diagnostic Message: "Redirect traversal exceeded the allowed hop limit before reaching a final URL. Simplify redirect routing."

  • Trigger Rule 5: Broken redirect destination

    • Target Field: final_status_code
    • Evaluation Logic: final_status_code IN (404, 410, 500, 502, 503)
    • Severity: CRITICAL
    • Diagnostic Message: "Broken redirect detected: The final destination URL is unreachable."

  • Trigger Rule 5: Wrong redirect type for permanent move

    • Target Field: status_code
    • Evaluation Logic: status_code = 302 AND hit_count > 0
    • Severity: WARNING
    • Diagnostic Message: "Long-lived 302 redirect detected. If permanent, use 301 to pass link equity."

  • Trigger Rule 6: Meta refresh or JS redirect

    • Target Field: redirect_type
    • Evaluation Logic: redirect_type IN ('meta_refresh', 'javascript')
    • Severity: WARNING
    • Diagnostic Message: "Client-side redirect detected. Use server-side redirects for reliable crawler handling."

  • Trigger Rule 7: HTTP not redirecting to HTTPS

    • Target Field: origin_url
    • Evaluation Logic: origin_url LIKE 'http://%' AND final_url NOT LIKE 'https://%'
    • Severity: WARNING
    • Diagnostic Message: "Insecure URL not redirecting to HTTPS."

  • Trigger Rule 8: High-traffic broken redirect

    • Target Field: hit_count
    • Evaluation Logic: hit_count > 100 AND final_status_code != 200
    • Severity: CRITICAL
    • Diagnostic Message: "Frequently accessed redirect pointing to a dead destination."

  • Trigger Rule 9: Obsolete redirect

    • Target Field: hit_count
    • Evaluation Logic: hit_count = 0
    • Severity: SUGGESTION
    • Diagnostic Message: "Redirect has zero hits in 90 days. Consider cleaning up if outdated."

Sources​

Long description​

A redirect is a server instruction that automatically forwards a request from one URL to a different URL. When a browser or crawler requests a URL that has a redirect configured, the server responds with a 3xx HTTP status code and a Location header pointing to the new URL. The client then makes a second request to the new URL.

Redirects serve many legitimate purposes: handling site migrations, consolidating duplicate content, enforcing HTTPS, managing seasonal or promotional pages, and preserving inbound links when URLs change. However, every redirect adds latency, complexity, and potential for misconfiguration — all of which have measurable SEO consequences.

Types and SEO Behaviour:

  • 301 (Moved Permanently): Passes equity (~99%). Page has permanently moved to a new URL.
  • 302 (Found/Temporary): Does not pass equity. Used for temporary redirects, tests, or maintenance.
  • 307 (Temporary Redirect): Like 302 but preserves HTTP method.
  • 308 (Permanent Redirect): Like 301 but preserves HTTP method.
  • Meta Refresh: Client-side (HTML), partial/delayed equity pass. Discouraged fallback.

Common Scenarios:

  • Redirect Chain (A → B → C): Each hop adds latency, wastes crawl budget, and dilutes link equity.
  • Redirect Loop (A → B → A): Chain never terminates. Crawlers abandon the URL, resulting in complete indexation loss.
  • Broken Redirect: Destination returns 4xx/5xx, wasting link equity pointed to the original URL.
  • HTTP Not Redirecting to HTTPS: Results in security warnings, penalized rankings, and exposes user data.
  • Wrong Type (302 instead of 301): Search engines continue to treat the old URL as canonical and fail to consolidate link equity to the final destination.
  • Dynamic Redirect Systems: Systems that emit malformed or unstable Location headers, or infinite redirect paths that only terminate due to client-side hop limits.
  • Conflicting Rewrite Rules: Separate redirect rules (e.g. HTTP→HTTPS and non-www→www implemented as two independent hops) stack into chains instead of being consolidated.

How to Fix​

  • Flatten redirect chains: For any chain A → B → C, update the origin to point directly to the final destination. Audit intermediate hops and update internal links.
  • Break redirect loops: Trace the chain until a URL repeats. Remove or correct the rule creating the cycle (often conflicts between .htaccess, CMS, or CDN rules).
  • Fix broken redirect destinations: If the destination was deleted, point to the most relevant live alternative (or homepage). If it moved, update the final URL. If it's a server error, fix the server.
  • Enforce HTTPS sitewide: Configure a blanket HTTP → HTTPS redirect at the server or CDN level (e.g., using nginx return 301 or Apache RewriteRule) instead of individually per URL.
  • Replace 302s with 301s for permanent moves: Audit active 302s. If they've been in place for over 30 days and the move is permanent, change to 301 to pass link equity.
  • Replace meta refresh and JS redirects: Implement an HTTP 301 (or 302) at the server level, remove the client-side tag from the HTML, and verify the response header contains the Location directive.
  • Clean up obsolete redirects: Standardize redirects using zero or near-zero hits over a 90-day period. Confirm they no longer receive traffic and remove them to optimize rule-set processing overhead.
  • Update internal links: Point all internal links directly to the final destination URL instead of intermediate redirect hops to eliminate latency and preserve link equity.