Skip to main content

Broken internal URLs

Issue No: 9

Category: Crawl Behaviour

Issue type: Issue

Priority: IMPORTANT

Description​

Checks for broken internal links (internal page routes and internal asset URLs such as images, script/style assets, video, and other same-site resources) that return network errors, DNS failures, or HTTP error status codes when fetched from the raw HTTP response.

How do we capture it​

  1. Crawl entry points using normal site crawl logic (respect robots rules). For each fetched HTML response (HTTP GET), extract candidate internal URLs from:
    • <a href> attributes.
    • <img src> and <img srcset> attributes (parse srcset URLs and descriptors).
    • <source src> and <source srcset> in <picture> / <video> / <audio>.
    • link[href] for stylesheets and script[src] for scripts.
    • Inline CSS url(...) occurrences in style attributes and <style> blocks (basic regex parse for url(\s*['"]?([^'")]+)['"]?\s*)).
    • Meta refresh tags (<meta http-equiv="refresh" content="X;url=...">) when present.
  2. Normalize each candidate URL to an absolute URL using the response's base URL and the HTML <base> tag when present. Normalization rules:
    • Remove fragment identifiers before fetch (fetch url#frag as url). Store original fragment separately.
    • Resolve relative paths and protocol-relative URLs (//example.com/path) to full https?:// scheme based on the original response.
    • Treat same-origin host (hostname + port) and same-registered-domain (configurable) links as internal. Also treat absolute paths beginning with / as internal.
  3. Internal link classification:
    • is_internal = true when hostname equals page hostname, or when URL is a same-site relative path starting with / or when allowed internal host list contains the hostname.
  4. Fetch strategy for validation:
    • Prefer HEAD for non-asset links to save bandwidth, but fall back to GET when the server responds with a method-not-allowed (405) or returns suspicious/empty responses to HEAD (some servers mis-handle HEAD). For images and many assets, use GET because HEAD may not provide accurate content-type or size.
    • Use a configurable request timeout (e.g., 10s default) and a retry policy for transient network errors (exponential backoff, max 2 retries).
    • Follow redirects up to a configured maximum (e.g., 5) and record the final status code. Strip fragment identifiers when following redirects.
    • Set request headers: User-Agent (crawler id), Accept appropriate to the resource type (e.g., image/* for images), and Accept-Language if relevant.
  5. HTTP and network handling:
    • Consider HTTP status codes 200-299 as success (subject to content-type checks for assets). Treat 301/302/303/307/308 as redirects to be followed and evaluated by their final status.
    • Treat 4xx and 5xx responses as failures for that resource.
    • Treat DNS failures, TLS errors, connection timeouts, and connection refused as failures.
  6. Asset-specific checks:
    • For srcset, validate every candidate URL separately. Prefer the most-reliable descriptor-less URL if present, but still validate all to detect broken responsive images.
    • For images, verify the Content-Type header indicates an image MIME type and that the response body length > 0. If Content-Length is present and 0, flag as broken.
  7. Deduplication and rate control:
    • Deduplicate identical URLs per crawl run to avoid noisy checks; still record all referrers that reference the URL.
    • Respect site politeness: concurrency limits, per-host delay, and configurable rate limits.
  8. Edge cases & constraints:
    • Do NOT rely on JavaScript execution or client-side routing to discover or validate links. Links generated or resolved only by JavaScript are out of scope for this static check and must be documented as a detection limitation.
    • Single-page apps that respond 200 for most routes (catch-all) can mask broken routes. Where site returns a generic app shell for arbitrary paths, mark as POSSIBLE_CATCHALL and surface as a non-fatal suggestion (see Long description). Full validation for SPA route correctness requires a rendering-capable check outside this static crawler.
    • Some servers treat HEAD differently; use GET fallback when HEAD is unreliable.

What to store​

FieldTypeComment
urlTEXTThe original extracted href/src value as found in HTML (un-normalized).
resolved_urlTEXTThe absolute, normalized URL used for validation (fragments removed).
referrer_urlTEXTThe page URL where this link was discovered.
is_internalBOOLEANTrue when the link was classified as internal to the site.
resource_typeTEXTOne of: page, image, script, stylesheet, video, audio, asset, other.
http_methodTEXTThe method used for validation (HEAD or GET).
status_code INTEGERINTEGERThe HTTP status code returned by the final request.
content_typeTEXTThe Content-Type returned by the server for the resource.
response_time_ms INTEGERINTEGERTime in milliseconds for final request to complete.
error_typeTEXTNetwork error classification (e.g., dns_error, timeout, tls_error, conn_refused) or NULL.
is_brokenBOOLEANTrue if the check determined the link is broken per trigger rules.
discovered_onTIMESTAMPCrawl timestamp when the link was observed.
noteTEXTAny crawler notes (e.g., HEAD_fallback_GET, POSSIBLE_CATCHALL).

Condition for trigger​

  • Trigger Rule 1: HTTP error status

    • Target Field: status_code
    • Evaluation Logic: status_code >= 400
    • Severity: CRITICAL
    • Diagnostic Message: "Internal link to [resolved_url] returned HTTP [status_code] when fetched from [referrer_url]."

  • Trigger Rule 2: Network or DNS failure

    • Target Field: error_type
    • Evaluation Logic: error_type IN ('dns_error','timeout','tls_error','conn_refused')
    • Severity: CRITICAL
    • Diagnostic Message: "Internal link to [resolved_url] could not be reached from [referrer_url]: [error_type]."

  • Trigger Rule 3: Zero-length or unexpected content for assets

    • Target Field: content_type, status_code, response_time_ms
    • Evaluation Logic: resource_type IN ('image','video','audio') AND status_code BETWEEN 200 AND 299 AND (content_type NOT LIKE 'image/%' OR response_body_length = 0)
    • Severity: WARNING
    • Diagnostic Message: "Asset at [resolved_url] returned Content-Type [content_type] or empty body when fetched from [referrer_url]."

  • Trigger Rule 5: Possible SPA catch-all masking

    • Target Field: note
    • Evaluation Logic: note = 'POSSIBLE_CATCHALL'
    • Severity: SUGGESTION
    • Diagnostic Message: "The site appears to return a generic app shell for arbitrary paths; static link validation may be inconclusive for [resolved_url]. Consider a rendering-capable check."

Sources​

Long description​

This audit detects internal links (both navigational routes and asset URLs) that are unreachable or return error responses when fetched directly by a non-rendering crawler. The crawler operates on raw HTTP responses only and cannot execute JavaScript, so it cannot discover or validate links that are created or resolved at runtime by client-side code.

Common failure scenarios checked by the crawler:

  • 404 Not Found pages caused by broken internal routings or moved pages.
  • 403/401 responses caused by authentication gating or misconfigured permissions on internal assets.
  • DNS resolution failures or TLS handshake errors for internal hostnames (often due to misconfigured host records or certificate issues).
  • Images referenced by srcset variants where one or more size variants point to missing resources.
  • Servers that return 200 for arbitrary paths (SPA catch-all) which mask actual broken routes; the static crawler cannot reliably determine whether a route is a valid content route without rendering.

ASCII mockup (referrer -> validation result):

referrer: https://example.com/products/widget-123

  • link: /images/widget-123@2x.jpg -> GET -> 200 image/jpeg (ok)
  • link: /images/widget-123-zoom.jpg -> GET -> 404 (broken)
  • link: /product/widget-123/spec -> HEAD -> 200 (ok)

Constraints:

  • No JavaScript execution: links injected or resolved by JS will not be discovered.
  • SPA catch-all pages: if the server responds with an app shell for arbitrary paths, the check cannot verify semantic route validity.

How to Fix​

  1. Replace or correct broken href/src targets to point to the correct absolute or relative internal URL.
  2. Ensure server routing includes the expected path (fix server-side routing rules or CMS link targets).
  3. For missing assets, re-upload the asset to the correct path or update references to the new asset location.
  4. Fix server configuration issues leading to DNS/TLS errors (proper DNS records, valid certificates, and correct virtual host bindings).
  5. If the site is an SPA and routes are client-side only, add a rendering-capable validation step (headless browser run) to validate route-level content rather than relying solely on static HTTP checks.