Crawl depth shallow (3 clicks from homepage)

Issue No: 14

Category: Crawl Behaviour

Issue type: Warning

Priority: IMPORTANT

Description

This audit verifies that important internal pages are discoverable within a shallow click depth from the homepage. Excessive depth can reduce crawl frequency and delay indexing.

How do we capture it

1. Fetch the homepage URL via GET and parse raw HTML text for internal anchor links (<a href>), ignoring JavaScript-generated navigation.
2. Normalize extracted URLs using deterministic rules:
   • Convert relative links to absolute URLs.
   • Remove fragment identifiers.
   • Normalize host and default ports.
   • Apply canonical trailing-slash policy used by the crawler.
3. Restrict traversal to same-site URLs based on configured crawl scope (host and allowed subdomains).
4. Build a directed internal link graph where each node is a normalized URL and each edge represents a discovered HTML anchor from source to target.
5. Perform breadth-first search (BFS) starting at homepage with depth 0:
   • First-level links are depth 1.
   • Continue until queue exhaustion or configured crawl page cap.
6. For every discovered page, store the minimum observed depth value.
7. Mark pages as depth violations when min_depth > 3.
8. Mark pages as depth-unknown when they were crawled by seed/sitemap but were never reached from homepage via link traversal.
9. Apply timeout and fetch error handling per URL; if a source page cannot be fetched, record the source error and continue graph traversal from other queued pages.
10. Constraint: this crawler does not execute JavaScript and does not render the DOM, so links that only exist after client-side rendering are out of capture scope and must be reported as a limitation.

What to store

Field	Type	Comment
page_url TEXT	TEXT	URL of the page evaluated for homepage click depth.
min_click_depth INTEGER	INTEGER	Smallest BFS depth from homepage to this page using HTML links only.
is_depth_violation BOOLEAN	BOOLEAN	True when minimum click depth exceeds 3.
is_depth_unknown BOOLEAN	BOOLEAN	True when page is not reachable from homepage via captured HTML links.
discovery_source TEXT	TEXT	Indicates whether page came from HOMEPAGE_GRAPH, SITEMAP, or SEED_URL.
parent_path JSONB	JSONB	Optional shortest discovered parent chain from homepage to page URL.
source_fetch_error TEXT	TEXT	Last fetch/parsing error affecting depth computation for source nodes.
checked_at TIMESTAMPTZ	TIMESTAMPTZ	Timestamp when depth analysis was completed.

Condition for trigger

A trigger is raised when a crawled page is too deep from homepage navigation or cannot be reached through captured internal links.

• Trigger Rule 1: Depth Greater Than Three

  • Target Field: min_click_depth
  • Evaluation Logic: min_click_depth > 3
  • Severity: WARNING
  • Diagnostic Message: "This page is more than three clicks away from the homepage. Reduce internal navigation depth to improve crawl access."

• Trigger Rule 2: Depth Unknown From Homepage Graph

  • Target Field: is_depth_unknown
  • Evaluation Logic: is_depth_unknown = TRUE
  • Severity: WARNING
  • Diagnostic Message: "This page could not be reached from homepage links captured in raw HTML. Ensure it is linked in crawlable navigation."

Sources

• Google Search Central - Crawl budget management: https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget
• Google Search Central - Internal links: https://developers.google.com/search/docs/crawling-indexing/links-crawlable
• WHATWG HTML Standard - The a element: https://html.spec.whatwg.org/multipage/text-level-semantics.html#the-a-element
• RFC 3986 URI Generic Syntax: https://www.rfc-editor.org/rfc/rfc3986

Long description

Click depth is a structural proxy for how easily crawlers discover and revisit content. Pages buried too deeply in internal navigation can be deprioritized, especially on larger sites where crawl resources are finite. A script crawler can model this reliably by constructing a graph from raw HTML links and running BFS from the homepage.

Common failure scenarios checked by the crawler:

• Key landing pages only reachable through multi-level menu nesting beyond three hops.
• Faceted or filtered paths that trap pages behind repeated navigation steps.
• Important URLs present in sitemap but not in homepage-reachable internal link graph.
• Navigation links injected only by JavaScript, which are not visible in raw server HTML.
• Broken intermediary links that prevent shortest-path traversal.

Constraint note: browser-rendered links produced only after JavaScript execution are intentionally excluded, with no heuristic approximation.

How to Fix

Bring important pages closer to primary navigation by adding direct internal links from high-authority hub pages, including homepage or top-level category pages. Reduce unnecessary hierarchy levels, ensure crawlable anchor-based links exist in server-delivered HTML, and validate that critical URLs are reachable within three clicks in a fresh crawl.