No orphan pages (all linked somewhere)

Issue No: 17

Category: Crawl Behaviour

Issue type: Suggestion

Priority: STANDARD

Description

This audit verifies that each indexable page is linked from at least one other internal page. Orphan pages are harder for crawlers to discover and may receive weak internal authority flow.

How do we capture it

1. Build the candidate page set from crawl seeds, sitemap URLs, and all successfully fetched internal URLs within crawl scope.
2. For each fetched internal page, parse raw HTML and extract all <a href> links that resolve to in-scope internal targets.
3. Normalize source and target URLs using the crawler's canonical URL normalization rules (scheme/host normalization, fragment removal, relative resolution).
4. Construct inbound-link counts by incrementing target page counters for every valid internal edge discovered from source pages.
5. Exclude self-links from inbound counts to prevent false non-orphan clearance.
6. Mark a page as orphan when inbound-link count from other internal pages is zero.
7. Track source coverage metadata:
   • Whether page appears in sitemap.
   • Whether page was discovered only via seed list.
   • Whether page was discovered only through redirects.
8. Record crawl constraints and fetch failures that may reduce link graph completeness.
9. Constraint: because the crawler does not execute JavaScript or render DOM state, links created only at runtime are out of scope and cannot be used to clear orphan status.

What to store

Field	Type	Comment
page_url TEXT	TEXT	Internal page URL evaluated for orphan status.
inbound_internal_link_count INTEGER	INTEGER	Number of unique in-scope source pages linking to this page via raw HTML anchors.
is_orphan_page BOOLEAN	BOOLEAN	True when no inbound internal links from other pages are detected.
linked_from_urls JSONB	JSONB	Array of normalized source URLs that link to this page.
discovered_via_sitemap BOOLEAN	BOOLEAN	Indicates whether page URL appears in sitemap sources.
discovered_via_seed BOOLEAN	BOOLEAN	Indicates whether page URL originated from configured seed URLs.
graph_completeness_warning TEXT	TEXT	Notes partial crawl conditions such as fetch failures that can affect orphan confidence.
checked_at TIMESTAMPTZ	TIMESTAMPTZ	Timestamp when orphan evaluation was computed.

Condition for trigger

A trigger is raised when a page has no inbound internal links in the captured HTML link graph.

• Trigger Rule 1: Orphan Page Detected

  • Target Field: is_orphan_page
  • Evaluation Logic: is_orphan_page = TRUE
  • Severity: SUGGESTION
  • Diagnostic Message: "This page has no inbound internal links from other crawled pages and appears orphaned. Add contextual internal links to improve discoverability."

• Trigger Rule 2: Zero Inbound Link Count

  • Target Field: inbound_internal_link_count
  • Evaluation Logic: inbound_internal_link_count = 0
  • Severity: SUGGESTION
  • Diagnostic Message: "No internal pages link to this URL in the captured crawl graph. Link this page from relevant indexable pages."

Sources

• Google Search Central - Links and crawlability: https://developers.google.com/search/docs/crawling-indexing/links-crawlable
• Google Search Central - Sitemaps overview: https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview
• WHATWG HTML Standard - Hyperlink elements: https://html.spec.whatwg.org/multipage/links.html
• RFC 3986 URI Generic Syntax: https://www.rfc-editor.org/rfc/rfc3986

Long description

Orphan pages sit outside the internal linking structure that search engines use to discover, prioritize, and understand document relationships. Even when an orphan URL appears in a sitemap, lack of contextual internal links can reduce crawl consistency and topical association signals.

Common failure scenarios checked by the crawler:

• Newly published pages included in XML sitemap but not linked from navigational or contextual pages.
• Legacy URLs reachable only by direct entry or campaign links with no on-site references.
• Pagination or faceted pages disconnected after template updates.
• Internal links injected only by client-side JavaScript and absent from server HTML.
• Crawl graph blind spots caused by intermittent fetch failures or blocked sections.

This check uses deterministic inbound-edge counting from raw HTML link extraction and does not rely on browser execution.

How to Fix

Ensure every important page is linked from at least one relevant, indexable internal page using crawlable anchor links in server-delivered HTML. Add links from category hubs, related-content modules, or navigation structures, then re-run crawl validation to confirm inbound link coverage.