No duplicate content

Issue No: 43

Category: page seo basics

Issue type: Issue

Priority: CRITICAL

Description

Verify that distinct URLs do not expose identical or near-identical primary content bodies without explicit canonical consolidation.

How do we capture it

1. Fetch HTML response for each crawled URL and keep final URL after redirects.
2. Parse raw HTML and extract primary textual payload:
   • Remove script, style, noscript, and boilerplate navigation blocks where deterministic selectors are available.
   • Extract visible text from remaining HTML nodes in source order.
3. Normalize extracted text for exact-match comparison:
   • Lowercase.
   • Collapse whitespace.
   • Strip repeated punctuation boundaries.
4. Generate exact-content fingerprint (hash) from normalized text.
5. Group pages by fingerprint within crawl scope.
6. Identify duplicate clusters where more than one distinct final URL shares identical fingerprint.
7. Compare canonical URLs to determine whether duplicates are properly consolidated.
8. Constraint: duplicate detection is based on raw HTML response text only. No rendered DOM or JavaScript-expanded content is used.

What to store

Field	Type	Comment
page_url TEXT	TEXT	Requested URL.
final_url TEXT	TEXT	Final resolved URL.
extracted_content_text TEXT	TEXT	Normalized content text used for matching.
content_text_length INTEGER	INTEGER	Character length of normalized content text.
exact_content_fingerprint TEXT	TEXT	Hash fingerprint used for exact duplicate clustering.
duplicate_cluster_id TEXT	TEXT	Cluster identifier shared by exact duplicate pages.
duplicate_cluster_size INTEGER	INTEGER	Number of URLs in duplicate cluster.
duplicate_urls JSONB	JSONB	List of URLs sharing same content fingerprint.
canonical_url TEXT	TEXT	Extracted canonical URL for consolidation analysis.
duplicate_content_is_consolidated BOOLEAN	BOOLEAN	Whether duplicates point to a single canonical target.

Condition for trigger

This audit triggers when exact duplicate content exists across multiple URLs without proper consolidation.

• Trigger Rule 1: Exact Duplicate Cluster Detected

  • Target Field: duplicate_cluster_size
  • Evaluation Logic: duplicate_cluster_size > 1
  • Severity: CRITICAL
  • Diagnostic Message: "Exact duplicate content appears on multiple URLs."

• Trigger Rule 2: Duplicate Content Not Canonically Consolidated

  • Target Field: duplicate_content_is_consolidated
  • Evaluation Logic: duplicate_cluster_size > 1 AND duplicate_content_is_consolidated = FALSE
  • Severity: CRITICAL
  • Diagnostic Message: "Duplicate pages are not canonically consolidated to a single preferred URL."

• Trigger Rule 3: Thin Extracted Content Prevents Reliable Uniqueness

  • Target Field: content_text_length
  • Evaluation Logic: content_text_length < 100
  • Severity: WARNING
  • Diagnostic Message: "Extracted content body is too small for reliable duplicate-content classification."

Sources

• Google Duplicate URL Consolidation: https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls
• Google Search Essentials: https://developers.google.com/search/docs/fundamentals/creating-helpful-content
• RFC 3986 URI Generic Syntax: https://www.rfc-editor.org/rfc/rfc3986

Long description

Duplicate content can split indexing signals, create crawl inefficiency, and make canonical selection unpredictable. Exact-match detection using normalized content fingerprints is a robust first-line mechanism for crawler-level duplicate auditing. When duplicates are intentional, canonicalization should clearly indicate the preferred URL to avoid competing index entries.

Common failure scenarios checked by the crawler:

• Same article accessible at multiple URL variants.
• Session, parameter, or path variants returning identical text.
• Duplicate product or landing pages without canonical alignment.
• Boilerplate-heavy pages where primary text extraction is too weak for reliable comparison.

How to Fix

Reduce duplicate URL surfaces by consolidating routing patterns and parameter handling. Add canonical tags that point all duplicate variants to one preferred URL, and update internal linking and sitemaps to reference only canonical destinations.