Skip to main content

Sitemap Analysis & Validation

Issue No: 3

Category: Crawlability & Indexation

Issue type: Issue, Warning, Suggestion

Priority: CRITICAL, IMPORTANT, STANDARD

Description​

A sitemap is an XML file that lists a site's URLs and metadata to guide search engine crawlers; missing, malformed, or conflicting sitemaps can significantly reduce crawl efficiency and delay indexation of important pages.

How do we capture it​

  1. Discover the sitemap URL:
    • Check robots.txt for any Sitemap: directives.
    • Try common paths: /sitemap.xml, /sitemap_index.xml, /sitemap/, /sitemap.php, /sitemap.txt.
    • Check <head> of every crawled HTML response for <link rel="sitemap" type="application/xml" href="..."> and record the href value; flag pages where the tag is absent. Note: The crawler parses raw HTTP responses only and does not execute JavaScript to find these tags.
  2. Fetch the sitemap:
    • Send HTTP GET request to the resolved sitemap URL(s).
    • Record HTTP status code, response size, and Last-Modified / Content-Type headers.
  3. Parse the XML:
    • Detect type: <sitemapindex> (index file) or <urlset> (standard sitemap).
    • For index files: extract all child <sitemap><loc> entries and fetch each recursively.
    • For urlset files: extract all <url> entries with <loc>, <lastmod>, <changefreq>, <priority>.
  4. Validate each URL:
    • Send HTTP GET requests to each <loc> URL.
    • Record HTTP status code, redirect chains, and Content-Type.
    • Cross-reference against robots.txt rules and HTTP/HTML noindex directives.
  5. Cross-check:
    • Compare sitemap URLs against robots.txt Disallow rules.
  6. Validate canonical ↔ <loc> alignment:
    • For each crawled HTML response, extract the canonical URL from <link rel="canonical" href="...">.
    • Look up the page's <loc> entry in the parsed sitemap data.
    • Compare both values exactly: same scheme (https), same domain, same path, same trailing-slash convention.
  7. Detect image sitemap coverage:
    • For each crawled page, collect all image URLs from raw <img src>, <img srcset>, <meta property="og:image">, and JSON-LD image fields.
    • Check whether those image URLs appear as <image:image><image:loc> entries under the parent page's <url> block in the sitemap.
  8. Detect video sitemap coverage:
    • For each crawled page, detect embedded video via <iframe> src patterns, HTML5 <video> tags, and JSON-LD VideoObject structured data from the raw HTML text.
    • Check whether the page has a corresponding <video:video> entry in the sitemap containing at minimum: <video:thumbnail_loc>, <video:title>, and <video:content_loc>/<video:player_loc>.

What to store​

FieldTypeComment
site_idintegerWhich site this sitemap belongs to
typetextSitemap type: 'index' or 'urlset'
slugtextIdentifier derived from filename, e.g. 'sitemap-products'
urltextFull absolute URL where this sitemap was fetched from
url_countintegerTotal number of <url> entries parsed from this sitemap
file_sizeintegerFile size in bytes of the raw fetched sitemap
fetched_attimestampTimestamp of when this sitemap was last fetched
last_modifiedtimestampValue from HTTP Last-Modified response header
generated_attimestampValue from <lastmod> on the sitemap index entry
http_statusintegerHTTP status code returned when fetching the sitemap file
is_activebooleanWhether this is the current active version for the site
statustextProcessing state: 'pending', 'generated', 'failed'
errortextError message if fetch or parse failed
sitemap_idintegerWhich sitemap a URL belongs to
loctextAbsolute URL value from the <loc> tag
lastmodtimestampDate from <lastmod> tag
changefreqtextHint from <changefreq>
prioritynumericPage priority 0.0–1.0 from <priority> tag
is_excludedbooleanWhether this URL was excluded from the sitemap build
exclude_reasontextReason for exclusion, e.g. 'noindex', 'redirected', 'blocked'
is_indexablebooleanWhether the URL is indexable (200 + no noindex + not disallowed)
redirect_targettextFinal URL after following redirects
canonical_urltextCanonical URL declared on the page
canonical_matches_locbooleanTrue if the canonical URL matches the <loc> value exactly
has_head_sitemap_linkbooleanTrue if <link rel="sitemap" type="application/xml"> was found
checked_attimestampWhen this URL's HTTP status was last verified
image_loctext<image:loc> — absolute URL of the image
image_titletext<image:title> — title of the image
image_captiontext<image:caption> — caption or description
image_geo_locationtext<image:geo_location> — location string
image_licensetext<image:license> — URL of the image licence
thumbnail_loctext<video:thumbnail_loc> — URL of the video thumbnail
titletext<video:title> — video title
descriptiontext<video:description> — short description
content_loctext<video:content_loc> — URL of the actual video file
player_loctext<video:player_loc> — URL of the video player embed
durationinteger<video:duration> — video length in seconds
expiration_datetimestamp<video:expiration_date> — when the video will expire
ratingnumeric<video:rating>
view_countinteger<video:view_count> — total view count
publication_datetimestamp<video:publication_date> — when the video was published
family_friendlyboolean<video:family_friendly> — safe for all ages
liveboolean<video:live> — true if the video is a live stream
enginetextTarget search engine submitted to
submitted_attimestampWhen the sitemap was submitted
response_codeintegerHTTP response code from search engine API

Condition for trigger​

The following logic defines when to surface issues, warnings, or suggestions based on gathered telemetry.

  • Trigger Rule 1: Sitemap Unreachable

    • Target Field: http_status
    • Evaluation Logic: http_status != 200
    • Severity: CRITICAL
    • Diagnostic Message: "Sitemap returned a non-200 HTTP status code. Search engines cannot process it."

  • Trigger Rule 2: Complete Absence

    • Target Field: id
    • Evaluation Logic: id IS NULL
    • Severity: CRITICAL
    • Diagnostic Message: "No sitemap discovered at any known path or in robots.txt."

  • Trigger Rule 3: Sitemap Size Over 50k URLs

    • Target Field: url_count
    • Evaluation Logic: url_count > 50000 AND type = 'urlset'
    • Severity: WARNING
    • Diagnostic Message: "Single sitemap exceeds 50,000 URLs without using a sitemap index."

  • Trigger Rule 4: File Size Limit Exceeded

    • Target Field: file_size
    • Evaluation Logic: file_size > 52428800
    • Severity: WARNING
    • Diagnostic Message: "Sitemap file exceeds the 50MB uncompressed size limit."

  • Trigger Rule 5: High Error Rate

    • Target Field: http_status
    • Evaluation Logic: COUNT(http_status != 200) / url_count > 0.10
    • Severity: WARNING
    • Diagnostic Message: "Over 10% of listed URLs are broken, redirected, or unavailable."

  • Trigger Rule 6: Blocked by Robots.txt

    • Target Field: exclude_reason
    • Evaluation Logic: exclude_reason = 'blocked'
    • Severity: WARNING
    • Diagnostic Message: "A URL in the sitemap is blocked by a Disallow rule in robots.txt."

  • Trigger Rule 7: Contains Noindex

    • Target Field: exclude_reason
    • Evaluation Logic: exclude_reason = 'noindex'
    • Severity: WARNING
    • Diagnostic Message: "A URL in the sitemap has a noindex meta tag or HTTP header."

  • Trigger Rule 8: Not Declared in Robots.txt

    • Target Field: site_id
    • Evaluation Logic: No directive = 'Sitemap'
    • Severity: SUGGESTION
    • Diagnostic Message: "Sitemap is not declared in robots.txt."

  • Trigger Rule 9: Missing Lastmod

    • Target Field: lastmod
    • Evaluation Logic: lastmod IS NULL
    • Severity: SUGGESTION
    • Diagnostic Message: "lastmod values are missing; search engines cannot prioritise correctly."

  • Trigger Rule 10: Canonical Mismatch

    • Target Field: canonical_matches_loc
    • Evaluation Logic: canonical_matches_loc = false
    • Severity: WARNING
    • Diagnostic Message: "Canonical URL on page doesn't match its <loc> entry exactly."

Sources​

Long description​

A sitemap is a file — most commonly in XML format — that lists all URLs belonging to a website along with optional metadata about each URL. It acts as a structured roadmap that tells search engine crawlers which pages exist on a site, when they were last updated, how frequently they change, and how important they are relative to one another.

While crawlers can discover pages by following links, a sitemap ensures that even pages with few or no internal links — often called "orphan pages" — are still discovered and considered for indexing. This makes sitemaps especially valuable for large sites, newly launched sites, or sites with complex navigation.

Common scenarios flagged by the crawler:

  • Sitemap returning 404 or 5xx: Wastes crawl budget and forces search engines to rely purely on links.
  • Sitemap URLs blocked by robots.txt: Conflicting signal between sitemap inclusion and robots parsing.
  • Sitemap exceeding limitations: Over 50,000 URLs or 50MB per sitemap will be ignored by Google.
  • Canonical Mismatches: Trailing slashes or mismatched HTTP/HTTPS between <loc> and <link rel="canonical"> split signals.

How to Fix​

  1. Ensure canonical status: The sitemap must be available via HTTP GET and return a 200 OK status without redirects.
  2. Resolve contradictory signals: Remove URLs from the sitemap that are blocked by robots.txt or marked with noindex.
  3. Split large sitemaps: For files over 50,000 URLs or 50MB, split them into smaller chunks using a <sitemapindex>.
  4. Standardize <loc> attributes: Ensure every URL in the sitemap exactly mirrors the canonical preference (e.g. enforcing HTTPS and consistent trailing slashes).
  5. Declare dynamically: Place a Sitemap: directive in the root robots.txt and keep the file dynamically generated to avoid serving stale values.
  6. Integrate media entries: Map images and videos on critical URLs using XML <image:image> and <video:video> namespaces.