Separate URLs per Language (Subdirectory or ccTLD, Not Just Cookies)

Issue No: 171

Category: International SEO Issue type: Issue Priority: IMPORTANT

Description

Google requires that each language or regional version of a page be accessible at a distinct, crawlable URL — using subdirectories (/en/, /fr/) or country-code top-level domains (ccTLDs like .fr, .de) — rather than relying solely on cookies, JavaScript redirects, or session-based language switching that hide content variations behind a single URL.

How do we capture it

Where to Find

A correctly structured multilingual site exposes separate URLs:

https://site.com/en/about/          ← subdirectory approach
https://site.com/fr/about/

https://en.site.com/about/          ← subdomain approach
https://fr.site.com/about/

https://site.co.uk/about/           ← ccTLD approach
https://site.fr/about/

A non-crawlable language switch relies on cookies or JavaScript:

https://site.com/about/             ← same URL for all languages
  Cookie: lang=fr                   ← language controlled by cookie

HTML Extraction Flow

When the XeoPix crawler downloads the HTML response, the parser:

Issues an HTTP GET to the target URL without any language cookies set.
Reads raw response bytes without JavaScript execution.
Inspects the Content-Language HTTP response header.
Parses the <head> for hreflang alternate tags and extracts all href values.
Parses the <head> for <link rel="canonical"> to get the canonical URL.
Checks whether hreflang href values reference distinct URL paths or domains.
Checks whether the page contains cookie-based language switching indicators (e.g., Set-Cookie: lang=, JavaScript navigator.language redirects in raw HTML script blocks).
Inspects for query-string language parameters (e.g., ?lang=fr, ?locale=en) as an alternative to path-based URLs.

Validation Logic

The crawler validates:

Whether hreflang alternates point to distinct, unique URL paths.
Whether language variants are distinguished by URL path segment, subdomain, or domain — not by cookie alone.
Whether the page serves different content based on query string parameters or path prefix.
Whether any inline script blocks perform cookie-based language redirects before crawler can read page content.

Examples of non-compliant approaches:

All language versions served at the same URL (/about/) with only a cookie or JS redirect.
Language controlled by ?lang=fr query parameter with no separate crawlable URL.
Language switching implemented entirely in client-side JavaScript (not readable from raw HTML).

How to Get It

Crawler should:

Fetch the page without any language cookies.
Extract all hreflang href values.
Analyze whether the hrefs represent distinct URL structures (different path segments or domains).
Check the raw HTML and HTTP headers for cookie-only or JS-only language mechanisms.
Record the detected language URL structure type.

What to store

Field	Type	Comment
`language_url_structure`	`TEXT`	Detected structure: `subdirectory`, `subdomain`, `cctld`, `query_param`, `cookie_only`, `unknown`
`hreflang_urls_distinct`	`BOOLEAN`	Whether hreflang alternate URLs use distinct paths or domains
`has_cookie_language_indicator`	`BOOLEAN`	Whether the page uses cookies for language switching (raw HTML or Set-Cookie header)
`has_query_param_language`	`BOOLEAN`	Whether language is switched via query string parameters
`hreflang_href_count`	`INT`	Number of distinct hreflang alternate URLs found on the page
`content_language_header`	`TEXT`	Value of the Content-Language HTTP response header

Condition for trigger

The following trigger rules evaluate whether language variants are served at distinct, crawlable URLs.

Trigger Rule 1: Cookie-Only Language Switching Detected
- Target Field: language_url_structure
- Evaluation Logic: = 'cookie_only'
- Severity: CRITICAL
- Diagnostic Message: "Language switching appears to rely on cookies rather than distinct URLs. Google cannot crawl cookie-based language variants. Each language version must have its own indexable URL."
Trigger Rule 2: Query Parameter Language Switching
- Target Field: has_query_param_language
- Evaluation Logic: = TRUE
- Severity: WARNING
- Diagnostic Message: "Language versions are differentiated only by query string parameters (e.g., ?lang=fr). While crawlable, this approach is not recommended by Google. Use subdirectories or ccTLDs for cleaner, more indexable language URLs."
Trigger Rule 3: No Distinct Language URLs Detected
- Target Field: hreflang_urls_distinct
- Evaluation Logic: = FALSE
- Severity: CRITICAL
- Diagnostic Message: "No distinct URLs were detected for language variants. Google requires each language version to have a unique, crawlable URL. Content hidden behind cookies or JavaScript cannot be indexed."

Sources

Google multilingual site URL structures: https://developers.google.com/search/docs/specialty/international/managing-multi-regional-sites#locale-specific-urls
Google hreflang documentation: https://developers.google.com/search/docs/specialty/international/localized-versions
Google Search Central — use different URLs for each language version: https://developers.google.com/search/docs/specialty/international/managing-multi-regional-sites#use-different-urls-for-different-language-versions
MDN Content-Language header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Language

Long description

Google's crawler is a server-side bot that fetches raw HTTP responses. It does not execute JavaScript, set cookies, or maintain session state between requests. As a result, if a website uses cookies or client-side JavaScript to serve different language versions of a page at the same URL, Google can only ever index one version — typically the default.

Google-recommended URL structures for multilingual content:

Approach	Example	Notes
Subdirectory	`site.com/fr/`	Easiest to set up; single domain
Subdomain	`fr.site.com/`	Separate subdomain per language
ccTLD	`site.fr/`	Strongest geo-targeting signal
Query param	`site.com/?lang=fr`	Crawlable but not preferred

Non-compliant approach — cookie-based switching:

GET /about/ HTTP/1.1
Cookie: lang=fr

HTTP/1.1 200 OK
Content-Type: text/html
<!-- Content served in French based on cookie -->

Google's crawler never sends language cookies, so it only indexes the default (cookie-absent) version.

Non-compliant approach — JavaScript redirect:

<script>
  if (navigator.language.startsWith('fr')) {
    window.location.href = '/fr/';
  }
</script>

The XeoPix crawler does not execute JavaScript, so the redirect never fires during crawling. The default page is indexed for all languages.

Common problems:

Cookie-based language switchers served from one URL.
JavaScript-only language detection and routing.
Query string language parameters without canonical differentiation.
Language selector pages that require user interaction to switch languages.

SEO Impact

Using a single URL for all language versions may cause:

Only the default language version being indexed by Google.
International users being served the wrong language version in search results.
Loss of hreflang effectiveness, since all alternates point to the same URL.
Duplicate content signals when the same URL appears to serve multiple languages.
Complete failure of international SEO for non-default language markets.

Common Failure Scenarios

# BAD: same URL, language controlled by cookie
GET /page/          Cookie: lang=fr → French content
GET /page/          Cookie: lang=en → English content

# GOOD: distinct URLs
GET /fr/page/       → French content
GET /en/page/       → English content

2. JavaScript-Only Language Redirect

<!-- BAD: Googlebot never executes this -->
<script>
  if (navigator.language === 'ar') location.href = '/ar/';
</script>

<!-- GOOD: Serve correct language at correct URL from server -->

3. Query Parameter Only

# ACCEPTABLE but not preferred
https://site.com/page?lang=fr

# PREFERRED
https://site.com/fr/page

Diagnostic Output Example

Cookie-only language switching detected.

Affected URL:
https://example.com/about/

Detected:
  - No hreflang alternate URLs found
  - Set-Cookie: lang=en in response headers
  - No distinct language paths or subdomains detected

Recommendation:
Implement separate URLs per language:
  https://example.com/en/about/
  https://example.com/fr/about/

How to Fix

Implement distinct, crawlable URLs for each language version using one of these approaches:

Option 1 — Subdirectory (Recommended)

https://site.com/en/
https://site.com/fr/
https://site.com/ar/

Option 2 — Subdomain

https://en.site.com/
https://fr.site.com/

Option 3 — ccTLD

https://site.co.uk/     ← UK English
https://site.fr/        ← French

After implementing distinct URLs:

Add complete hreflang clusters on every language page.
Set <link rel="canonical"> to the correct language URL on each page.
Submit language-specific sitemaps or include hreflang in a single XML sitemap.
Remove cookie-based or JavaScript-only language routing from server-side code.

Description​

How do we capture it​

Where to Find​

HTML Extraction Flow​

Validation Logic​

How to Get It​

What to store​

Condition for trigger​

Sources​

Long description​

SEO Impact​

Common Failure Scenarios​

1. Cookie-Based Language Switching​

2. JavaScript-Only Language Redirect​

3. Query Parameter Only​

Diagnostic Output Example​

How to Fix​