Scrape Google AI Mode Citations with Residential Proxies (2026)

AI Mode citation pipeline

Pin the market
Country (and city when needed) on the Elite username matches gl, hl, and Accept-Language so the answer is the one a household in that market would see
Launch a browser context
Playwright Chromium with credentials in username/password fields and a per-context residential proxy — not a user:pass URL, not requests
Render AI Mode
Wait for the conversational surface and citation chips; treat DOM drift as expected and extract stable citation hrefs rather than generated class names
Classify the document
Soft 200 with no AI Mode, challenge/consent interstitial, or a populated answer — three labels, never one empty bucket
Key the citations
Store query, market, collected_at, resolved exit country, answer text, and citation URLs. Empty is not written as not cited

Form pairing gl and hl with a matching country residential exit and optional city targeting. — gl and hl must match the exit country. A mismatched locale returns the wrong answer set.

How to collect Google AI Mode citations with Playwright

1
Pin Elite and make locale agree
Start on Elite (proxy.aethyn.io:5499). Encode the market on the username (-country-de) and pass matching gl, hl, and Playwright locale. A country-de exit with gl=us is a contradictory session and a wrong answer set. For local-language or local-pack prompts, add Elite city targeting and verify the city you landed in before you trust the row.
cURL
```
curl -x "http://aethyn-XXXXX-country-de:PASSWORD@proxy.aethyn.io:5499" \
  "https://www.google.com/search?q=best+residential+proxies&hl=de&gl=de"
```
Field note: cURL here is only to prove the exit and locale. Do not parse AI Mode from this response — the conversational UI will not be in it.

Launch Playwright with a per-context residential proxy

Put credentials in username and password fields. Chromium strips user:pass from a proxy URL and answers 407. Launch with Playwright’s per-context placeholder, then set the real Elite proxy on each newContext. One context per independent prompt so cookies never leak onto the next exit.

Python (Playwright)

from playwright.sync_api import sync_playwright

PROXY = {
    "server": "http://proxy.aethyn.io:5499",
    "username": "aethyn-XXXXX-country-de",
    "password": "PASSWORD",
}

def new_context(browser, locale="de-DE"):
    return browser.new_context(proxy=PROXY, locale=locale)

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": "per-context"}, headless=True)
    context = new_context(browser)
    page = context.new_page()
    page.goto("https://www.google.com/?hl=de&gl=de", wait_until="domcontentloaded")
    # Drive the AI Mode UI from here — entry chrome changes; classify the document.
    context.close()
    browser.close()

Field note: If per-context proxies are ignored, you launched Chromium without the placeholder. See the Playwright residential guide — launch proxy: { server: 'per-context' }, then set the real gateway on newContext().

3
Render AI Mode and extract citations, not class names
Wait for the conversational surface and for citation chips to settle. Google rotates generated class names; treat them as decoration. Stable enough signals are: an answer container that is not a SERP #rso list, numbered citation pills, and outbound hrefs that look like publisher URLs rather than google.com/search internals. Store those hrefs. If the Overview block is present on a classic SERP, that is a different product — see the AI Overviews post and do not merge the rows.
Python (classify + extract)
```
CHALLENGE_MARKERS = ("unusual traffic", "sorry/index", "cf-challenge", "verify you are human")

def classify_ai_mode(html: str, citation_hrefs: list[str]) -> str:
    low = html.lower()
    if any(m in low for m in CHALLENGE_MARKERS):
        return "challenge"
    if citation_hrefs:
        return "cited"
    # Rendered shell, no citations — could be "no AI Mode for this query/IP"
    return "soft_empty"

# Never write soft_empty as cited=False. Write status=soft_empty instead.
```
Field note: Entry URLs for AI Mode drift (query params, app chrome, consent). Keep a small adapter around navigation; keep classification and citation keying stable.
4
Classify missing answers before you store them
A 200 with no citations is not automatically “this brand was not cited.” It may be a consent wall, a /sorry/ interstitial, a challenge page, or a genuine empty — Google did not attach AI Mode to this query from this IP. Those four outcomes must not share one boolean. Fail closed: if you cannot name the document, do not persist a GEO metric.
Python (record shape)
```
record = {
    "query": "best residential proxies",
    "market": "de",           # gl/hl + intended exit country
    "collected_at": "2026-09-12T12:00:00Z",
    "resolved_country": "DE", # from an IP lookup at session start
    "status": "cited",        # cited | soft_empty | challenge | consent
    "answer_text": "...",
    "citations": ["https://example.com/a", "https://example.com/b"],
}
```
Field note: Assert resolved_country (and city, if you asked for one) before the first Google request. A mis-typed username is a silent wrong market, not a block.
5
Rotate per prompt; sticky only for consent
Close the context when the prompt is done so the next prompt gets a new exit. Use a sticky session only while a consent or challenge cookie must stay on the IP that earned it — typically well inside the default 30-minute window — then drop the session id. Do not pin one IP for a 5,000-keyword morning run.
Python (sticky username)
```
# Same rotating-pool IP for ~30 minutes while a consent cookie lives
user = "aethyn-XXXXX-country-de-session-aimode42-lifetime-30"
proxy = {
    "server": "http://proxy.aethyn.io:5499",
    "username": user,
    "password": "PASSWORD",
}
```
Field note: Sticky holds a rotating-pool residential IP. It is not a static, dedicated, or ISP product. Elite ISP suffixes are targeting, not a different SKU.

Why scrape Google AI Mode instead of the Overview block

GEO teams now buy measurement of citations in AI answers, not only blue-link rank. Otterly, Cloro, and Profound sell the dashboard; someone still has to collect the raw conversation from the market the buyer cares about. Google AI Overviews sit above organic results on a classic SERP. Google AI Mode is a different surface: a Gemini conversational UI that streams an answer and attaches numbered citations. Collecting the Overview HTML and labeling it “AI Mode” silently mixes two products. Collecting AI Mode with requests — or from a US datacenter while reporting a DE market — writes a 200 that describes nobody. The failure is quiet: dashboards stay green, “not cited” rates look decisive, and a brand lead in Berlin says that answer has never been what they saw. This guide is the collector for AI Mode citations. The Overview block has its own post. Classic rank still lives under SERP tracking.

What breaks AI Mode citation collection

AI Mode is a rendered conversation — there is no stable public JSON endpoint to poll
Entry URLs and DOM chrome change; generated class names are not a parser contract
Answers localize by exit geography and language; a wrong country is a wrong dataset
Soft 200s (no AI Mode for this query/IP) look like “not cited” if you do not classify them
Consent walls and challenges bind cookies to the IP and browser that earned them

Session rail rotating a fresh household exit per prompt, with a short sticky window only while a consent cookie lives. — Sticky is for identity on a consent or challenge. New prompt, new exit.

How Aethyn Elite fits Google AI Mode collection

AI Mode collection is a high-defense, geo-true workload. Aethyn Elite maps onto that: city and ISP on the username, HTTP and SOCKS5 on the same residential pool, sticky lifetime encoded next to the session id. Premium remains the cheaper country pool when you are sampling Overviews at keyword scale rather than driving the conversational UI.

Elite port 5499 (HTTP) / 3499 (SOCKS5) with -country-XX and optional -city- / ISP for local answers
Per-request rotation by omitting a session id so each independent prompt gets a fresh household exit
Sticky 1–1440 minutes (default 30) as -session-…-lifetime-N only while a consent or challenge lives
Premium port 2099 when the job is high-volume AI Overview sampling on a milder SERP
One credential, many markets — change the username, not the gateway

Google AI Mode citations FAQ

Is Google AI Mode the same as AI Overviews?

No. AI Overviews are the generated block that can appear above organic results on a classic SERP. AI Mode is a separate Gemini conversational surface with streaming answers and citation chips. Collect them as different objects. The Overview how-to is the AI Overviews residential-proxies post; this page is AI Mode citations.

Can I scrape Google AI Mode with Python requests?

Not usefully. AI Mode is a rendered UI. requests sees a SERP or a challenge, fails TLS fingerprint checks on defended queries, and will not hydrate citation chips. Use Playwright through a residential proxy so the browser supplies a real TLS stack and can wait for the conversation to render.

Why Elite instead of Premium?

AI Mode is a high-defense Google surface. Elite is the reputation pool and adds city and ISP targeting when local-language or local-pack answers matter. Premium is the right cheaper pool for high-volume AI Overview sampling on milder SERPs — country targeting, rotate often — not the default for this conversational UI.

How do I match the proxy to Google’s locale?

Put -country-XX on the Elite username and set gl, hl, Accept-Language, and Playwright locale to the same market. Then verify the exit country with an IP lookup. A mismatched pair produces an answer no real searcher saw and is easier for Google to score as inconsistent.

What should I store when AI Mode does not appear?

Store a status, not a fake negative citation. Use challenge or consent when you see those documents, and soft_empty when the UI loaded but attached no answer for this query and IP. Do not write cited=false unless you actually rendered an answer that omitted the brand.

Should I rotate or use a sticky session?

Rotate: one household exit per independent prompt. Sticky (1–1440 minutes, default 30) only while a consent wall or challenge cookie must stay on the IP that earned it. Close the Playwright context when the prompt is done. This is not a static IP product.

Do you scrape ChatGPT Search in this guide?

No. ChatGPT Search is named on the AI search visibility buyer page because GEO teams track it. This how-to stays on Google AI Mode — a public rendered UI — rather than a logged-in Cloudflare application. Do not copy this pipeline onto chatgpt.com.

HTTP or SOCKS5?

Same Elite pool. HTTP CONNECT on port 5499 is the usual Playwright path. SOCKS5 on 3499 if the client requires it. There is no second residential SKU.

Best practices for AI Mode citation pipelines

Collect public AI Mode answers only. Do not log in, do not solve CAPTCHAs, and do not treat a proxy as a license to ignore Google’s terms.
Match exit country, gl, hl, and Playwright locale on every context. Contradictory locale is both a bad dataset and a bot tell.
Verify the exit with an IP lookup at the start of the context and store resolved_country next to the citations.
Key rows on query + market + timestamp. Citations are the GEO metric; organic rank belongs in a different table.
Classify challenge vs consent vs soft_empty vs cited before insert. Empty-as-absent is how GEO dashboards lie.
One Playwright context per independent prompt. Reusing a context across prompts leaks cookies onto a new identity.
Reserve sticky (~30 min default) for the identity that holds a consent or challenge cookie; rotate again when that flow ends.
Keep Overview collection on its own pipeline. Merging Overview HTML into an AI Mode table poisons both products.

Common mistakes when scraping Google AI Mode

Using requests or httpx and parsing a SERP as if it were AI Mode.
Reporting a US datacenter crawl as a DE (or any other) market’s citations.
Storing a 200 with zero citations as cited=false instead of status=soft_empty or challenge.
Embedding user:pass in the Playwright proxy server URL and debugging the resulting 407 as a Google block.
Rotating the IP while a consent cookie is still required, then calling the next empty page “not cited.”
Pinning one sticky IP for the entire keyword list and recreating a per-IP velocity problem.
Scraping ChatGPT.com as the first GEO how-to — login, Cloudflare, and terms make that a different, worse project.

AI Mode citation pipeline

Pin the market
Country (and city when needed) on the Elite username matches gl, hl, and Accept-Language so the answer is the one a household in that market would see
Launch a browser context
Playwright Chromium with credentials in username/password fields and a per-context residential proxy — not a user:pass URL, not requests
Render AI Mode
Wait for the conversational surface and citation chips; treat DOM drift as expected and extract stable citation hrefs rather than generated class names
Classify the document
Soft 200 with no AI Mode, challenge/consent interstitial, or a populated answer — three labels, never one empty bucket
Key the citations
Store query, market, collected_at, resolved exit country, answer text, and citation URLs. Empty is not written as not cited

How to collect Google AI Mode citations with Playwright

1
Pin Elite and make locale agree
Start on Elite (proxy.aethyn.io:5499). Encode the market on the username (-country-de) and pass matching gl, hl, and Playwright locale. A country-de exit with gl=us is a contradictory session and a wrong answer set. For local-language or local-pack prompts, add Elite city targeting and verify the city you landed in before you trust the row.
cURL
```
curl -x "http://aethyn-XXXXX-country-de:PASSWORD@proxy.aethyn.io:5499" \
  "https://www.google.com/search?q=best+residential+proxies&hl=de&gl=de"
```
Field note: cURL here is only to prove the exit and locale. Do not parse AI Mode from this response — the conversational UI will not be in it.

Launch Playwright with a per-context residential proxy

Python (Playwright)

from playwright.sync_api import sync_playwright

PROXY = {
    "server": "http://proxy.aethyn.io:5499",
    "username": "aethyn-XXXXX-country-de",
    "password": "PASSWORD",
}

def new_context(browser, locale="de-DE"):
    return browser.new_context(proxy=PROXY, locale=locale)

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": "per-context"}, headless=True)
    context = new_context(browser)
    page = context.new_page()
    page.goto("https://www.google.com/?hl=de&gl=de", wait_until="domcontentloaded")
    # Drive the AI Mode UI from here — entry chrome changes; classify the document.
    context.close()
    browser.close()

3
Render AI Mode and extract citations, not class names
Wait for the conversational surface and for citation chips to settle. Google rotates generated class names; treat them as decoration. Stable enough signals are: an answer container that is not a SERP #rso list, numbered citation pills, and outbound hrefs that look like publisher URLs rather than google.com/search internals. Store those hrefs. If the Overview block is present on a classic SERP, that is a different product — see the AI Overviews post and do not merge the rows.
Python (classify + extract)
```
CHALLENGE_MARKERS = ("unusual traffic", "sorry/index", "cf-challenge", "verify you are human")

def classify_ai_mode(html: str, citation_hrefs: list[str]) -> str:
    low = html.lower()
    if any(m in low for m in CHALLENGE_MARKERS):
        return "challenge"
    if citation_hrefs:
        return "cited"
    # Rendered shell, no citations — could be "no AI Mode for this query/IP"
    return "soft_empty"

# Never write soft_empty as cited=False. Write status=soft_empty instead.
```
Field note: Entry URLs for AI Mode drift (query params, app chrome, consent). Keep a small adapter around navigation; keep classification and citation keying stable.
4
Classify missing answers before you store them
A 200 with no citations is not automatically “this brand was not cited.” It may be a consent wall, a /sorry/ interstitial, a challenge page, or a genuine empty — Google did not attach AI Mode to this query from this IP. Those four outcomes must not share one boolean. Fail closed: if you cannot name the document, do not persist a GEO metric.
Python (record shape)
```
record = {
    "query": "best residential proxies",
    "market": "de",           # gl/hl + intended exit country
    "collected_at": "2026-09-12T12:00:00Z",
    "resolved_country": "DE", # from an IP lookup at session start
    "status": "cited",        # cited | soft_empty | challenge | consent
    "answer_text": "...",
    "citations": ["https://example.com/a", "https://example.com/b"],
}
```
Field note: Assert resolved_country (and city, if you asked for one) before the first Google request. A mis-typed username is a silent wrong market, not a block.
5
Rotate per prompt; sticky only for consent
Close the context when the prompt is done so the next prompt gets a new exit. Use a sticky session only while a consent or challenge cookie must stay on the IP that earned it — typically well inside the default 30-minute window — then drop the session id. Do not pin one IP for a 5,000-keyword morning run.
Python (sticky username)
```
# Same rotating-pool IP for ~30 minutes while a consent cookie lives
user = "aethyn-XXXXX-country-de-session-aimode42-lifetime-30"
proxy = {
    "server": "http://proxy.aethyn.io:5499",
    "username": user,
    "password": "PASSWORD",
}
```
Field note: Sticky holds a rotating-pool residential IP. It is not a static, dedicated, or ISP product. Elite ISP suffixes are targeting, not a different SKU.

Why scrape Google AI Mode instead of the Overview block

What breaks AI Mode citation collection

AI Mode is a rendered conversation — there is no stable public JSON endpoint to poll
Entry URLs and DOM chrome change; generated class names are not a parser contract
Answers localize by exit geography and language; a wrong country is a wrong dataset
Soft 200s (no AI Mode for this query/IP) look like “not cited” if you do not classify them
Consent walls and challenges bind cookies to the IP and browser that earned them

How Aethyn Elite fits Google AI Mode collection

Elite port 5499 (HTTP) / 3499 (SOCKS5) with -country-XX and optional -city- / ISP for local answers
Per-request rotation by omitting a session id so each independent prompt gets a fresh household exit
Sticky 1–1440 minutes (default 30) as -session-…-lifetime-N only while a consent or challenge lives
Premium port 2099 when the job is high-volume AI Overview sampling on a milder SERP
One credential, many markets — change the username, not the gateway

Google AI Mode citations FAQ

Is Google AI Mode the same as AI Overviews?

Can I scrape Google AI Mode with Python requests?

Why Elite instead of Premium?

How do I match the proxy to Google’s locale?

What should I store when AI Mode does not appear?

Should I rotate or use a sticky session?

Do you scrape ChatGPT Search in this guide?

HTTP or SOCKS5?

Same Elite pool. HTTP CONNECT on port 5499 is the usual Playwright path. SOCKS5 on 3499 if the client requires it. There is no second residential SKU.

Best practices for AI Mode citation pipelines

Collect public AI Mode answers only. Do not log in, do not solve CAPTCHAs, and do not treat a proxy as a license to ignore Google’s terms.
Match exit country, gl, hl, and Playwright locale on every context. Contradictory locale is both a bad dataset and a bot tell.
Verify the exit with an IP lookup at the start of the context and store resolved_country next to the citations.
Key rows on query + market + timestamp. Citations are the GEO metric; organic rank belongs in a different table.
Classify challenge vs consent vs soft_empty vs cited before insert. Empty-as-absent is how GEO dashboards lie.
One Playwright context per independent prompt. Reusing a context across prompts leaks cookies onto a new identity.
Reserve sticky (~30 min default) for the identity that holds a consent or challenge cookie; rotate again when that flow ends.
Keep Overview collection on its own pipeline. Merging Overview HTML into an AI Mode table poisons both products.

Common mistakes when scraping Google AI Mode

Using requests or httpx and parsing a SERP as if it were AI Mode.
Reporting a US datacenter crawl as a DE (or any other) market’s citations.
Storing a 200 with zero citations as cited=false instead of status=soft_empty or challenge.
Embedding user:pass in the Playwright proxy server URL and debugging the resulting 407 as a Google block.
Rotating the IP while a consent cookie is still required, then calling the next empty page “not cited.”
Pinning one sticky IP for the entire keyword list and recreating a per-IP velocity problem.
Scraping ChatGPT.com as the first GEO how-to — login, Cloudflare, and terms make that a different, worse project.

How to Collect Google AI Mode Answers and Citations

Key takeaways

On this page

AI Mode citation pipeline

Pin the market

Launch a browser context

Render AI Mode

Classify the document

Key the citations

How to collect Google AI Mode citations with Playwright

Pin Elite and make locale agree

Launch Playwright with a per-context residential proxy

Render AI Mode and extract citations, not class names

Classify missing answers before you store them

Rotate per prompt; sticky only for consent

Why scrape Google AI Mode instead of the Overview block

How Aethyn Elite fits Google AI Mode collection

Google AI Mode citations FAQ

Best practices for AI Mode citation pipelines

Solve this with Aethyn residential proxies

How to Collect Google AI Mode Answers and Citations

Key takeaways

On this page

AI Mode citation pipeline

Pin the market

Launch a browser context

Render AI Mode

Classify the document

Key the citations

How to collect Google AI Mode citations with Playwright

Pin Elite and make locale agree

Launch Playwright with a per-context residential proxy

Render AI Mode and extract citations, not class names

Classify missing answers before you store them

Rotate per prompt; sticky only for consent

Why scrape Google AI Mode instead of the Overview block

How Aethyn Elite fits Google AI Mode collection

Google AI Mode citations FAQ

Best practices for AI Mode citation pipelines

Solve this with Aethyn residential proxies

Key takeaways

On this page

AI Mode citation pipeline

Pin the market

Launch a browser context

Render AI Mode

Classify the document

Key the citations

How to collect Google AI Mode citations with Playwright

Pin Elite and make locale agree

Launch Playwright with a per-context residential proxy

Render AI Mode and extract citations, not class names

Classify missing answers before you store them

Rotate per prompt; sticky only for consent

Why scrape Google AI Mode instead of the Overview block

How Aethyn Elite fits Google AI Mode collection

Google AI Mode citations FAQ

Best practices for AI Mode citation pipelines

Related guides & resources

Solve this with Aethyn residential proxies

Key takeaways

On this page

AI Mode citation pipeline

Pin the market

Launch a browser context

Render AI Mode

Classify the document

Key the citations

How to collect Google AI Mode citations with Playwright

Pin Elite and make locale agree

Launch Playwright with a per-context residential proxy

Render AI Mode and extract citations, not class names

Classify missing answers before you store them

Rotate per prompt; sticky only for consent

Why scrape Google AI Mode instead of the Overview block

How Aethyn Elite fits Google AI Mode collection

Google AI Mode citations FAQ

Best practices for AI Mode citation pipelines

Related guides & resources

Solve this with Aethyn residential proxies