You Don't Choose Your Scraping Strategy — the Platform's API Surface Does

I wanted the same boring thing twice: pull my own chat history out of an AI product and save it as Markdown. Once for Claude, once for Gemini. Same author, same monorepo, same output format, same week. I assumed I'd write one script and copy-paste it with the hostname swapped.

I ended up with two scripts that share almost no extraction code, and it took me a while to accept that this wasn't me being sloppy. The two platforms forced two completely different strategies, and the deciding factor had nothing to do with my preferences. It was entirely about what each site's private API happened to expose.

Here's the story of how the same problem produced opposite solutions, and why that's the interesting part.

The Setup

Both scripts are Tampermonkey userscripts. Both ride your already-authenticated browser session — no API keys, no tokens I have to store, just same-origin requests carrying the cookies you already have. Both emit Markdown plus a normalized JSON schema, bundle a bulk export into a dependency-free ZIP, and persist settings through GM_getValue / GM_setValue. From the outside they look like twins.

The moment you look at how each one actually reads a conversation, they stop looking related at all.

Claude: The Platform Handed Me a Clean Door

Claude Chat Exporter never touches the DOM. Not once.

When Claude.ai renders your conversation, it's driven by a same-origin JSON endpoint that returns the entire message tree in one structured payload. So the script does the obvious thing: it reads the conversation ID out of the URL, resolves your organization UUID (from the lastActiveOrg cookie, falling back to GET /api/organizations if the cookie isn't there), and fetches the whole thing directly.

// The conversation ID comes straight from the URL: claude.ai/chat/<id>
async function fetchConversation(orgId: string, chatId: string): Promise<Conversation> {
  const url =
    `/api/organizations/${orgId}/chat_conversations/${chatId}` +
    `?tree=True&rendering_mode=messages&render_all_tools=true`;
  const res = await fetch(url, { credentials: "include" });
  if (!res.ok) throw new Error(`conversation ${res.status}`);
  return res.json() as Promise<Conversation>;
}

That's the entire read path. Because the payload arrives already structured, the rich stuff comes along for free — extended thinking, tool calls paired with their results, extracted attachment text. I'm reading fields, not reverse-engineering rendered HTML. When the data is already JSON, "parsing" is just knowing the key names.

This is about as low-fragility as browser scraping gets. There's exactly one request shape to maintain, it's isolated in one place, and if Claude changes it, the repair is local.

I felt clever for about four days. Then I opened Gemini.

Gemini: There Was No Door, So I Learned to Pick the Lock

Gemini has no clean per-conversation JSON endpoint. Everything — loading a chat, listing your history, all of it — routes through Google's batchexecute RPC pipeline: one obfuscated endpoint, rotating build parameters, RPC IDs like hNvQHb, and payloads that are serialized arrays nested inside serialized arrays. There is no polite GET /conversation/{id} waiting for you.

So the script splits by workload, and neither half looks anything like the Claude version.

Single chat falls back to scraping the rendered DOM. Gemini lazy-loads older turns as you scroll up, so the script first scrolls the container to the top repeatedly until the rendered turn count stops changing, then does a single-pass extraction:

// Gemini lazy-loads older turns on upward scroll but does NOT evict rendered
// nodes, so scroll to the top until the count holds steady twice, then collect.
async function ensureAllTurnsLoaded(): Promise<void> {
  const scroller = document.querySelector(SEL.scroller);
  if (!scroller) return;
  let prev = -1;
  let stable = 0;
  for (let i = 0; i < 60 && stable < 2; i++) {
    const count = document.querySelectorAll(SEL.turn).length;
    stable = count === prev ? stable + 1 : 0;
    prev = count;
    scroller.scrollTop = 0;
    await new Promise((r) => setTimeout(r, 400));
  }
}

Note the belt-and-suspenders bits I would not have written from memory: it waits for the count to hold steady across two passes, not one (a single stable read can catch a mid-load pause), and it caps the loop at 60 iterations so a chat that never stabilizes can't hang the export forever.

Bulk export is where it gets properly gnarly. You can't navigate conversation-to-conversation from a userscript — I verified every path and they're all blocked. A hard navigation reloads the page and kills the script's execution context mid-loop. history.pushState keeps the context alive but Angular's router refuses to actually load the conversation. A synthetic click on a sidebar item doesn't fire the router either, because a userscript can't forge a trusted event. A same-origin iframe is blocked by frame-ancestors.

Every door was locked. So the only way in is Gemini's own data API — the same batchexecute its frontend uses. And the only sane way to call it is to observe, then replay:

1. At document-start, patch the page's network layer (XMLHttpRequest, and fetch defensively) before the app boots.
2. When you naturally open a chat, the interceptor catches the outgoing batchexecute request and learns the template — the URL params, the headers, the request-body envelope, and the exact spot where the conversation ID lives.
3. To export everything, replay that learned template with swapped IDs, one paced request at a time so Google doesn't get twitchy.

The beautiful part is that replaying the app's real, current request means the script self-heals across Google's weekly build rotations. When the bl build label or the session tokens rotate, the next intercepted request just teaches the script the new values automatically. I never touch the code for that.

The ugly part is that two things stay hardcoded — the RPC IDs (hNvQHb for content, the list RPC for history) and the leaf paths where prompt and response text sit inside that nested-array response. If Google rotates an RPC ID or reshapes the payload, self-healing doesn't save me; it's a manual one-line update. (The script prints the learned RPC IDs to the console precisely so future-me can find the new value quickly. You're welcome, future-me.)

The Part I Actually Want You to Take Away

I kept trying to make the Gemini script more like the Claude one. Cleaner. More direct. Fetch the JSON, read the fields, done. It was never going to happen, and here's why that isn't a skill issue:

The extraction strategy isn't a design decision. It's dictated by the target's API surface.

A clean same-origin JSON API (Claude) invites a direct fetch: high data fidelity, minimal fragility, one request shape to babysit. An obfuscated RPC pipeline with no per-resource endpoint (Gemini) forces the observe-replay-plus-DOM approach — more moving parts, more fragility, but it's the only viable access path. You don't get to pick the resilient option when the platform only offers the brittle one.

Which flips how I now scope this kind of tool. The first question isn't "how do I want to build this." It's "what does this platform's private API actually expose," because the honest answer to that question has already chosen your architecture before you've written a line. Claude's shape gave me a fetch call. Gemini's shape gave me a network interceptor with a self-healing template cache and a defensive nested-array walker. Same requirements, same author, same week — and the target picked, not me.

If you're about to build a scraper or an exporter for a site you don't control, spend your first hour in the Network tab, not the editor. Find out which door the platform left open. That reconnaissance decides more about your final code than any amount of taste will.

Both scripts are open source in my user-scripts monorepo if you want to see the two shapes side by side. And if you've fought batchexecute and found a cleaner path than observe-replay — please, genuinely, tell me. I'd love to delete some of that interceptor.