The Rules I Wrote Down Before Touching a Private API

A small project of mine needed data that a company exposes on its own website but not through any documented API. The data was about my listing — a business I run — and it was already visible in a browser to anyone who loaded the page.

That combination is the tempting one. It feels like it should be fine. It is your data, on a page you can already see, and the only thing standing between you and a tidy archive is that nobody published a REST endpoint for it.

I want to talk about what I decided before writing the collection code, because that turned out to matter more than any of the code. I am deliberately not publishing the technique — no endpoints, no header set, no request shape. If that is what you came for, this is not that article, and the last section explains why.

Rule 1: the scope is one listing, and the scope is the point

The first thing I wrote down was not a technical constraint. It was a sentence about what the project is allowed to be:

This collects reviews for the operator's own business listing. Keep it to that: do not build CAPTCHA-solving, detection-evasion, or mass-targeting tooling.

Not "start with one listing." Not "one listing for now." One listing, and the sentence goes on to name the three things that are out of bounds, so that the scope is a property of the project rather than a phase of it.

That sounds like a limitation you would relax later. It is actually the load-bearing wall, because almost every uncomfortable thing you can do with this kind of code is a scale change, not a kind change. Fetching your own reviews and fetching ten thousand businesses' reviews are the same three functions with a different loop around them. There is no natural stopping point in between, which means the stopping point has to be declared up front and treated as a spec, not a preference.

Those three exclusions are doing specific work, so it is worth saying what each one rules out. No CAPTCHA solving — if the platform puts up a challenge, that is a conversation, and the answer is to stop rather than to automate the reply. No detection evasion — nothing whose purpose is to look like something I am not. No mass targeting — the loop does not grow.

I have delays between requests. It matters that I am precise about why: it is there so I do not put load on someone else's servers. It is not there to slip under a rate limiter. Those two motivations produce identical code and completely different projects, and the only place the difference is recorded is in a comment and in your own head. Write it down.

Rule 2: replay your own session, never forge one

The technical principle I settled on is narrow and worth stating precisely: use the material the browser already has, do not manufacture credentials.

I drive a real browser, logged in as me, to a page I am allowed to see. Whatever that page needs in order to make its own request, it obtains for itself, the way it always does. My code reuses that. It does not mint a token, does not reconstruct an auth scheme, does not reverse-engineer a signature.

The distinction is not cosmetic. Replaying a session you legitimately hold is closer to "I automated my own clicking" than to "I broke in." Forging credentials means you have modelled the auth system well enough to produce something it did not issue — and at that point, whatever you tell yourself about intent, you have built the load-bearing part of a very different tool.

There is a corollary I did not expect to find useful. These private endpoints validate more than one thing about a request: they expect a coherent picture of who is asking, and an inconsistent picture gets refused. When I first got that refusal, my instinct was to treat it as a puzzle. Which piece is it checking? What do I have to match?

That instinct is the exact wrong one, and noticing it was the most useful moment in the project. The check is not an obstacle between me and my data. It is the platform describing what it considers a legitimate request. The right response is to make my request genuinely legitimate — same session, same context, same identity — rather than to determine the minimum set of properties that produces a 200.

If you find yourself enumerating which signals a service inspects so you can satisfy them without being the thing they describe, stop. That is the line. It does not announce itself, and you will not get a second, clearer warning.

Rule 3: the captures never enter git

This one is mundane and I nearly got it wrong.

When you are figuring out an undocumented response shape, you save examples. Raw dumps, sample payloads, a scratch file with a real response pasted in so you can write a parser against it. Then you write tests, and the tests want fixtures, and the fixtures are right there.

Committing those would have published live private request and response material to a repository — including whatever session-shaped material rode along in the capture — permanently, in git history, where deleting the file later does nothing.

So the rule became: raw captures are local diagnostic evidence and are git-ignored. Tests use invented fixtures.

Synthetic fixtures cost a little more to write and are better tests anyway. A real capture pins your test to one moment of someone else's production data; an invented fixture pins it to the shape your parser claims to handle. When the format changes, the synthetic fixture fails for the right reason.

One distinction I did keep: the curated output — the reviewed, cleaned archive the project exists to produce — is committed on purpose. That is my data in a form I chose. The difference between "raw capture of a private exchange" and "reviewed project output" is worth encoding in your .gitignore rather than in your memory of which file is which.

Rule 4: assume it breaks, and keep the blast radius small

The endpoints are private and unstable. That is not a complaint, it is the deal: nobody promised me an interface, so nobody owes me a deprecation notice. It will break, probably without warning, possibly this week.

The design response is to keep the request and parsing code isolated behind a small surface, so that when it breaks I can repair that and nothing else. This sounds like ordinary good structure, and it is, but it has a specific defensive purpose here: it keeps a breakage from becoming a scope negotiation.

When something stops working and the fix is tangled through the whole project, the cheapest path is often to do something slightly broader than before — a different entry point, a wider query, one more thing scraped while you are in there. Isolation makes repair boring, and boring repairs do not expand scope.

Why there is no code in this article

I have written this whole piece without naming the platform, the endpoints, or the request shape, and that is the deliberate part.

The discipline generalizes. The recipe does not — it collects into exactly the mass-collection tool that rule 1 exists to forbid. A working replay recipe is roughly the same artifact whether it runs against one listing or fifty thousand; the restraint lives entirely in who is holding it. I am reasonably confident in my own restraint. I have no way to publish it alongside the code.

And publication is one-directional in a way that a private note is not. A repository I can make private; a page that has been indexed, cached, scraped, and folded into a training set is out of my hands the moment it goes up. So the thing I can responsibly share is the part that makes someone more careful rather than more capable.

If you are about to do something like this, the checklist that actually helped:

1. Write the scope as a property of the project, not a phase. "One listing" beats "one listing for now."
2. Never manufacture credentials. Replay a session you legitimately hold, or stop.
3. Treat a refusal as a statement of intent, not a puzzle. The moment you start enumerating checks to satisfy, you have changed projects.
4. Keep captures out of version control. Test with invented fixtures.
5. Isolate the fragile part, so that repairing it never becomes an argument about scope.

None of that is technically difficult. All of it is easier to decide before you have working code than after.