What rotation does
A rotating proxy sends each request through a different exit address from a pool. The intent is simple: a site that counts requests per address never sees enough from any one of them to act. Against rate limiting by address, this works exactly as advertised.
Rotation happens in two shapes. Per request, where every call gets a new address, and per session, where a chosen identifier keeps the same address for a window of time. Both are useful, and using the wrong one is a common cause of a crawl that fails for no visible reason.
When per request rotation is right
Independent pages, no state, nothing to carry between calls. Product pages, articles, listings, search results, documentation. Each request stands alone, so a fresh address costs nothing and spreads your volume across the pool.
This is the default and covers most of what a crawl does.
When you need a sticky session
The moment a site remembers something about the visitor between requests, rotation becomes the thing that breaks you. Consider what the site sees: a visitor opens page one from Frankfurt, page two from São Paulo four hundred milliseconds later, and page three from Chicago. No human does that. A crawl that looked plausible on single pages becomes obviously automated on flows.
Sticky sessions hold one address for a set window, up to thirty minutes on our Scale plan, so a multi step flow arrives from one consistent place:
- Pagination that carries a cursor or a session token
- Anything that sets a cookie on the first page and reads it on the second
- Localised pricing or availability that depends on where the session started
- Basket, quote and availability flows where each step builds on the last
Why requests still get banned
Here is the part that surprises people. The address is not the first signal a modern block reads, and on defended sites it is not the strongest one.
- TLS fingerprint: your client negotiates the connection in a way that does not match the browser it claims to be. This is checked before a single byte of HTML is sent.
- Header set and order: real browsers send a specific set in a specific order. A handwritten request usually sends fewer headers, tidier, in the wrong sequence.
- JavaScript execution: the page loads a script that computes a value and sends it back. A client that never runs scripts never returns it.
- Behavioural rhythm: exactly one request every two hundred milliseconds, forever, from every address in your pool, is a pattern no population of humans produces.
- Session history: a visitor with no cookies, no referrer and no prior page views, on every single request, is a visitor who has never existed.
- Fingerprint coherence: claiming a recent browser on one platform while presenting the TLS profile of a different client library is a contradiction, and contradictions are cheap to detect.
Rotating addresses fixes exactly one of those six. That is why the second proxy provider rarely helps, and why teams conclude the target is unscrapeable when the target is simply reading the other five signals.
Coherence beats volume
The goal is not to look like many visitors. It is to look like one plausible visitor, repeated. Every part of the identity has to agree: the TLS fingerprint matches the user agent, the headers match the browser version, the viewport matches the platform, the timing has variance in it, and the session carries the cookies a real visit would have collected.
A pool of ten thousand addresses attached to an incoherent client is worse than a hundred addresses attached to a coherent one, because you are producing ten thousand pieces of evidence rather than a hundred.
What retry policy should do
When a request is blocked, retrying it immediately through a different address and nothing else is the most common mistake in homemade crawlers. It confirms the pattern and burns the pool.
A retry should change the identity, not just the exit. New address, new fingerprint, new header set, a browser if the page needs one, and a pause with variance in it. It should also stop: three failed attempts on a target that is refusing everything means a parameter is wrong, and the fourth attempt will not discover that.
And the retries should not be on your bill. That is the argument for counting successful responses only, which is how our plans work on the best web scraping api pricing page.
Where this leaves you
If you are building this yourself, the order of work is: coherent client identity first, rendering second, rotation third, sticky sessions where flows demand them, and a retry policy that changes more than the address. Rotation alone was never the answer, which is why it stops working precisely when the target matters.
If you would rather not build it, the same work arrives as two request parameters: the country and the proxy type, with the identity handled underneath. The pool, the sessions and the retries are described on the web scraping proxy page, and the choice between pool types is in datacenter proxies versus residential.