- Websites restrict addresses for requests that are too frequent, suspicious subnets and datacenter addresses
- Errors 429 and 403 and captchas are a signal to slow down, not to push harder
- A calm pace, pauses, caching and official APIs reduce restrictions more than changing addresses does
- Rotation suits data collection, a sticky address suits accounts
- Proxies do not cancel a website's rules: restrictions under a platform's terms apply to any address
An IP block is almost never random. A website sees behavior from an address that it does not like and restricts access. Sometimes for a minute, sometimes for a day, sometimes for a whole subnet at once.
The good news: most restrictions are predictable. If you understand why websites impose them, you can work calmly and within the rules. Below: why websites restrict addresses, how to recognize it and what to do to avoid trouble.
Why websites restrict addresses
The first reason is frequency. Most websites have a rate limit: how many requests from one address they are willing to accept per minute or hour. A script that opens dozens of pages per second hits this limit almost immediately.
The second is abuse. Mass registrations, spam, password guessing, load that looks like an attack. The addresses it came from end up in shared databases, and other websites start treating them with caution too.
The third is subnet reputation. If there is a lot of suspicious activity from neighboring addresses in the same block, a website may restrict the whole block. The fourth is the network type: the ASN number shows that an address belongs to a datacenter, and some websites do not trust such addresses by default.
How to tell that you have been restricted
A website rarely says outright "you have been blocked". Usually you can tell from the responses:
| Sign | What it usually means | What to do |
|---|---|---|
| Error 429 | Too many requests in a short time | Pause, slow down and reduce the number of threads |
| Error 403 | Access denied: the address, network type or a website rule | Check the address and proxy type, reread the website's rules |
| A captcha instead of a page | The website doubts that a person is visiting | Slow down, check the browser profile |
| Cloudflare 1015 | A rate limit was triggered on the website's protection side | Wait and come back at a lower rate |
| An empty page or different data | The website serves bots a simplified version | Compare with the response in an ordinary browser |
More about each error: 429 Too Many Requests, 403 Forbidden and Cloudflare 1015. If you get a captcha or 403 on a website you simply opened in a browser, check whether it is on the list of websites closed to everyone.
Pace matters more than the number of addresses
The most common mistake is trying to solve restrictions with more addresses. The website returns 429, and the script adds threads and keeps going. This only makes the restrictions worse: the website sees a wave of identical requests and tightens the screws for everyone.
The opposite works. The calmer your script behaves on a website, the less often it runs into restrictions. Here are the rules to start with:
- Put pauses between requests and make them random, for example 2 to 6 seconds, rather than exactly one second.
- Limit the number of simultaneous threads per website. Start with a few and increase only if there are no errors.
- Cache what you have already downloaded. Do not open the same page twice a day if it does not change.
- Collect at night in the website's time zone, when it has fewer live visitors.
- Stop immediately on 429 and respect the Retry-After header if there is one.
The technical limit at Prokso is up to 1000 simultaneous connections per plan. This is headroom for different tasks, not a recommendation: for one website, a reasonable number of threads is usually many times lower.
Introduce yourself to the website honestly
Many scripts send an empty or technical User-Agent by default. A website does not like it when it cannot tell who is visiting. For open data collection, it helps to put a clear program name and a way to contact you in the header.
If you regularly collect data from one website, it is sometimes easier to write to its owners. Small websites often offer an export or a convenient time for crawling themselves. This removes the question of restrictions completely and saves traffic for both of you.
The official way, if there is one
Before writing a scraper, check whether the website has an open API or a data export. Many stores, directories and services provide data through an API with clear limits. This is faster, uses less traffic and does not break any rules.
If there is no API, read robots.txt and the terms of use. In robots.txt, a website states which sections robots may crawl, and sometimes the preferred pause between requests. The terms of use say whether data collection is allowed at all. If a website explicitly forbids collection, proxies do not change that.
Rotation or a sticky address
The address change mode is chosen for the task. A rotating proxy gives a new address for every request: in Prokso this is port 823 for HTTP and 824 for SOCKS5. The load is spread across the pool, and no single address takes the whole stream.
A sticky address is kept for the duration of a session: ports from 10000 to 20000, each holding a single address for 1 to 120 minutes. You need it when a website expects the same visitor, for example when logging into an account or going through a multi-step checkout.
| Task | Mode | Why |
|---|---|---|
| Collecting open catalog pages | Rotation, port 823 or 824 | Requests are independent, the load is spread out |
| Logging into an account and working in it | Sticky port 10000-20000 | The platform sees one stable user |
| A multi-step process: search, filter, product page | Sticky port for the duration of the step | The website links the steps to one session |
| Checking search results from different cities | A sticky port for each city | The result does not jump between requests |
Rotation should not become a way to get around a website's limit. If you split the same flood of requests across a hundred addresses, the website sees it by other signs: the same browser, the same paths, the same intervals.
For accounts: a consistent profile
With accounts, an IP block is rarely only about the address. A platform looks at the whole picture: the address, the browser, the time zone, the language, the login history. If any of this does not add up, it asks for verification or restricts the login.
- One account, one browser profile and one sticky port.
- An address from the account's country. For an account from Kyiv, a Ukrainian address; for a Polish store, a Polish one.
- Browser time zone and language that match the address's country.
- No sudden change of country for an account that has been active for a long time.
- Home or mobile addresses for platforms that are strict about datacenters.
You can check how a website sees you on the my IP page: it shows the address's country, the time zone and WebRTC leaks.
What to do if you are already restricted
- Stop the task. Repeated requests in response to 429 or 403 extend the restriction.
- Wait. Rate limits are usually lifted on their own within minutes or hours.
- Find the cause: pace, number of threads, address type or the website's rules.
- If the address itself has a bad reputation, change it: take a different port or a new session. The pool settings have an anonymity filter that cuts out addresses websites have already marked as proxy or VPN.
- Come back at a slower pace and test on a small volume before running everything.
Why datacenters get restricted more often
Datacenter addresses come in blocks, and their owner is visible by ASN. A website that does not expect robots may close such networks entirely. If a website returns 403s and captchas to your datacenter addresses even at a calm pace, test the same task on residential ones. More in the article residential vs datacenter proxies.
Keep in mind that the pool is shared: another client may have used the address before you. So sometimes a restriction comes on the very first request, through no fault of yours. In this case changing the address helps, not repeating the request.
And finally: keep a simple log. Write down how many requests per minute went to the website, how many errors there were and at what time. After a week you will see at what pace the website responds calmly, and you can fix that pace in your script's settings.
FAQ
Can I be sure to avoid an IP block?
No. Each website decides for itself whom to restrict, and the rules change. You can reduce the risk: work at a reasonable pace, follow the website's rules and do not create unnecessary load.
What does error 429 mean?
The website is telling you that too many requests came from your address in a short time. You need to pause and lower the request rate. The response often has a Retry-After header that hints at how long to wait.
Will rotation help if a website returns 429?
Rotation spreads requests across different addresses, but it does not cancel the website's limits. If you simply continue with the same intensity, the restrictions will come back. First slow down, then decide whether you need rotation.
Why was I restricted on the very first request?
The address may already have had a bad reputation: the pool is shared, and someone else used it before. Another reason is that the website filters datacenter addresses. Try a different address or the residential type.
Do I need to follow robots.txt if I work through a proxy?
Yes. A proxy changes your address, but it does not change the website's rules. robots.txt and the terms of use apply to any address, and you should plan data collection around them.
