Threads (concurrent connections)

A scraper with 50 threads opens 50 connections at once and loads 50 pages in parallel. A browser also opens several connections per page: images, scripts, styles.

The more threads, the faster the work goes, but the greater the load on the site. If all threads go to one site, a rate limit will kick in quickly.

The number of threads and the request frequency are different things. You can have many threads and a low pace for each site by spreading requests across different sites. For large volumes of public data, rotation is used.

In Prokso up to 1000 concurrent connections are available per plan. Traffic is counted by volume, not by the number of threads.

How many threads to start with

Begin with 5-10 threads per site and watch the responses for a few minutes. If almost all of them are 200 and the response time stays flat, add threads in small steps. As soon as 429, 503 or timeouts appear, go back to the previous step and stay there.

Threads and IP addresses in Prokso

On ports 823 and 824 every request gets a new IP, so 50 threads mean 50 different exit addresses at once. On a port from the 10000-20000 range all threads share one sticky IP for the holding time. If a task needs ten stable addresses at the same time, use ten different ports, one sticky session per port.

The limit of 1000 connections belongs to the plan, not to a program. If two scripts use the same plan username, they share this limit between them.

A connection is not the same as a request

With keep-alive one connection carries many requests in a row, so 20 threads can make thousands of requests without opening a new connection each time. Browsers limit connections too: Chrome opens up to 6 at once to one host over HTTP/1.1. Over HTTP/2 a browser usually sends all requests to a host through a single connection.