Reverse proxy

A regular proxy works on the user's side and hides the user's address from the site. A reverse proxy sits on the site's side and hides the site's servers from visitors.

Through a reverse proxy, websites distribute load, cache pages, encrypt connections and filter suspicious requests. This is how CDNs and website protection services work.

When a request through a proxy is stopped by a protection page, it is often the site's reverse proxy that decided the request needed checking. Examples of such responses are on the 1020 error page.

You do not need to buy a reverse proxy to work with other people's websites: it is a tool for the site owner.

How a reverse proxy is set up

A typical example is nginx in front of an application. The block location / { proxy_pass http://127.0.0.1:8000; } passes all requests to the application on port 8000, while visitors see only nginx. The application does not see visitors' addresses directly: to it, every request comes from nginx.

To pass the real address on, nginx adds headers, for example proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;. The application should trust X-Forwarded-For only from its own proxy, because any client can write anything into this header. Caching is set up in a similar way: nginx can store the application's responses and serve them without contacting it.

Signs that a response came from a reverse proxy

It often shows in the response headers: Server with the name of a CDN, Via, Age with the age of a cached copy, or X-Cache: HIT. A HIT means the page came from the cache and the site's server did not even receive the request.

Errors differ too, and the status codes tell you where to look for the cause. 502 and 504 mean the reverse proxy did not get a response from the site's server, so the problem is on the site's side, not in your proxy. A 403 or 429 with the CDN's own page means the site's edge layer decided to limit requests.

What to do when a site limits you

Treat it as a signal from the site owner. Lower the pace and the number of threads, follow the Retry-After header if it is present, and check whether the site offers an official API or a data export. If you need the data regularly, write to the site owner: an agreed method is more stable than any setting on your side.