Headless browser

Pages built by scripts do not return data to a plain request: instead of prices there are empty blocks. A headless browser loads the page fully, like a person in Chrome, but without a window on screen.

The most popular tools are Puppeteer, Playwright and Selenium. The proxy is set for them when the browser is launched.

The browser loads everything: images, fonts, video. Because of this, it uses much more traffic than a plain request. Unneeded resources are worth blocking in the script settings.

The traffic calculator will help you estimate the volume.

A proxy with a login in Playwright and Puppeteer

Chrome's --proxy-server flag takes only the host and port, without a username and password. That is why the credentials are passed separately. In Playwright they go into the launch options: proxy: { server: "http://gw.prokso.com:823", username: "LOGIN", password: "PASSWORD" }.

In Puppeteer you launch the browser with --proxy-server=gw.prokso.com:823 and call page.authenticate({ username: "LOGIN", password: "PASSWORD" }) before opening the page. Another option is to add the server's IP to the plan's IP whitelist, and then no username or password is needed at all.

How to cut off heavy resources

Resources the browser does not load do not pass through the proxy and are not counted as traffic. In Playwright, page.route lets you abort requests by type: for example, reject image, media and font and let everything else through. In Puppeteer the same is done with page.setRequestInterception(true).

Do not block scripts and XHR requests: the data you need usually arrives through them. After the setup, check the result on a few pages. Another saving is to reuse one browser for many pages instead of starting a new one each time: cached scripts and styles are not downloaded again.

How to treat sites gently

A headless browser loads a site the same way a person does, only much faster. Open pages one at a time or in a few tabs per site, and wait for the needed element with waitForSelector instead of fixed pauses. Before you start, read the site's robots.txt and check whether it offers an official API.

In Playwright every browser.newContext() gets its own cookies and storage. Close the contexts you no longer need: this frees memory and discards their cookies.