Proxy setup in Crawl4AI

A step-by-step guide to connecting Prokso proxies in Crawl4AI. It takes a few minutes.

Step 1. Get the proxy details from your dashboard

Open your plan and copy the host gw.prokso.com, port, username and password. Port 823 is HTTP and 824 is SOCKS5 (both give a new IP for every request); ports 10000–20000 keep one IP (sticky session) for up to 120 minutes and work as both HTTP and SOCKS5 (each port is a separate address). Set the country in the username: username__cr.ua.

Step 2. Install Crawl4AI

pip install -U crawl4ai, then crawl4ai-setup: this command installs the browser Crawl4AI uses to open pages. You can check the installation with crawl4ai-doctor. This guide is written for version 0.9.5.

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, ProxyConfig

run_config = CrawlerRunConfig(
    proxy_config=ProxyConfig(
        server="http://gw.prokso.com:823",
        username="USERNAME",
        password="PASSWORD",
    )
)


async def main():
    async with AsyncWebCrawler(config=BrowserConfig()) as crawler:
        result = await crawler.arun(url="https://api.ipify.org", config=run_config)
        print(result.success, result.markdown)


asyncio.run(main())

Save the code to a file such as crawl.py and run it with python crawl.py.

Step 3. Pass the proxy to CrawlerRunConfig

Create a ProxyConfig with the address http://gw.prokso.com:823 and separate username and password fields, and pass it as the proxy_config parameter of CrawlerRunConfig. That's what the official docs recommend: the proxy is set for each arun call, and the proxy parameter on BrowserConfig is marked as deprecated there.

Step 4. Username, password and country

Take the username and password from the dashboard generator. Separate username and password fields are easier than a URL with the password in it: special characters don't need encoding. The username sets the country: USERNAME__cr.de opens pages from German addresses. Use the http:// scheme for the browser: Chromium only sends a username and password to an HTTP proxy.

Step 5. Pick a port for the job

Port 823 gives a new IP on every request, which suits one-off collection of separate pages. If the crawler walks through several pages of one website in one session (signing in, a cart, pagination), use a sticky port from 10000-20000: the website ties the session to the IP, and an address change in the middle of it looks suspicious. The address on a port holds for 1 to 120 minutes; you set the time in the dashboard.

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, ProxyConfig


def sticky(port):
    return CrawlerRunConfig(proxy_config=ProxyConfig(
        server=f"http://gw.prokso.com:{port}",
        username="USERNAME",
        password="PASSWORD",
    ))


async def main():
    async with AsyncWebCrawler(config=BrowserConfig()) as crawler:
        for port in (10001, 10002):
            config = sticky(port)
            for _ in range(2):
                result = await crawler.arun(url="https://api.ipify.org", config=config)
                print(port, result.markdown)


asyncio.run(main())

Save the code to a file such as crawl.py and run it with python crawl.py.

Step 6. Several sessions - one port each

Because the proxy lives in CrawlerRunConfig, you can build a separate configuration with its own sticky port for each session. The example uses two ports: requests through one port return the same IP, and different ports give different addresses.

Step 7. Check the output

Run the example: the crawler opens https://api.ipify.org and prints the IP the website sees. It should be the proxy's address, not yours. A 407 error means a wrong username or password, or the plan has run out of traffic. The My IP page in your own browser won't help here: the script launches its own browser, while yours goes direct.

arun_many with a sticky port per worker

arun_many() takes the whole list of URLs and opens pages in parallel through a dispatcher. The proxy lives in CrawlerRunConfig, so with one config every page uses the same proxy: on port 823 each page gets a new IP. If pages should come from a fixed set of addresses, split the list and give each part its own sticky port.

import asyncio
import os
from crawl4ai import (AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig,
                      MemoryAdaptiveDispatcher, ProxyConfig, RateLimiter)

PASSWORD = os.environ["PROKSO_PASSWORD"]  # keep the password out of the code

URLS = [f"https://example.de/catalog?page={n}" for n in range(1, 31)]
PORTS = [10001, 10002, 10003]  # one sticky port = one IP per worker


def config_for(port):
    return CrawlerRunConfig(
        proxy_config=ProxyConfig(
            server=f"http://gw.prokso.com:{port}",
            username="USERNAME__cr.de",
            password=PASSWORD,
        ),
        cache_mode=CacheMode.ENABLED,  # do not download again what is already cached
        page_timeout=90000,  # milliseconds
    )


async def worker(crawler, port, urls):
    dispatcher = MemoryAdaptiveDispatcher(
        max_session_permit=2,
        rate_limiter=RateLimiter(base_delay=(2.0, 5.0), max_retries=2),
    )
    results = await crawler.arun_many(urls=urls, config=config_for(port), dispatcher=dispatcher)
    for r in results:
        print(port, r.url, r.status_code if r.success else r.error_message)


async def main():
    browser = BrowserConfig(text_mode=True)  # no images, fonts or media
    async with AsyncWebCrawler(config=browser) as crawler:
        parts = [URLS[i::len(PORTS)] for i in range(len(PORTS))]
        await asyncio.gather(*(worker(crawler, p, part) for p, part in zip(PORTS, parts)))


asyncio.run(main())

MemoryAdaptiveDispatcher with max_session_permit=2 keeps at most two pages open at once per worker, and RateLimiter adds a random pause of 2-5 seconds between requests to one domain. page_timeout is in milliseconds, 60000 by default: through residential addresses pages load slower than directly, so 90 seconds is a reasonable margin. Set the IP hold time in the dashboard longer than one worker's share of the list takes.

Reading Crawl4AI error_message: net::ERR codes and timeouts

Crawl4AI doesn't raise an exception for a failed page: result.success is False and the reason is in result.error_message, after "Failed on navigating ACS-GOTO". The most common reasons through a proxy:

• net::ERR_PROXY_CONNECTION_FAILED. Chromium couldn't connect to the proxy: wrong host or port in server, or outgoing connections to that port are blocked on your machine or in the container.
• net::ERR_TUNNEL_CONNECTION_FAILED. The proxy is reachable but didn't open a tunnel to the website. Check that the plan has traffic and isn't paused, that the website isn't on the blocked sites list, and that it runs on an open port such as 80, 443, 8080 or 8443.
• Timeout 60000ms exceeded. The page didn't load within page_timeout. Raise it and keep wait_until="domcontentloaded" (the default): waiting for networkidle on pages with endless background requests often ends in a timeout.
• success is True but status_code is 403 or 429. The website answered but refuses. RateLimiter reacts to 429 and 503 by default: it doubles the pause for that domain and stops after max_retries such answers in a row. For 403 slow down yourself and check the website's rules.

Saving traffic in Crawl4AI: cache, text_mode and robots.txt

A browser downloads everything a page asks for, and traffic is counted in both directions. In version 0.9.5 CrawlerRunConfig doesn't read the cache by default, so every run downloads pages again. CacheMode.ENABLED reads and writes the local cache, and a second run takes pages you already have from disk.

BrowserConfig(text_mode=True) blocks images, fonts and media files and turns off JavaScript. That cuts traffic a lot, but pages that build their content with JavaScript come back empty: for them leave text_mode off and use avoid_css=True instead. Note that exclude_all_images and exclude_external_images only clean the result: the images are still downloaded.

check_robots_txt=True makes the crawler read robots.txt and skip disallowed pages with the message "Access denied by robots.txt". It's a simple way to keep the crawl within the website's rules and our acceptable use policy.

If the proxy does not work

• Proxy login in Puppeteer and Playwright
• 407 Proxy Authentication Required
• ERR_TUNNEL_CONNECTION_FAILED