Proxy setup in LangChain

A step-by-step guide to connecting Prokso proxies in LangChain. It takes a few minutes.

Step 1. Get the proxy details from your dashboard

Open your plan and copy the host gw.prokso.com, port, username and password. Port 823 is HTTP and 824 is SOCKS5 (both give a new IP for every request); ports 10000–20000 keep one IP (sticky session) for up to 120 minutes and work as both HTTP and SOCKS5 (each port is a separate address). Set the country in the username: username__cr.ua.

Step 2. Install the packages

pip install -U langchain-community beautifulsoup4. The WebBaseLoader page loader lives in the langchain-community package, and it needs beautifulsoup4 to parse HTML. Regular loading goes through the requests library and async loading through aiohttp, and the proxy is set differently for each.

Step 3. Pass the proxy to the loader

For load() and lazy_load() pass the proxies parameter: a dictionary with the proxy URL for http and https. Put the username and password straight into the URL. The requests_kwargs parameter passes extra settings to every request, such as timeout, so the script doesn't hang on a slow page.

from langchain_community.document_loaders import WebBaseLoader

PROXY = "http://USERNAME:PASSWORD@gw.prokso.com:823"

loader = WebBaseLoader(
    "https://api.ipify.org?format=json",
    proxies={"http": PROXY, "https": PROXY},
    requests_kwargs={"timeout": 30},
)
docs = loader.load()
print(docs[0].page_content)

Step 4. Async loading uses environment variables

alazy_load() and fetch_all() (and the deprecated aload()) run on aiohttp and ignore the proxies parameter. For them, set the HTTP_PROXY and HTTPS_PROXY variables and create the loader with trust_env=True: without it aiohttp ignores the variables. AsyncHtmlLoader is set up the same way.

import asyncio
import os

os.environ["HTTP_PROXY"] = "http://USERNAME:PASSWORD@gw.prokso.com:823"
os.environ["HTTPS_PROXY"] = "http://USERNAME:PASSWORD@gw.prokso.com:823"

from langchain_community.document_loaders import WebBaseLoader


async def main():
    loader = WebBaseLoader(
        web_paths=["https://api.ipify.org?format=json"] * 3,
        trust_env=True,
        requests_per_second=2,
    )
    async for doc in loader.alazy_load():
        print(doc.page_content)


asyncio.run(main())

Step 5. Environment variables for other loaders

The requests library reads HTTP_PROXY and HTTPS_PROXY by default, so variables set before the script starts also apply to requests-based loaders that have no proxy parameter of their own. Keep in mind that, according to the requests docs, environment variables override a session-level proxy, which is what the proxies parameter of WebBaseLoader sets. Pick one method per script.

Step 6. Pick a port and pace

Port 823 gives a new IP for every page, which suits collecting many pages. To load a series of pages from one IP, use a sticky port from 10000-20000. In async mode the requests_per_second parameter limits how many pages load at the same time; the default is 2.

Step 7. Check the output

Run the example: the loader opens https://api.ipify.org and prints the page content, which is the IP the website sees. It should be the proxy's address, not yours; in the async example on port 823 the three lines show different addresses. A 407 error means a wrong username or password, or the plan has run out of traffic. The My IP page in your browser won't help here: the browser goes direct.

Country-specific loaders with a session per source

When the knowledge base needs local versions of pages (prices, stock, local news), create a separate WebBaseLoader for each country. The country goes into the username, the password comes from an environment variable, and both are passed through urllib.parse.quote(..., safe="") so that the ; and special characters don't break the proxy URL. requests decodes them back before sending.

import os
from urllib.parse import quote
from langchain_community.document_loaders import WebBaseLoader

USER = "USERNAME"
PASSWORD = os.environ["PROKSO_PASSWORD"]  # keep the password out of the code


def proxy_url(country, session=None, port=823):
    login = f"{USER}__cr.{country}"
    if session:
        login += f";sessid.{session}"  # one IP for about 30 minutes
    return f"http://{quote(login, safe='')}:{quote(PASSWORD, safe='')}@gw.prokso.com:{port}"


SOURCES = {
    "de": ["https://example.de/catalog?page=1", "https://example.de/catalog?page=2"],
    "pl": ["https://example.pl/catalog?page=1", "https://example.pl/catalog?page=2"],
}

docs = []
for country, urls in SOURCES.items():
    proxy = proxy_url(country, session=f"src{country}")
    loader = WebBaseLoader(
        web_paths=urls,
        proxies={"http": proxy, "https": proxy},
        requests_kwargs={"timeout": 30},
        raise_for_status=True,
    )
    for doc in loader.lazy_load():  # one page at a time, not the whole list in memory
        doc.metadata["country"] = country
        docs.append(doc)

print(len(docs))

The session number in the username (;sessid.srcde) keeps one IP for about 30 minutes on port 823, so all pages of one source, such as catalog pagination, come from the same address. Each country gets its own session. lazy_load() returns documents one at a time: with a long list you can write each one to the vector store straight away, and the country field in the metadata helps filter answers later.

LangChain loader errors through a proxy and how to fix them

• A 403 or 429 page ends up in the knowledge base. By default WebBaseLoader doesn't check the response code, so an "Access denied" page is parsed like a normal document. Set raise_for_status=True: load() and lazy_load() then raise requests.HTTPError, and you can skip the page instead of indexing it.
• InvalidProxyURL: Please check proxy URL. It is malformed and could be missing the host. requests couldn't find the host in the proxy URL. Usually an unencoded @ or / in the password moved the split point. Build the URL with quote() as in the example above.
• aiohttp.ClientHttpProxyError with status 407 in alazy_load() or fetch_all(). The proxy rejected the username or password from HTTPS_PROXY. The async loader doesn't retry this error, it fails at once. Encoded characters are fine here too: aiohttp decodes them itself.
• aiohttp.ClientProxyConnectionError. aiohttp couldn't reach gw.prokso.com at all: check the port and that outgoing connections to it are allowed on your server. The loader makes 3 attempts with a growing pause before it gives up.
• One failed page stops the whole async batch. By default an error on one URL aborts loading. With continue_on_failure=True the loader logs a warning and returns an empty document for that page: filter out empty documents before indexing.

Pace and traffic in a LangChain pipeline

In async mode requests_per_second is not a rate per second: in the source it is the size of a semaphore, that is, how many pages load at the same time. On port 823 each of them goes out from its own IP, but the website's own limits still apply. Keep the value low for one website and raise it only for lists of different websites.

Traffic is counted in both directions. WebBaseLoader downloads only the HTML of the page, without images and scripts, so it costs less than a browser-based loader when the page doesn't need JavaScript. Remove duplicate URLs before loading and store the URLs you've already loaded so the next run skips them. Purchased gigabytes don't expire, so a pause between runs costs nothing. Collect only what the website's rules allow and stay within our acceptable use policy.

If the proxy does not work

• 429 Too Many Requests
• 403 Forbidden
• Proxy traffic runs out fast