Proxies for Collecting Google Search Results

How this differs from rank tracking

If you need to know where your own site ranks, that is the SEO and search rankings use case. And for your own site's rankings, start with Google Search Console: the data there comes from Google itself, with no queries to the SERP.

This page covers a different task. Research firms and SEO agencies collect SERP data at scale: thousands of queries across several countries, cities and languages. They care about the whole page: who ranks at the top in a niche, which sites show up most often, where ads, shopping results and maps appear.

What the results depend on

Google tailors results to the visitor. The SERP is shaped by:

• country - the same query puts different sites at the top in the US and in Canada;
• city - especially for queries like "dentist" or "flower delivery";
• language - the hl parameter sets the interface language, and gl sets the results country;
• device - mobile and desktop SERPs differ in which blocks they include.

The hl and gl parameters help, but they do not replace location. Google draws local results partly from the visitor's IP. So for data on Germany, use a German IP; for Poland, a Polish IP. The device type is determined by the browser and its headers, not by the proxy: for mobile SERPs, use a mobile browser profile.

What to collect from the SERP

A typical SERP has long since stopped being ten blue links. For research, people usually save:

• organic results and their order;
• the local pack with the map and businesses - it depends on the city more than anything else;
• ads at the top and bottom of the page;
• shopping blocks with prices and stores;
• blocks with questions, videos and news.

For the local pack, turn on advanced mode in the generator: there you can specify a region (a state for the US), any city by name, a ZIP or a carrier (ASN). Traffic with precise geo is counted double, so turn it on only for queries with local intent. For general country-level queries, choosing the country is enough.

Save the collection conditions along with the results: the query, country, city, language, device type, date and time. Without them, results from Toronto and Vancouver collected on different days cannot be compared fairly later. Repeat the collection over the same list of queries so you see real changes rather than random noise.

A new IP for every request

For large-scale collection, the "Rotating (new IP per request)" mode works well: port 823 (HTTP) or 824 (SOCKS5) gives a new address for every request. Your script works through a single connection address, and the IP changes on our side. A code example is in the Python requests guide.

A plan allows up to 1000 concurrent connections. That is a ceiling, not a recommendation: start with a few threads and watch how the share of responses with a check changes.

Why Google shows a check

Google watches how closely a stream of queries resembles human behavior. When there are too many queries and they come too often, Google shows an unusual traffic page and a CAPTCHA. What that page means and what to do about it is covered in the breakdown of the Google "unusual traffic" error.

We do not promise there will be no CAPTCHA, and we do not give advice on getting around it. Calm measures are what work:

1. Slow down. Pause between queries and limit the number of threads.
2. Spread out the collection. Distribute queries across the day instead of sending everything within an hour.
3. Cache. Results for most queries do not change within minutes - do not request them again.
4. Collect only what you need. One SERP page instead of five often gives you enough data.

More on working calmly within limits is on the 429 error page.

Which proxies to choose

For Google SERPs, people usually go with residential proxies at $1.59 per GB. These are addresses from home internet providers, and Google sees them as regular users. Turn on the anonymity filter in the pool settings: it filters out IPs that sites have already flagged as proxies or VPNs.

Datacenter proxies at $0.80 per GB get checks on Google Search more often, but they are fine for small volumes. Mobile proxies at $2.50 per GB come in handy if you are studying mobile SERPs and want a 4G/5G carrier address.

What to know in advance

• A proxy does not override Google's terms of service: the pace and volume of collection are your responsibility.
• The pool is shared: another customer may have used an address before, so some IPs may get a check right away.
• We do not promise that Google will accept a particular address: that decision is Google's own.

How much traffic you need

A SERP page without images is light, but thousands of queries a day add up to a noticeable volume. Turn off loading of images and scripts if you only need the HTML. Traffic is counted in both directions. The traffic calculator helps you estimate the volume; the gigabytes you buy never expire, and prices are on the pricing page.