Google Scraper - Rate Limit

[theobserver]

Hi Guys

Any ideas of Safe numbers for Search queries on google.

Im building a google scraper, plan is to get about 10 proxies and cycle through them.

Any ideas of a safe number of queries per IP per minute?

Or any tips for preventing the IP's getting blocked.

Ill be cycling the IP and have a different UserAgent for each IP etc.

Cheers
TO

 

[mattseh]

Pick a number.

If the proxy gets a captcha, don't use it for a day.

In a week, come back to it, and adjust the number (or make the number autoadjust).

1 query / minute / proxy may be safe depending on the query type.

 

[theobserver]

Cheers, It will be a different query everytime (Which should make our life easier as it wont look automated) . Ill be querying google shopping.

Ill have a play around with it.

Cheers
TO

 

[acamso]

I would just trial and error it to find the number. Best of luck

 

[itspj]

May be use more proxies and set up a timeout for each of them.

 

[flashpoint]

Hmm. Things have probably changed since the last time I did something like this but way back when it was the volume of searches in a given time frame that got me captchad.

That is, the number of searches didn't seem to matter so much as the speed that I submitted them.

How to: Scrape search engines without pissing them off | Search News Central

Old info but maybe a jumping off point.