Python 2.
This is a
google wonderwheel scraper, simply add the starting keyword as shown, and select the amount of levels deep you want to go.
Each level will be a list within the items list.
And before I get bitched at for sucky coding, I know. I'm a newb at Python and if you don't like it, then don't use it, although if anybody would like to make a suggestion on something I could have done better, then go ahead
Code:
import urllib
import urllib2
from lxml import objectify
def grab_keywords(items,level = 2):
if level <= 0:
return items
#Append a empty list to the end
items.append([])
#Loop through last got items (but not the empty list just appended) - This avoids doing allready processed keywords
for i in xrange(len(items[-2])):
try:
kw = urllib.quote_plus(items[-2][i])
source = urllib2.urlopen('http://google.com/complete/search?output=toolbar&q=%s' % kw).read()
root = objectify.fromstring(source)
except Exception as excp:
#If this occurs, Google has banned you, so just return what we have
print excp
return items
try:
for e in root.CompleteSuggestion:
#Add to the new list
if kw != e.suggestion.attrib['data']:
items[-1].append(e.suggestion.attrib['data'])
except:
#If this happens, current keyword has no suggestions
pass
return grab_keywords(items, level - 1)
items = []
#Add your keyword here
items.append(['cheese'])
#Amount of levels deep to go can be passed, defaults to 2
grab_keywords(items,level=3)
print items
PS: If you need proxy support, use mattseh's Proxy Manager, but to be honest, you can use this a
shitload without being banned.
