Scraping google.com

March 5, 2007

I don’t know when this happened, but it seems you can’t scrape (parse) google.com as easily as before. At least with Python, a simple

s=urllib.urlopen('http://www.google.com/search?q=define%3Aesoteric')
r=s.read()
print r

will give you this notice instead:

/snip/ Your client does not have permission to get URL /search?q=define%3Aesoteric from this server /snip/

And here’s me not understanding why my regexes are not matching. :) Apparently, you should at least show some kind of User Agent. In Python you can do this easily by subclassing URLopener and setting it’s version property to something, like:

class MyOpener(urllib.URLopener):
  version = "InternetExploiter/666"
urllib._urlopener = MyOpener()

And I’m really starting to love TextMate. Just type your little Python script and hit Cmd-R. Brilliant.