Scraping Google Instant Results - Plus Tits

[vcm]

Since google implemented instant, all of my tools that use curl to scrape the results are broken.

Is anyone else having the same issues?

Google's serps now appear to be all javascript - there isn't any html output with the results.

Any ideas how to get around this would be appreciated.

Thanks

 

[jamesvick]

The last one is OOOMPHHH!!!!

Anyways, there does not exist anything like no HTML output. If you can browse it in a browser, it has to have HTML output. Iam not familiar with CURL but surely there will be a way to read html output.

Iam a .NET coder so i use web posts, http requests, watin and web browser control to code. Read about this google scraper i posted on BHT :

Black Hat Seo & Affiliate Marketing Forum

Just try to find if there is any way by which you can download the page in memory and then parse it looking for tags the way i did.

 

[dchuk]

Are you sure they didn't just change up the dom/css so your scrapers can't find anything anymore? I haven't scraped google with my own scripts in a while so I can't confirm, though I'm going to whip up a cool experiment soon and will know in a bit.

 

[davidne]

You can definitely still scrape results even with it on...and if you don't like it why not just turn it off?

 

[motivated]

Amazing tits. Live results is a cliche AJAX solution which modifies the HTML DOM in real-time. If you want to scrape live results, use WatiN / WatiR. They enable you to access the DOM directly. Disclaimer: WatiN is pretty fricking slow, so there's a trade between performance and flexibility.

All this being said, why don't you turn of live results?

 

[dchuk]

All this being said, why don't you turn of live results?

this. and watir/celerity is my favorite fucking thing ever, so much fun

 

[acidie]

Or you could do this. But either way.

 

[yosaito]

I wrote a scraper in python a while back. Checked it today and still works great. Message me if youre stuck and will look back into the script to findout how im doing it.