Decent overview of general web crawling techniques

[dchuk]

Iron.io Blog: Webcrawling at Scale with Nokogiri and IronWorker (Part 1)

While it's specific to their product, the general concepts are still solid and it's a good read if you're curious about getting into (and optimizing) large scale web crawling.

 

[(O_o)]

bump for a good read

 

[datawrangler]

Very cool. I use Nokogiri all the time, first time I've heard of iron.io though. A good find, thanks.