Content Scraping Platforms

[Blackstack]

I'm aware of (but not interested) in individual scripts for scraping particular pages. I've been able to do those myself without *too* much hair-pulling.

However, I see that there are some complete full-service scraping/crawling packages that are tailored more for business use that may be of service to me. One example that might give you an idea of the capabilities of these packages is Automation Anywhere
(not meant as an endorsement in any way)

I'm interested in any opinions and guidance as to the benefits...especially from those that *could* write their own scrapers if need be. While these packages may deliver a service that you could do yourself, have you found any commercial scraping platforms who's ease of use or time savings justify the price?

-Blackstack

 

[ArtRage]

I believe most cases needs custom scripting. You need to define rules of content extraction in any case.

I've written scraper framework to automate data mining, but it still needs
url patterns & datamining regexp'es set up for each particular site/engine.

 

[ctp5182]

There's also a package called Web Scraper + which dumbs down the scraping process. I think SpyFu uses Web Scraper +. I have a copy of it but have only tried setting up some test scrapes with it. The nice thing it allows is that once you have a template set up for a particular scrape, you can apply it over and over. It comes with some pre-made templates (like an adwords campaign scrape). I think they have a demo version you can download. I can see the benefits of a program like this. I just haven't had the time to get into it very deep. I have other projects at the moment.