Program to rip/scrape entire blog into txt, database, or pdf?

[ImagesAndWords]

I have looked around the web for a program that will download each and every blog post (not just the most recent posts) off a given blog - and save it all into a file format.

Yes, I know rss readers "download" blog posts, but the ones I have tested only gets a certain amount of posts (FeedDemon says it can get 200 posts but always only gets max 20, and not older posts).

I have found a couple of great blogs in a niche I'm going after and I wanna download all their posts instead of sifting through the categories and archives - especially since there's soo many older posts. If I could just rip all the posts into some exportable file, it would save tons of time.

Does anyone know of such an app?

Thanks!

 

[zany zoroaster]

maybe look into wget. its a command line utility that you can use to recursively download an entire site to your hard drive. You can tell it to only download certain fily types (html), and there are many more options that could help you get at just the posts. Then you can probably convert all these files using (maybe) ghostscript, or some html2pdf utility, or even have a browser display the page and print it to a pdf driver. This will all take a long while, and I believe there are paid services that already provide this service, although many of them are focused on creating physical books out of blogs.

 

[stmadeveloper]

If you feel like learning a little you can do this with mechanize and hpricot pretty easily. Have to learn a little ruby in the process but I'm sure it will do you good

 

[ImagesAndWords]

Thanks guys. Maybe I will try the wget first. I've used it before, so I can play around with some of the options to see if I can pull the posts.

 

[Deliguy]

Fairly easy. Use this script to grab all the pages on a blog and append them to a txt file.
http://www.bluehatseo.com/wp-content/uploads/2006/11/crawlercgi.txt

Then look in their template for a starting content code and ending content code (something in their HTML that indicates the beginning of the actual post content and ending post content).

Then lwp/wget all that content and have it write to txt file.

 

[seoname]

web scraper plus is a pretty good program to do this, takes some setting up, but after you do it will crawl the entire site pulling only the data you want.