Curl Proxies and Dreamhost

fatmoocow

fatmoocow

Señor Member
Feb 1, 2009
480
9
0
Anyone been able to run a proxy through curl with Dreamhost? As soon as I add "curl_setopt($ch, CURLOPT_PROXY, $proxy);" everything fails.

Recommended hosts where I can scrape without running into this shit?
 


run curl on your own machine -> localhost.

Most scraping techniques take long and lots (of RAM, CPU time, etc) ... so localhost is often the way to go.

::emp::
 
emp said:
run curl on your own machine -> localhost.

Most scraping techniques take long and lots (of RAM, CPU time, etc) ... so localhost is often the way to go.

::emp::
Click to expand...

This is probably good advice. I started to dump some stuff on an old laptop but I've got to rebuild as the appache install is from like 1995. Long term it will have to go on a VPS but I can probably develop it from here.
 
I always scrape local. There is no reason not to.

a) Any old machine can handle curl
b) I got the data where I need it
c) Scraping is really only limited by bandwidth or restrictions an the site you are hitting.
d) Scraping is almost never time-sensitive. So what if it takes 2 hours instead of 1?

::emp::
 
After a few hiccups, I'm raping away at google as we speak. Thanks guys, sometimes you just need to be told what you already know. Time to expand on that proxy checker.
 
If your using proxies and bandwidth isn't to big an issue definitely go with what emp says and do it local.
 
You must log in or register to reply here.
Top Bottom
Back