simple simple CURL help

Status
Not open for further replies.
psychoul

psychoul

New member
May 22, 2007
90
2
0
Hello!

The situation:
- I access a page through my browser at home and everything is ok.
- I try to access the page using CURL (from my VPS server) but it redirects me to the registars parking page (I am sure the CURL setup is correct,used it before). A logical explanation is that DNS hasnt yet propagated(??).
- Now for the strange part :D I login to the server ssh and I use lynx to connect to the page and connects just fine!!!

How come the lynx connects correctly to the page while CURL is redirected to the parked page???

It's a bit confusing for me.

Thank you for reading.
 


Try accesing the site using Firefox and LiveHTTPHeaders and see if any cookies are set and try curl with those.
 
But I am telling you...

When I access it with Firefox I get 200 header and the page displays ok...

When I access it with CURL I get a 302 redirect to the parking page!

This is what is strange...

UPDATE : The CURL now works ok. But the question remains the same:
Why CURL was accessing a different DNS (redirecting to parking) while lynx from the same server was accessing the page correctly?
 
Well... I don't know why it used a different DNS.

Some of the sites I scrape use some sort of redirect so a simple CURL- command will get redirected to the front-page. Pain in the arse, but work-around-able. I thought it was something like that you experienced.
 
If they have a script of some sort that handles incoming http requests, and handles bots differently maybe? People get parked landing pages with ads, bots get something else? Dunno, just guessing.

If it were DNS, Lynx would not have worked on your server. The only difference between Curl and Lynx from the same box is the user agent.
 
Supergeek said:
If they have a script of some sort that handles incoming http requests, and handles bots differently maybe? People get parked landing pages with ads, bots get something else? Dunno, just guessing.

If it were DNS, Lynx would not have worked on your server. The only difference between Curl and Lynx from the same box is the user agent.
Click to expand...

Hmm yes that seems logical. :thumbsup:
 
Status
Not open for further replies.
Top Bottom
Back