Building bots in VB.NET..httpwebrequest or webbrowser?

simpleonline12

simpleonline12

New member
Sep 29, 2009
191
3
0
What's a better route to learn when starting out building bots in VB.NET?

I can currently build basic bots that can navigate to a website, enter text into textboxes, and press buttons, etc but it seems somewhat slow using the webbrowser.

I'm looking to build some professional type bots that can use multi threads and move lightening fast when processing the data.

I'm at a cross roads right now because I can dive into either learning everything about webbrowser options or httpwebrequest options.

Of course I will be learning them both in time but which would be the better route to learn to use when building great little bots?

Thanks
 


simpleonline12 said:
What's a better route to learn when starting out building bots in VB.NET?

I can currently build basic bots that can navigate to a website, enter text into textboxes, and press buttons, etc but it seems somewhat slow using the webbrowser.

I'm looking to build some professional type bots that can use multi threads and move lightening fast when processing the data.

I'm at a cross roads right now because I can dive into either learning everything about webbrowser options or httpwebrequest options.

Of course I will be learning them both in time but which would be the better route to learn to use when building great little bots?

Thanks
Click to expand...

httprequests are the best way to follow. The biggest reason is the problem of working with proxies which you might need to implement eventually. You see web browsers use proxy information from registry and if you multi thread it all will use the same proxy. But in httpwebrequests you can go for individual proxies for each thread. Also webbrowsers are com objects of IE, hence version dependent. So basically its as useless as the browser. There are many other reasons to go with httprequests rather than web-browser control.
 
The WebBrowser control that ships with .net is horrible man. for eg. bad socks support, memory leaks, cache is system based, closed source.. the list can go on man.

I do realize that you might have to create some apps where you need some sort of browser control to give the user some flexibility. Or when you have to process js / closed ajax calls that you can't simulate using direct socket connections (webrequest etc).

My recommendation would be to create your own Browser control first and don't use an ie engine. Go with xulrunner/ geckoFX or something similar. GeckoFx is open-source so you can optimize & extend the build to your preferences.

And finally, there is no one method to creating a good scraper/bot. You have to just use the right tools for the job. Most of the time, you can get away with just using webRequest though.

So invest some time building some base classes and compiling your own browser control and libraries that you can use in all your scraping projects. It will make your life much simpler. :)

All the best man. I've been doing this for the past 10 years and the only constant is change. :)

Cheers,
Ash




simpleonline12 said:
What's a better route to learn when starting out building bots in VB.NET?

I can currently build basic bots that can navigate to a website, enter text into textboxes, and press buttons, etc but it seems somewhat slow using the webbrowser.

I'm looking to build some professional type bots that can use multi threads and move lightening fast when processing the data.

I'm at a cross roads right now because I can dive into either learning everything about webbrowser options or httpwebrequest options.

Of course I will be learning them both in time but which would be the better route to learn to use when building great little bots?

Thanks
Click to expand...
 
You must log in or register to reply here.
Top Bottom
Back