Create Multiple Background workers with HttpWebRequest

simpleonline12

simpleonline12

New member
Sep 29, 2009
191
3
0
Okay so I've dug around all day and found that using multiple "background workers" would do what I would need for my application.

How can I create 4 background workers and then feed the urls to the webrequest from a listbox?

Code: 
Dim request As HttpWebRequest = HttpWebRequest.Create(URLGOESERE)                 
Dim response As HttpWebResponse                 
response = CType(request.GetResponse, HttpWebResponse)                 
Dim webstream As Stream = response.GetResponseStream                 
Dim streamreader As New StreamReader(webstream, Encoding.UTF8)                 
Dim html As String = streamreader.ReadToEnd                 
Messagebox.show(html.tostring)
I'm basically needing to have each webrequest access the listbox, grab a URL, process the webrequest and then go grab the next url on file.

I am trying to have that running 4x at a time until the sites are all looped through.

I've read a background worker is good vs. multi threading because it's easier to code.
 


Hey, you're heading in the wrong direction but first things first. Background workers are threads too, so the creation of multiple background workers would be multithreading as well! What's the advantage of background workers? Honestly, I never knew why they exist...Pointless to me!

However, you can't access the listbox from a thread because this would cause an exception. Only the main thread is allowed to access visual components. If you still want to access the listbox via threads, you should use events and invoke the procedure.

I'm well known with this stuff because it's my everyday business but I'm not familiar with VB. If you need help or something else, hit me up on skype!
 
Hi,

I totally agree with Khalil.

Khalil said:
Hey, you're heading in the wrong direction but first things first. Background workers are threads too, so the creation of multiple background workers would be multithreading as well! What's the advantage of background workers? Honestly, I never knew why they exist...Pointless to me!

However, you can't access the listbox from a thread because this would cause an exception. Only the main thread is allowed to access visual components. If you still want to access the listbox via threads, you should use events and invoke the procedure.

I'm well known with this stuff because it's my everyday business but I'm not familiar with VB. If you need help or something else, hit me up on skype!
Click to expand...
 
Nathan I've sampled extensively from your code. Many thanks for SimpleBrowser and XBrowser!
 
Hey no probs! XBrowser is kind of dead (wasn't really ever alive, to be fair), but SimpleBrowser seems to be getting a bit of use. Have had someone else doing most of the development of late, and doing a good job of it. He is using it for Selenium testing and has written a driver to integrate the two.
 
Honestly, I don't get the point of 3rd party lib's & components when it comes to content scraping. Besides that, threading as it, was never easier before!
 
Khalil said:
What's the advantage of background workers? Honestly, I never knew why they exist...Pointless to me!
Click to expand...

Because they are terminated when you exit the program, ordinary threads ere not, so they may be left hanging in the memory.
Alternatively you can do something like (in C#):

Code: 
Thread t = new Thread(function);
t.IsBackground = true;
t.start();

(Not the correct syntax, i know, too lazy to look up now.)
As for OP, just loop through items in listbox and spawn a thread for each (unless you have lots of items, then you should use ThreadPool instead)..
You might want to look up "VB parameterized thread start" in Google first though.
 
Khalil said:
However, you can't access the listbox from a thread because this would cause an exception. Only the main thread is allowed to access visual components. If you still want to access the listbox via threads, you should use events and invoke the procedure.
Click to expand...

Delegates, man, ever heard about them? Be careful using them though, they can mess your shit up if used improperly.
 
If you're using WinForms, run your processing in a background thread (or BackgroundWorker if you want, which really is just wrapping a thread anyway), passing messages/updates/data/whatever into a System.Collections.Concurrent.ConcurrentQueue<T>, then use a timer (System.Windows.Forms.Timer) to poll the queue for updates and update your UI that way.
 
NathanRidley said:
passing messages/updates/data/whatever into a System.Collections.Concurrent.ConcurrentQueue<T>, then use a timer (System.Windows.Forms.Timer)
Click to expand...

Oh, I don't think I've used that, I'll have to try this :)

BTW, that SimpleBrowser of yours is awesome! Tried it a couple of months ago, there's so much to learn for it. Though I believe it was missing something, can't quite remember. I think it was proxy support, not sure though, maybe something else, so I ended up using GeckoFX instead in the final project..
 
Ah cool, yeah GeckoFX is good, though a bit out of date these days. If you want a headless browser that renders, check out Awesomium. SimpleBrowser proxy support should be there in full though. If it's missing anything, log an issue, they usually don't take too long to get addressed these days.
 
Nathan, just wanted to say thank you for SimpleBrowser! It's super easy to work with and it's going to save me soooo much time going forward. Much appreciated!
 
You must log in or register to reply here.
Top Bottom
Back