c# - Opening several websites in WebBrowser -


fairly new c#.

i have written web scraper collect data website. impossible retrieve data html since data want displayed via javascript or something, need access rendered website using webbrowser. excludes using e.g. webclient class.

i want collect data 10 different pages on website on period of 1 hour, code beneath permits me 2 @ time. if start third program, first 1 stops. after googling tried fix adding

system.net.servicepointmanager.defaultconnectionlimit = 1000; 

but did absolutely nothing.

i still in development, right running seperate windows form every page scraping.

here code: (i have added a-variable because site completes loading 4 times).

public partial class form1 : form {     //global variables     int = 0;      public form1()     {         initializecomponent();     }      private void button1_click(object sender, eventargs e)     {         runprogram();     }      void runprogram()     {         system.net.servicepointmanager.defaultconnectionlimit = 1000;          uri link1 = new uri("http://www.somesite.com/sdf4575gfn");          webbrowser wb = new webbrowser();         wb.allownavigation = true;         wb.navigate(link1);          wb.documentcompleted += new webbrowserdocumentcompletedeventhandler(wb_documentcompleted);     }      //website loaded     private void wb_documentcompleted(object sender, webbrowserdocumentcompletedeventargs e)     {         webbrowser wb = sender webbrowser;          = + 1;          string content = "";          if (a == 4)         {             wb.document.execcommand("selectall", false, null);             wb.document.execcommand("copy", false, null);             content = clipboard.gettext();             //store file         }     } } 

after each time data, dispose webbrowser , reuse each url. loop through of urls , you'll near-simultaneous screen scrape. also, what's wrong making more instances of webbrowser , staggering screens scrapes?


Comments

Popular posts from this blog

get url and add instance to a model with prefilled foreign key :django admin -

css - Make div keyboard-scrollable in jQuery Mobile? -

ruby on rails - Seeing duplicate requests handled with Unicorn -