c# - Opening several websites in WebBrowser -
fairly new c#.
i have written web scraper collect data website. impossible retrieve data html since data want displayed via javascript or something, need access rendered website using webbrowser. excludes using e.g. webclient class.
i want collect data 10 different pages on website on period of 1 hour, code beneath permits me 2 @ time. if start third program, first 1 stops. after googling tried fix adding
system.net.servicepointmanager.defaultconnectionlimit = 1000;
but did absolutely nothing.
i still in development, right running seperate windows form every page scraping.
here code: (i have added a-variable because site completes loading 4 times).
public partial class form1 : form { //global variables int = 0; public form1() { initializecomponent(); } private void button1_click(object sender, eventargs e) { runprogram(); } void runprogram() { system.net.servicepointmanager.defaultconnectionlimit = 1000; uri link1 = new uri("http://www.somesite.com/sdf4575gfn"); webbrowser wb = new webbrowser(); wb.allownavigation = true; wb.navigate(link1); wb.documentcompleted += new webbrowserdocumentcompletedeventhandler(wb_documentcompleted); } //website loaded private void wb_documentcompleted(object sender, webbrowserdocumentcompletedeventargs e) { webbrowser wb = sender webbrowser; = + 1; string content = ""; if (a == 4) { wb.document.execcommand("selectall", false, null); wb.document.execcommand("copy", false, null); content = clipboard.gettext(); //store file } } }
after each time data, dispose webbrowser , reuse each url. loop through of urls , you'll near-simultaneous screen scrape. also, what's wrong making more instances of webbrowser , staggering screens scrapes?
Comments
Post a Comment