r - How to submit login form in Rvest package w/o button argument -


i trying scrape web page requires authentication using html_session() & html_form() rvest package. found e.g. provided hadley wickham, not able customize case.

united <- html_session("http://www.united.com/") account <- united %>% follow_link("account") login <- account %>%          html_nodes("form") %>%          extract2(1) %>%          html_form() %>%          set_values(                 `ctl00$contentinfo$signin$onepass$txtfield` = "gy797363",                 `ctl00$contentinfo$signin$password$txtpassword` = password) account <- account %>%  submit_form(login, "ctl00$contentinfo$signinsecure")

in case, can't find values set in form, hence trying give user , pass directly: set_values("email","password")

i don't know how refer submit button, tried: submit_form(account,login)

the error got submit_form function is: error in names(submits)[[1]] : subscript out of bounds

any idea on how go appreciated. thank you

currently, issue same open issue #159 in rvest package, causes issues not fields in form have type value. buy may fixed in future release.

however, can work around issue monkey patching underlying function rvest:::submit_request.

the core problem helper function is_submit. initially, it's defined this:

is_submit <- function(x) tolower(x$type) %in% c("submit",          "image", "button")

as logical is, however, fails in 2 scenarios:

  1. there no type element.
  2. the type element null.

both of these happen occur on united login form. can resolve adding 2 checks inside function.

custom.submit_request <- function (form, submit = null)  {   is_submit <- function(x) {     if (!exists("type", x) | is.null(x$type)){       return(f);     }     tolower(x$type) %in% c("submit", "image", "button")   }    submits <- filter(is_submit, form$fields)   if (length(submits) == 0) {     stop("could not find possible submission target.", call. = false)   }   if (is.null(submit)) {     submit <- names(submits)[[1]]     message("submitting '", submit, "'")   }   if (!(submit %in% names(submits))) {     stop("unknown submission name '", submit, "'.\n", "possible values: ",           paste0(names(submits), collapse = ", "), call. = false)   }   other_submits <- setdiff(names(submits), submit)   method <- form$method   if (!(method %in% c("post", "get"))) {     warning("invalid method (", method, "), defaulting get",              call. = false)     method <- "get"   }   url <- form$url   fields <- form$fields   fields <- filter(function(x) length(x$value) > 0, fields)   fields <- fields[setdiff(names(fields), other_submits)]   values <- pluck(fields, "value")   names(values) <- names(fields)   list(method = method, encode = form$enctype, url = url, values = values) }

to monkey patch, need use r.utils package (install via install.packages("r.utils") if don't have it).

library(r.utils)  reassigninpackage("submit_request", "rvest", custom.submit_request)

from there, can issue our own request. 

account <- account %>%       submit_form(login, "ctl00$contentinfo$signinsecure")

and works!

(well, "works" misnomer. due united employing more aggressive authentication requirements -- including known browsers -- results in 301 unauthorized. however, fixes error).

a full reproducible example involved couple of other minor code changes:

library(magrittr) library(rvest)  url <- "https://www.united.com/web/en-us/apps/account/account.aspx" account <- html_session(url) login <- account %>%   html_nodes("form") %>%   extract2(1) %>%   html_form() %>%   set_values(     `ctl00$contentinfo$signin$onepass$txtfield` = "user",     `ctl00$contentinfo$signin$password$txtpassword` = "pass") account <- account %>%    submit_form(login, "ctl00$contentinfo$signinsecure")

Comments

Post a Comment

Popular posts from this blog

get url and add instance to a model with prefilled foreign key :django admin -

Image
i have page in each contest shown in row. each row contains contest id , link other model names have foreign key contest. news model: class news(models.model): title = models.charfield(max_length=50, null=true) description = models.textfield() time = models.timefield( default= datetime.datetime.now().time()) contest = models.foreignkey(contest) i want use admin adding , changing these models. want when user clicks on 'news' has foreign key contest id=1, shows list of news value foreign key. use {% instance in instances %} <tr class="{% cycle 'row1' 'row2' %}"> <td>{{ instance.title }}</td> <td>{{ instance.starttime }}</td> <td>{{ instance.timespan }}</td> <td>{{ instance.date }}</td> <td><a href="/adm...
Read more

android - Keyboard hides my half of edit-text and button below it even in scroll view -

i having trouble keyboard covers edit-text , half of button in scroll view. here's layout , please tell me how should resolve problem. many questions on topic none of them s working me. <?xml version="1.0" encoding="utf-8"?> <linearlayout xmlns:android="http://schemas.android.com/apk/res/android" android:layout_width="match_parent" android:background="@drawable/bg" android:layout_height="match_parent"> <scrollview android:layout_width="match_parent" android:fillviewport="true" android:layout_height="match_parent"> <relativelayout android:layout_width="match_parent" android:layout_height="wrap_content" android:orientation="vertical"> <linearlayout android:layout_width="match_parent" android:layout_height="w...
Read more

css - Make div keyboard-scrollable in jQuery Mobile? -

from can tell, although jquery-mobile-powered pages can contain divs overflow set scroll or auto, , these divs can scrolled one-screen bar or mouse wheel, cannot scrolled using arrow keys, page-up/page-down, or home/end. instead, official "page" div (with data-role="page") absorbs input. perhaps other divs can't acquire focus, i'm not sure. is there way around this? edit: jsfiddle of simple example: https://jsfiddle.net/qogz0shx/ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script> <script src="https://ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.js"> </script> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"/> <style> #outer { overflow:scroll; height: 50vh; width: 50vw; } #inner { height: 500vh; width: 500vw; } ...
Read more