ibm bluemix - web URL information to apache spark in web app -


i trying retrieve information epa our web app, needs utilize ibm bluemix , apache spark. information gathering epa this:

https://aqs.epa.gov/api , ftp://ftp.cdc.noaa.gov/datasets/ncep.reanalysis.dailyavgs/surface/

but not gathering historical data, want update data inserting new data every hour web app. hence concerning have few questions:

1) need open hdfs store data? or retrieve data url , store in dataframe? ibm bluemix said provide 5 gb of storage, how 1 utilize store historical data , store updated data per hour?

2) if going update data per hour inserting new data data storage / data frame, should still use spark streaming? if yes, how use spark streaming url data? lot of resources see online useful if 1 has hdfs / formal database.

what doing import urls through pandas:

url = "https://aqs.epa.gov/api/rawdata?user=sogun3@gmail.com&pw=baycrane57&format=json&param=44201&bdate=20110501&edate=20110501&state=37&county=063" import urllib2 content = urllib2.urlopen(url).read() print content

however, if use method, means spark needs running 24-7 ensure updated data utilized. how 1 configure spark run 24-7? or there better method process data , put them nicely in dataframe data accessed later?

also, in web app, can 1 still use ipython data processing? or ipython interacting data , understanding data experimentally?

thanks lot!

you have options ;-) if need read source epa data , process before use in web app, can use spark service etl (extract transform load) source data epa web site, manipulate or wrangle data shape , size want, , save storage service bluemix object storage. web app read data in format want directly object storage. however, if source epa data largely in format want use in web app, can create rdds directly web site , pull in data , when need it. these datasets small quick peek, don't think need worry spark pulling directly memory work on it; i.e. no need try store locally spark in bluemix service cluster. besides, there no hdfs provided spark service; mentioned earlier, use external storage service. re: "ibm bluemix said provide 5 gb of storage", intended storing personal , 3rd-party spark libraries , such.

re: "spark needs running 24-7". spark service runs 24x7. spark code running on service run long program run ;-)

ipython (or jupyter notebooks) intended repl web. so, yes, interactive. in case, can write spark code in ipython notebook , have run long necessary, pulling , processing epa data web app, storing in object storage. web app can pull data needs object storage. said in future apis provided spark service, @ point web app talk directly spark service; in meantime, can make work notebooks.


Comments

Post a Comment

Popular posts from this blog

get url and add instance to a model with prefilled foreign key :django admin -

Image
i have page in each contest shown in row. each row contains contest id , link other model names have foreign key contest. news model: class news(models.model): title = models.charfield(max_length=50, null=true) description = models.textfield() time = models.timefield( default= datetime.datetime.now().time()) contest = models.foreignkey(contest) i want use admin adding , changing these models. want when user clicks on 'news' has foreign key contest id=1, shows list of news value foreign key. use {% instance in instances %} <tr class="{% cycle 'row1' 'row2' %}"> <td>{{ instance.title }}</td> <td>{{ instance.starttime }}</td> <td>{{ instance.timespan }}</td> <td>{{ instance.date }}</td> <td><a href="/adm...
Read more

css - Make div keyboard-scrollable in jQuery Mobile? -

from can tell, although jquery-mobile-powered pages can contain divs overflow set scroll or auto, , these divs can scrolled one-screen bar or mouse wheel, cannot scrolled using arrow keys, page-up/page-down, or home/end. instead, official "page" div (with data-role="page") absorbs input. perhaps other divs can't acquire focus, i'm not sure. is there way around this? edit: jsfiddle of simple example: https://jsfiddle.net/qogz0shx/ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script> <script src="https://ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.js"> </script> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"/> <style> #outer { overflow:scroll; height: 50vh; width: 50vw; } #inner { height: 500vh; width: 500vw; } ...
Read more

android - Keyboard hides my half of edit-text and button below it even in scroll view -

i having trouble keyboard covers edit-text , half of button in scroll view. here's layout , please tell me how should resolve problem. many questions on topic none of them s working me. <?xml version="1.0" encoding="utf-8"?> <linearlayout xmlns:android="http://schemas.android.com/apk/res/android" android:layout_width="match_parent" android:background="@drawable/bg" android:layout_height="match_parent"> <scrollview android:layout_width="match_parent" android:fillviewport="true" android:layout_height="match_parent"> <relativelayout android:layout_width="match_parent" android:layout_height="wrap_content" android:orientation="vertical"> <linearlayout android:layout_width="match_parent" android:layout_height="w...
Read more