scala - filter spark dataframe with row field that is an array of strings -


using spark 1.5 , scala 2.10.6

i'm trying filter dataframe via field "tags" array of strings. looking rows have tag 'private'.

val report = df.select("*")   .where(df("tags").contains("private"))

getting:

exception in thread "main" org.apache.spark.sql.analysisexception: cannot resolve 'contains(tags, private)' due data type mismatch: argument 1 requires string type, however, 'tags' of array type.;

is filter method better suited?

updated:

the data coming cassandra adapter minimal example shows i'm trying , gets above error is:

  def testdata (sc: sparkcontext): dataframe = {     val stringrdd = sc.parallelize(seq("""       { "name": "ed",         "tags": ["red", "private"]       }""",       """{ "name": "fred",         "tags": ["public", "blue"]       }""")     )     val sqlcontext = new org.apache.spark.sql.sqlcontext(sc)     import sqlcontext.implicits._     sqlcontext.read.json(stringrdd)   }   def run(sc: sparkcontext) {     val df1 = testdata(sc)     df1.show()     val report = df1.select("*")       .where(df1("tags").contains("private"))     report.show()   }

updated: tags array can length , 'private' tag can in position

updated: 1 solution works: udf

val filterpriv = udf {(tags: mutable.wrappedarray[string]) => tags.contains("private")} val report = df1.filter(filterpriv(df1("tags")))

i think if use where(array_contains(...)) work. here's result:

scala> import org.apache.spark.sparkcontext import org.apache.spark.sparkcontext  scala> import org.apache.spark.sql.dataframe import org.apache.spark.sql.dataframe  scala> def testdata (sc: sparkcontext): dataframe = {      |     val stringrdd = sc.parallelize(seq      |      ("""{ "name": "ned", "tags": ["blue", "big", "private"] }""",      |       """{ "name": "albert", "tags": ["private", "lumpy"] }""",      |       """{ "name": "zed", "tags": ["big", "private", "square"] }""",      |       """{ "name": "jed", "tags": ["green", "small", "round"] }""",      |       """{ "name": "ed", "tags": ["red", "private"] }""",      |       """{ "name": "fred", "tags": ["public", "blue"] }"""))      |     val sqlcontext = new org.apache.spark.sql.sqlcontext(sc)      |     import sqlcontext.implicits._      |     sqlcontext.read.json(stringrdd)      |   } testdata: (sc: org.apache.spark.sparkcontext)org.apache.spark.sql.dataframe  scala>         | val df = testdata (sc) df: org.apache.spark.sql.dataframe = [name: string, tags: array<string>]  scala> val report = df.select ("*").where (array_contains (df("tags"), "private")) report: org.apache.spark.sql.dataframe = [name: string, tags: array<string>]  scala> report.show +------+--------------------+ |  name|                tags| +------+--------------------+ |   ned|[blue, big, private]| |albert|    [private, lumpy]| |   zed|[big, private, sq...| |    ed|      [red, private]| +------+--------------------+

note works if write where(array_contains(df("tags"), "private")), if write where(df("tags").array_contains("private")) (more directly analogous wrote originally) fails array_contains not member of org.apache.spark.sql.column. looking @ source code column, see there's stuff handle contains (constructing contains instance that) not array_contains. maybe that's oversight.


Comments

Post a Comment

Popular posts from this blog

get url and add instance to a model with prefilled foreign key :django admin -

Image
i have page in each contest shown in row. each row contains contest id , link other model names have foreign key contest. news model: class news(models.model): title = models.charfield(max_length=50, null=true) description = models.textfield() time = models.timefield( default= datetime.datetime.now().time()) contest = models.foreignkey(contest) i want use admin adding , changing these models. want when user clicks on 'news' has foreign key contest id=1, shows list of news value foreign key. use {% instance in instances %} <tr class="{% cycle 'row1' 'row2' %}"> <td>{{ instance.title }}</td> <td>{{ instance.starttime }}</td> <td>{{ instance.timespan }}</td> <td>{{ instance.date }}</td> <td><a href="/adm...
Read more

android - Keyboard hides my half of edit-text and button below it even in scroll view -

i having trouble keyboard covers edit-text , half of button in scroll view. here's layout , please tell me how should resolve problem. many questions on topic none of them s working me. <?xml version="1.0" encoding="utf-8"?> <linearlayout xmlns:android="http://schemas.android.com/apk/res/android" android:layout_width="match_parent" android:background="@drawable/bg" android:layout_height="match_parent"> <scrollview android:layout_width="match_parent" android:fillviewport="true" android:layout_height="match_parent"> <relativelayout android:layout_width="match_parent" android:layout_height="wrap_content" android:orientation="vertical"> <linearlayout android:layout_width="match_parent" android:layout_height="w...
Read more

css - Make div keyboard-scrollable in jQuery Mobile? -

from can tell, although jquery-mobile-powered pages can contain divs overflow set scroll or auto, , these divs can scrolled one-screen bar or mouse wheel, cannot scrolled using arrow keys, page-up/page-down, or home/end. instead, official "page" div (with data-role="page") absorbs input. perhaps other divs can't acquire focus, i'm not sure. is there way around this? edit: jsfiddle of simple example: https://jsfiddle.net/qogz0shx/ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script> <script src="https://ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.js"> </script> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"/> <style> #outer { overflow:scroll; height: 50vh; width: 50vw; } #inner { height: 500vh; width: 500vw; } ...
Read more