apache spark - Map slices of RDD/Dataframe based on column value in PySpark -

-

i have dataframe 1 below:

timestamp   key  value 2016-06-29     88 2016-06-28     89 2016-06-27     90 2016-06-29   b   78 2016-06-28   b   79 2016-06-27   b   80 2016-06-29   c   98 2016-06-27   c   99

the goal convert rdd of panda.series in efficient way. want result:

(a, pandas.series) (b, pandas.series) (c, pandas.series)

so want operate on slice of dataframe has same key , give output tuple of (key, pandas.series) every slice.

things tried/ thoughts:

  1. spark-ts seems ideal use, seems python version broken.
  2. tried window function, it's not suitable such cases.
  3. instead of read them in bulk, read based on key convert panda.series , repeat. slow - not viable.

any ideas/suggestions achieve in fast , efficient manner?


Comments

Post a Comment

Popular posts from this blog

Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12:test (default-test) on project.Error occurred in starting fork -

-
i searched on internet provided answers did not in building project. the error starting fork. tried following methods: mvn clean install -u when searched, people said because of java 7 , should work java 6. using java 6 only. some said set surefire plugin this: <groupid>org.apache.maven.plugins</groupid> <artifactid>maven-surefire-plugin</artifactid> <version>2.5</version> <configuration> <skiptests>false</skiptests> <testfailureignore>true</testfailureignore> <forkmode>once</forkmode> </configuration> the actual error is: [error] failed execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12:test (default-test) on project junitcategorizer.instrument: error occurred in starting fork, check output in log -> [help 1] org.apache.maven.lifecycle.lifecycleexecutionexception: failed execute goal org.apache.maven.plugins:maven-surefire-plugin:2...
Read more

android - CoordinatorLayout, FAB and container layout conflict -

-
Image
i have main activity drawer have container layout replace each fragment fragmentmanager . i want add fab 1 of child fragments hide/show on scroll im not sure im doing wrong , get: layout: <?xml version="1.0" encoding="utf-8"?> <android.support.design.widget.coordinatorlayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:app="http://schemas.android.com/apk/res-auto" android:layout_width="match_parent" android:layout_height="match_parent"> <android.support.design.widget.appbarlayout android:id="@+id/appbar" android:layout_width="match_parent" android:layout_height="wrap_content" android:fitssystemwindows="true" android:theme="@style/themeoverlay.appcompat.dark.actionbar"> <android.support.v7.widget.toolbar android:id="@+id/toolbar" android:layout_width="match_parent...
Read more