apache spark - What is right way of Broadcasting a big Dataframe -

-

i have need broadcast 1 big file (converted dataframe) in spark 1.6.1 lookup. whether below code right way -

val file = sc.broadcast(dataframe_df)

then accessing using below code in udf.

def abc{ file.value.sqlcontext.sql(query) .... }

i read sc.broadcast brings data driver first , sends executors not approach this. right? help.


Comments

Post a Comment

Popular posts from this blog

iis - ASP.Net Core CreatedAtAction in HttpPost action returns 201 but entire request ends with 500 -

-
i'm following[1] setting first .net core webapi (moving "old" webapi). when httppost action, in tutorial: [httppost] public iactionresult create([frombody] todoitem item) { if (item == null) { return badrequest(); } todoitems.add(item); return createdataction("gettodo", new { id = item.key }, item); } i httperror 500 after exiting controller. createdataction returns proper object status code 201 expect, , after method exits server somehow turns 500. don't seem able step rest of pipeline. trace gives me following error: 81. -general_request_entity 82. -notify_module_completion 83. -module_set_response_error_status [warning] 171ms modulename aspnetcoremodule notification execute_request_handler httpstatus 500 httpreason internal server error httpsubstatus 0 errorcode operation completed successfully. (0x0) configexcepti...
Read more

gcc - Neither ld wrap nor LD_PRELOAD working to intercept system call -

-
i trying use -wl,-wrap=sendto -wl,-wrap,sendto in final g++ link command links app replace standard sendto function own. i compile following source code gcc -c -o wrap.o wrap.c , include wrap.o in final g++ command links app (the rest of app c++ hence use of g++) #include <stdio.h> #include <sys/types.h> #include <sys/socket.h> ssize_t __real_sendto(int, const void *, size_t, int, const struct sockaddr *, socklen_t); ssize_t __wrap_sendto ( int sockfd, const void *buf, size_t len, int flags, const struct sockaddr *dest_addr, socklen_t addrlen ) { printf("my wrap sendto ...\n"); return __real_sendto(sockfd, buf, len, flags, dest_addr, addrlen); } when use sendto in own source code, wrapper in fact used ok 3rd party shared objects linked against in final g++ command use sendto still use system sendto i.e. not wrapper. how can sendto wrapper used throughout ? i have tried ld_preload approach sendto , dlsym(rtld_nex...
Read more

ssh - Vagrant Windows - ssh_exchange_identification: read: Connection reset by peer -

-
it seems can't ssh virtual box. i have virtual box running win7. host pc in win7. virtual box created through vagrant following vagrantfile. vagrant.configure("2") |config| config.vm.box = "http://aka.ms/vagrant-win7-ie11" end open cmd. go root folder of vagrantfile. execute following command: vagrant cmd displays: bringing machine 'default' 'virtualbox' provider... ==> default: clearing set forwarded ports... ==> default: clearing set network interfaces... ==> default: preparing network interfaces based on configuration... default: adapter 1: nat ==> default: forwarding ports... default: 22 (guest) => 2222 (host) (adapter 1) ==> default: booting vm... ==> default: waiting machine boot. may take few minutes... default: ssh address: 127.0.0.1:2222 default: ssh username: vagrant default: ssh auth method: private key timed out while waiting machine boot... time o...
Read more