1307579 - On EMR 5.0.0 Jupyter Console correctly loads Spark libs, but Ipython console doesn't

Status: RESOLVED DUPLICATE of bug 1310475

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P3)

Description

["Saptarshi Guha[:joy]"]

For example

[hadoop@ip-172-31-6-109 ~]$ pyspark
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/mnt1/ -Xmx15000M -Xms1000M
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/mnt1/ -Xmx15000M -Xms1000M
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/mnt1/ -Xmx15000M -Xms1000M
Jupyter Console 4.1.1

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).

In [1]: sc
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.0.0
      /_/

Using Python version 2.7.11 (default, Dec  6 2015 18:08:32)
SparkSession available as 'spark'.
Out[1]: <pyspark.context.SparkContext at 0x7f95045438d0>


In [2]: %cpaste
ERROR: Line magic function `%cpaste` not found.


============================ END PYSPARK SESSION =========================================


but, if i start ipython

 ipython
Python 2.7.11 |Anaconda 4.0.0 (64-bit)| (default, Dec  6 2015, 18:08:32)
Type "copyright", "credits" or "license" for more information.

IPython 4.1.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: sc
Out[1]: ''

In [2]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:


============================== END IPYTHON ===============================================

From https://github.com/jupyter/jupyter_console/issues/20 it appears cpaste wont be supported in the console. But pasting doesn;t always work(jupyter console reformatted this code [1] so that the yield came in the for loop.

Dont know what the solution is apart from fixing IPython setup to load the spark libs.



[1] https://gist.github.com/saptarshiguha/b70227c449394c6f12e82cdde261bd2f

Comment 1

["Saptarshi Guha[:joy]"]

From frank in IRC, to get spark working in ipython

import findspark
findspark.init()

import pyspark
sc = pyspark.SparkContext(appName="myAppName")

Comment 2

[Mauro Doglio [:mdoglio]]

The issue here is that the version of jupyter console installed on emr notebooks doesn't handle pasted code very well.
This is solved in the latest version, which will be installed in bug 1310475.

Comment 3

[Mauro Doglio [:mdoglio]]

Someone has already opened an issue [1] on jupyter console about that and the authors suggest to upgrade to the latest version.
I'll dupe this in favor of bug 1310475

[1]https://github.com/jupyter/jupyter_console/issues/70