1303012 - Jupyter (with runipy?) breaks with EMR release 5

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Roberto Agostino Vitillo (:rvitillo)]

I have tried to schedule a very simple notebook (https://github.com/mozilla/telemetry-airflow/blob/master/examples/spark/example_date.ipynb) with Airflow using EMR release 5 and I saw this error:

time env $environment runipy $runner_args "./$job" "./output/$job" --pylab
09/15/2016 01:02:53 PM INFO: Reading notebook ./example_date.ipynb
09/15/2016 01:02:53 PM WARNING: --pylab is deprecated and will be removed in a future version
Traceback (most recent call last):
  File "/home/hadoop/anaconda2/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/hadoop/anaconda2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/ipykernel/__main__.py", line 3, in <module>
    app.launch_new_instance()
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/traitlets/config/application.py", line 595, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-122>", line 2, in initialize
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/traitlets/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 412, in initialize
    self.init_sockets()
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 237, in init_sockets
    self.stdin_port = self._bind_socket(self.stdin_socket, self.stdin_port)
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 174, in _bind_socket
    s.bind("tcp://%s:%i" % (self.ip, port))
  File "zmq/backend/cython/socket.pyx", line 487, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5156)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7535)
    raise ZMQError(errno)
ZMQError: Address already in use
Traceback (most recent call last):
  File "/home/hadoop/anaconda2/bin/runipy", line 11, in <module>
    sys.exit(main())
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/runipy/main.py", line 153, in main
    nb, args.pylab, args.matplotlib, profile_dir, working_dir
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/runipy/notebook_runner.py", line 106, in __init__
    self.kc.wait_for_ready()
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/jupyter_client/blocking/client.py", line 59, in wait_for_ready
    raise RuntimeError('Kernel died before replying to kernel_info')
RuntimeError: Kernel died before replying to kernel_info

Comment 1

[Mauro Doglio [:mdoglio]]

I can see errors opening a jupyter notebook, so the problem must be in the jupyter configuration. It looks like the 'sc' variable isn't in the context when the notebook is opened.

Comment 2

[GitHub Autolander Bot]

Attachment: [emr-bootstrap-spark] mozilla:bug-1303012-emr-5 > mozilla:master