1269766 - Test / deploy the Spark progress bar extension on a.t.m.o

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mark Reid [:mreid]]

Install the extension from https://github.com/mozilla/jupyter-spark and test to ensure that it works as expected on our a.t.m.o infrastructure.  If so, add it to our deploy scripts.

This provides a progress bar for Spark cells in notebooks, as well as offering a basic proxy to the Spark Web UI (to avoid having to configure tunnels / proxies / port forwarding).

Comment 1

[Jannis Leidel [:jezdez]]

Making this P2 since I'm not sure if I'll do this in the short sprint this week and it depends on #1281209.

Comment 2

[Jannis Leidel [:jezdez]]

I've tested it on the infrastructure after doing some code re-organization. This needs more tests, a PyPI release now and an update to the EMR repo next.

Comment 3

[Jannis Leidel [:jezdez]]

When I tested the extension on a Spark cluster I found some issues with the proxy code and had to rewrite some of it to use Tornado's async code. That just got released to PyPI as 0.3.0 (https://github.com/mozilla/jupyter-spark/compare/0.2.0...0.3.0). Code test coverage is now at 100% for the Python code.

Next TODO is another round of testing on a Spark cluster and an update to the EMR repo.

Comment 4

[Jannis Leidel [:jezdez]]

Okay, manual tests on a fresh cluster were successful.

I opened https://github.com/mozilla/emr-bootstrap-spark/pull/29

Comment 5

[Jannis Leidel [:jezdez]]

This has been deployed now and is available on newly created clusters.