1478093 - Deploy emr-bootstrap-spark to pick up telemetry-spark-packages-assembly

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, enhancement)

Description

[Jeff Klukas [:klukas] (UTC-4)]

We previously deployed and rolled back a change [0] to make spark-hyperloglog available by default. We've now merged a new approach to spark packages into emr-spark-bootstrap [1] and are ready for a new deploy with that code.

Before merging, I tested on a variety of cluster sizes and EMR versions and found that spark-hyperloglog was available (so the new functionality works) and that I was able to run the entire Telemetry Hello World notebook (which lends confidence that this will not break existing workflows).

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1474347 
[1] https://github.com/mozilla/emr-bootstrap-spark/pull/407#issuecomment-407488823

Comment 1

[Wesley Dawson [:whd]]

I'm at a conference this week but per https://bugzilla.mozilla.org/show_bug.cgi?id=1474347#c18 it looks like you know how to deploy emr-bootstrap-spark. Deploying to production is documented at https://github.com/mozilla/emr-bootstrap-spark/#deploy-to-aws-via-ansible and you have the requisite access to do so. Since ops isn't really providing anything of value here I suggest you deploy this to stage/prod yourself and verify it works as expected.

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

That's a totally reasonable point. I will go ahead and move this forward.

Comment 3

[Jeff Klukas [:klukas] (UTC-4)]

Deployed to stage and all looks good on a test ATMO cluster I spun up. Deploying to prod now.

Comment 4

[Jeff Klukas [:klukas] (UTC-4)]

Deployed to prod and looks good there too. Resolving.