1495460 - Avoid describe-clusters call in telemetry-sh

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Jeff Klukas [:klukas] (UTC-4)]

As :whd warned, calling aws emr describe-clusters in telemetry.sh is not reliable and sometimes fails. We had an airflow job fail bootstrap due to this.

We could add retry logic around the request, or we could add extra parameters to the bootstrap script to provide the metadata we're pulling from describe-clusters.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/emr-bootstrap-spark/pull/540

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

PR posted. I experimented with adding retry logic and decided it wasn't too unreasonable an amount of code to implement retries. I've decided to take that route rather than sending in extra parameters, which would complicate the interface.

The EMR API needs to be available for us to create clusters anyway, so it it seems reasonable that we should expect to be able to get a response in this script. It's also reasonable to expect that an HTTP call will lead to occasional transient failures and needs to be wrapped.