1311285 - ATMO V2: Use ListClusters for updating cluster info

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Jannis Leidel [:jezdez]]

When fetching the cluster info we should use ListClusters instead of iterating over each cluster and getting info individually

Comment 1

[Jannis Leidel [:jezdez]]

Hm, turns out ListClusters doesn't provide `MasterPublicDnsName` in the response but just the state of the cluster (http://botocore.readthedocs.io/en/latest/reference/services/emr.html#EMR.Client.list_clusters). So we can't easily move to this API endpoint.

Mauro, should we just close this as INVALID?

Comment 2

[Mauro Doglio [:mdoglio]]

I would try to still use list clusters and eventually run a separate call if the SparkCluster object doesn't have a ip address set and the status is waiting. That would mean we only call the detail endpoint no more than once per cluster. every other info update would be retrieved using the list endpoint. How does that sound to you?

Comment 3

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] mozilla:bug1311285 > mozilla:master