Closed Bug 1306721 Opened 9 years ago Closed 8 years ago

Use exponential back-off for DescribeCluster

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P3)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: bugzilla, Unassigned)

References

Details

About 2:53pm CT I tried to kill a cluster (https://analysis.telemetry.mozilla.org/cluster/kill/j-1SPXM1UOLAHTJ) and received a page with "No such cluster: j-1SPXM1UOLAHTJ" (even though I could still ssh into it.) I tried refreshing and got the expected "Your cluster is now dying" page on the 3rd or 4th refresh. This all happened a few minutes after I intermittently started getting "No such cluster: j-1SPXM1UOLAHTJ -- error: EmrResponseError: 400 Bad Request Sender Throttling Rate exceeded 26ce9b80-874b-11e6-bff9-d3faa7125d52" on the cluster monitoring pages
This is due to reaching API limits. We should make sure that we implement something similar to [1] in atmo v2. [1] https://github.com/mozilla/telemetry-airflow/commit/dee5db5c4c073e9f15b7648f23694f2d18b24f11
Summary: "No such cluster" encountered while trying to kill active cluster on atmo → Use exponential back-off for DescribeCluster
Points: --- → 1
Priority: -- → P3
Moved to https://github.com/mozilla/telemetry-analysis-service/issues/194
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → INVALID
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.