Closed Bug 1458105 Opened 2 years ago Closed 2 years ago

Gecko Decision task bustage on Autoland


(Firefox Build System :: Task Configuration, task)

3 Branch
Not set


(firefox61 fixed)

Tracking Status
firefox61 --- fixed


(Reporter: dluca, Assigned: dustin)




(1 file)

Assignee: nobody → dustin
Comment on attachment 8972208 [details]
Bug 1458105 - try more to fetch automationrelevance;
Attachment #8972208 - Flags: review?(gps) → review+
hg error in cmd: hg pull upstream: pulling from
searching for changes
abort: HTTP Error 500: Internal Server Error
Well, isn't that a coincidence? I'll happily review a patch to make autoland retry more!
I suspect the elevated error rates are due to us replacing 2 beefy hgweb servers with 2 not-as-beefy servers. I'll VPN in and put the beefy servers back in service. We can look at the logs tomorrow to see what capacity issues were in play.
Pushed by
try more to fetch automationrelevance; r=gps
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
From IRC logs yesterday I gather that this was the issue that cause:
  ReadTimeout: HTTPSConnectionPool(host='', port=443): Read timed out
on try.

And the fix is:
 A) gps rolling back to using beefy machines,
 B) ensure requests are retried more than twice.
Duplicate of this bug: 1460415
We had some capacity problems on We mitigated those last week.

We also landed a commit to retry harder.

So double mitigation.

We may still see sporadic failures on Try and older release repos due to old code only retrying twice.
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.