Closed Bug 1458105 Opened 2 years ago Closed 2 years ago

Gecko Decision task bustage on Autoland

Categories

(Firefox Build System :: Task Configuration, task)

3 Branch
task
Not set

Tracking

(firefox61 fixed)

RESOLVED FIXED
mozilla61
Tracking Status
firefox61 --- fixed

People

(Reporter: dluca, Assigned: dustin)

References

Details

Attachments

(1 file)

Assignee: nobody → dustin
Comment on attachment 8972208 [details]
Bug 1458105 - try more to fetch automationrelevance;

https://reviewboard.mozilla.org/r/240872/#review246650
Attachment #8972208 - Flags: review?(gps) → review+
hg error in cmd: hg pull upstream: pulling from https://hg.mozilla.org/integration/autoland
searching for changes
abort: HTTP Error 500: Internal Server Error
Well, isn't that a coincidence? I'll happily review a patch to make autoland retry more!
I suspect the elevated error rates are due to us replacing 2 beefy hgweb servers with 2 not-as-beefy servers. I'll VPN in and put the beefy servers back in service. We can look at the logs tomorrow to see what capacity issues were in play.
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6f975a367b79
try more to fetch automationrelevance; r=gps
https://hg.mozilla.org/mozilla-central/rev/6f975a367b79
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
From IRC logs yesterday I gather that this was the issue that cause:
  ReadTimeout: HTTPSConnectionPool(host='hg.mozilla.org', port=443): Read timed out
on try.

And the fix is:
 A) gps rolling back to using beefy machines,
 B) ensure requests are retried more than twice.
Duplicate of this bug: 1460415
We had some capacity problems on hg.mo. We mitigated those last week.

We also landed a commit to retry harder.

So double mitigation.

We may still see sporadic failures on Try and older release repos due to old code only retrying twice.
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.