Closed Bug 1458105 Opened 2 years ago Closed 2 years ago
Gecko Decision task bustage on Autoland
59 bytes, text/x-review-board-request
Comment on attachment 8972208 [details] Bug 1458105 - try more to fetch automationrelevance; https://reviewboard.mozilla.org/r/240872/#review246650
Attachment #8972208 - Flags: review?(gps) → review+
hg error in cmd: hg pull upstream: pulling from https://hg.mozilla.org/integration/autoland searching for changes abort: HTTP Error 500: Internal Server Error
Well, isn't that a coincidence? I'll happily review a patch to make autoland retry more!
I suspect the elevated error rates are due to us replacing 2 beefy hgweb servers with 2 not-as-beefy servers. I'll VPN in and put the beefy servers back in service. We can look at the logs tomorrow to see what capacity issues were in play.
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/autoland/rev/6f975a367b79 try more to fetch automationrelevance; r=gps
From IRC logs yesterday I gather that this was the issue that cause: ReadTimeout: HTTPSConnectionPool(host='hg.mozilla.org', port=443): Read timed out on try. And the fix is: A) gps rolling back to using beefy machines, B) ensure requests are retried more than twice.
We had some capacity problems on hg.mo. We mitigated those last week. We also landed a commit to retry harder. So double mitigation. We may still see sporadic failures on Try and older release repos due to old code only retrying twice.
You need to log in before you can comment on or make changes to this bug.