1458105 - Gecko Decision task bustage on Autoland

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Dorel Luca [:dluca]]

Log:
https://treeherder.mozilla.org/logviewer.html#?job_id=176368130&repo=autoland

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1458105 - try more to fetch automationrelevance;

Review commit: https://reviewboard.mozilla.org/r/240872/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/240872/

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 8972208 [details]
Bug 1458105 - try more to fetch automationrelevance;

https://reviewboard.mozilla.org/r/240872/#review246650

Comment 3

[Mozilla Autoland]

hg error in cmd: hg pull upstream: pulling from https://hg.mozilla.org/integration/autoland
searching for changes
abort: HTTP Error 500: Internal Server Error

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Well, isn't that a coincidence? I'll happily review a patch to make autoland retry more!

Comment 5

[Gregory Szorc [:gps]]

I suspect the elevated error rates are due to us replacing 2 beefy hgweb servers with 2 not-as-beefy servers. I'll VPN in and put the beefy servers back in service. We can look at the logs tomorrow to see what capacity issues were in play.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6f975a367b79
try more to fetch automationrelevance; r=gps

Comment 7

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/6f975a367b79

Comment 8

[Jonas Finnemann Jensen (:jonasfj)]

From IRC logs yesterday I gather that this was the issue that cause:
  ReadTimeout: HTTPSConnectionPool(host='hg.mozilla.org', port=443): Read timed out
on try.

And the fix is:
 A) gps rolling back to using beefy machines,
 B) ensure requests are retried more than twice.

Comment 10

[Gregory Szorc [:gps]]

We had some capacity problems on hg.mo. We mitigated those last week.

We also landed a commit to retry harder.

So double mitigation.

We may still see sporadic failures on Try and older release repos due to old code only retrying twice.