1460415 - Auto-Retry when Gecko Decision Task Fails with ReadTimeout: HTTPSConnectionPool(host='hg.mozilla.org', port=443): Read timed out.

Status: RESOLVED DUPLICATE of bug 1458105

(Firefox Build System :: Task Configuration, task)

Description

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

I quite regurally (maybe... 5-10% of the time?) get Decision Task Failures saying 

> ReadTimeout: HTTPSConnectionPool(host='hg.mozilla.org', port=443): Read timed out. (read timeout=5)

Example: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d5d76018a05ef0ea02cd959b1edc07a3cca5ff9&selectedJob=177732938

All I do is retrigger the Decision Task and the second one usually succeededs.  IT would be very helpful if the DEcision Task could auto-retrigger if this happens.  

Or perhaps the Decision Task itself could catch the exception, sleep for 60 seconds, and try again?

Comment 1

[Gregory Szorc [:gps]]

It is already retrying. From the log:

[task 2018-05-09T20:14:39.703422Z] Querying version control for metadata: https://hg.mozilla.org/try/json-automationrelevance/7d5d76018a05ef0ea02cd959b1edc07a3cca5ff9
[task 2018-05-09T20:14:39.703465Z] attempt 1/2
[task 2018-05-09T20:14:39.705505Z] retry: calling get_automationrelevance, attempt #1
[task 2018-05-09T20:14:39.710482Z] Starting new HTTPS connection (1): hg.mozilla.org
[task 2018-05-09T20:14:44.763014Z] retry: Caught exception: 
[task 2018-05-09T20:14:44.763090Z] sleeping for 10.00s (attempt 1/2)
[task 2018-05-09T20:14:54.769855Z] attempt 2/2
[task 2018-05-09T20:14:54.769955Z] retry: calling get_automationrelevance, attempt #2
[task 2018-05-09T20:14:54.771269Z] Starting new HTTPS connection (1): hg.mozilla.org
[task 2018-05-09T20:14:59.939545Z] retry: Caught exception: 
[task 2018-05-09T20:14:59.939620Z] retry: Giving up on get_automationrelevance

And we bumped the number of retries up to 10 in bug 1458105. So rebase your Try push and this won't be a problem any more.