1476446 - talos hg.mo connectivity failures (mozilla-beta)

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Andreea Pavel [:apavel]]

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=03de027328011ffb4b48e05b1fdee10dcbcefe4a&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified&selectedJob=188638239

Failure log: https://taskcluster-artifacts.net/DybQRkFGQviXh95eIO-FAw/0/public/logs/live_backing.log

[taskcluster]  workerType: gecko-t-linux-talos
[taskcluster:error]  Error downloading https://hg.mozilla.org/releases/mozilla-beta/raw-file/03de027328011ffb4b48e05b1fdee10dcbcefe4a/taskcluster/scripts/tester/test-linux.sh: Error downloading 'https://hg.mozilla.org/releases/mozilla-beta/raw-file/03de027328011ffb4b48e05b1fdee10dcbcefe4a/taskcluster/scripts/tester/test-linux.sh': Get https://hg.mozilla.org/releases/mozilla-beta/raw-file/03de027328011ffb4b48e05b1fdee10dcbcefe4a/taskcluster/scripts/tester/test-linux.sh: EOF

Comment 1

[Andreea Pavel [:apavel]]

rwood can you please take a look at this? 

From irc:

gps> apavel|sheriffduty: i'm not sure what's up with those talos hg.mo connectivity failures. i think you should file a new bug against the task owner. at the very least it is a failure of the task (or TC worker) to retry a failing HTTP request

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

this is the taskcluster layer, as I don't see anywhere in-tree where we download test-linux.sh

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

It's linked here
 https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/job/mozharness_test.py#356
and the worker downloads it.
 https://github.com/taskcluster/taskcluster-worker/blob/09e0dc2481e3a96b9c6efb2cedfc1391f9e69383/engines/native/sandbox.go#L75
that uses util.Download:
 https://github.com/taskcluster/taskcluster-worker/blob/09e0dc2481e3a96b9c6efb2cedfc1391f9e69383/runtime/util/download.go#L12
which doesn't retry.  Those TODO comments about using the fetcher are probably relevant, as it does retry:
 https://github.com/taskcluster/taskcluster-worker/blob/09e0dc2481e3a96b9c6efb2cedfc1391f9e69383/runtime/fetcher/urlfetcher.go

Pete, does generic-worker retry fetching context?  If so, this will be fixed as we switch to generic-worker..

Comment 4

[Intermittent Failures Robot]

39 failures in 653 pushes (0.06 failures/push) were associated with this bug in the last 7 days. 

This is the #18 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-beta: 39

Platform breakdown:
* linux64: 21
* linux64-qr: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1476446&startday=2018-07-16&endday=2018-07-22&tree=trunk

Comment 5

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #3)
> It's linked here
>  https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/
> transforms/job/mozharness_test.py#356
> and the worker downloads it.
>  https://github.com/taskcluster/taskcluster-worker/blob/
> 09e0dc2481e3a96b9c6efb2cedfc1391f9e69383/engines/native/sandbox.go#L75
> that uses util.Download:
>  https://github.com/taskcluster/taskcluster-worker/blob/
> 09e0dc2481e3a96b9c6efb2cedfc1391f9e69383/runtime/util/download.go#L12
> which doesn't retry.  Those TODO comments about using the fetcher are
> probably relevant, as it does retry:
>  https://github.com/taskcluster/taskcluster-worker/blob/
> 09e0dc2481e3a96b9c6efb2cedfc1391f9e69383/runtime/fetcher/urlfetcher.go
> 
> Pete, does generic-worker retry fetching context?  If so, this will be fixed
> as we switch to generic-worker..

Yes:
https://github.com/taskcluster/generic-worker/blob/v10.10.0/mounts.go#L746-L788

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

so, should be fixed now :)