1494241 - Multiple failures eg: ERROR - The following files failed: 'AVDs-armv7a-android-4.3.1_r1-build-2016-08-02.tar.gz' on all platforms

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: ServicesCore, defect)

Description

[Noemi Erli[:noemi_erli]]

Failures logs:
https://treeherder.mozilla.org/logviewer.html#?job_id=201644750&repo=autoland&lineNumber=813
https://treeherder.mozilla.org/logviewer.html#?job_id=201647724&repo=autoland&lineNumber=869
https://treeherder.mozilla.org/logviewer.html#?job_id=201644309&repo=autoland&lineNumber=468
https://treeherder.mozilla.org/logviewer.html#?job_id=201645278&repo=mozilla-inbound&lineNumber=612

Comment 1

[Eliza Balazs [:ebalazs_]]

Trees are closed for this.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Seems like a general tooltool issue. Beta is hitting other failures like OSX SDK and MSVC toolchain.

Comment 3

[Zsolt Fay [:zfay]]

Logs are showing errors to fetch files from https://tooltool.mozilla-releng.net/ and http://relengapi/tooltool/. 
Have escalated this into #moc and we were unsuccessful to get to the root of the issue. The logs seem to point to an invalid hostname.
SSh-ed into a machine that ran a test, and tried "wget http://relengapi/tooltool/sha512/8811050fe375bcc566c8b85173d86b8a87aa2148edfed93023735c2de44b66a5a28cbaa1cfd396032447fd803e03f308ed941a200c0e2a1ad9fbe16b5606ee7c" which returned https://irccloud.mozilla.com/pastebin/W9DFymIE/
After speaking with adelbarrio from #moc he told me it's weird why this is the complete URL that appears in the logs.

Comment 4

[Pete Moore [:pmoore][:pete]]

The hostname 'relengapi' is resolved inside the docker container running the task, which links to a sidecar container running https://github.com/taskcluster/relengapi-proxy/

There have been no changes to gecko-3-b-linux since 27 July, so I suspect a hosting issue here with the releng services.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Note that TreeStatus (also on mozilla-releng.net) is also down with a Network Error at the moment.

Comment 6

[Intermittent Failures Robot]

78 failures in 520 pushes (0.15 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 78

Platform breakdown:
* windows2012-32-devedition: 3
* android-5-0-aarch64: 1
* osx-cross-devedition: 3
* osx-cross: 12
* windows2012-64-add-on-devel: 3
* windows2012-32: 18
* linux64-nightly: 2
* android-4-0-armv7-api16: 2
* windows2012-64: 21
* taskcluster-images: 2
* osx-cross-add-on-devel: 3
* windows2012-32-add-on-devel: 3
* windows2012-64-devedition: 3
* android-4-2-x86: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1494241&startday=2018-09-26&endday=2018-09-26&tree=all

Comment 7

[Dave Miller [:justdave]]

Where on the network are these located? In AWS? MDC1?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Sorry, this isn't a Netops issue. It's related to an issue with Release Services which :garbas resolved yesterday.

Comment 9

[Intermittent Failures Robot]

78 failures in 2773 pushes (0.028 failures/push) were associated with this bug in the last 7 days.

This is the #23 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-beta: 78

Platform breakdown:
* windows2012-32: 18
* osx-cross-devedition: 3
* osx-cross-add-on-devel: 3
* windows2012-64-add-on-devel: 3
* windows2012-64-devedition: 3
* android-4-2-x86: 2
* windows2012-32-devedition: 3
* osx-cross: 12
* windows2012-64: 21
* linux64-nightly: 2
* windows2012-32-add-on-devel: 3
* taskcluster-images: 2
* android-5-0-aarch64: 1
* android-4-0-armv7-api16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1494241&startday=2018-09-24&endday=2018-09-30&tree=all