Closed Bug 1165759 Opened 9 years ago Closed 9 years ago

Intermittent TC build failures due to 500/502/503/404 from wherever things are fetched

Categories

(Taskcluster :: General, defect)

Product:
Taskcluster
Component:
General
Type:
defect
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Assigned: garndt)

References

Details

(Keywords: intermittent-failure)

Sheriffs have been pretty much just ignoring this, but with talk about moving real jobs, not just b2g, to TaskCluster, something needs to actually be done about it.
Depends on: 1165760
A big spike in these is now closing trees.
Severity: normal → blocker
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #40) > A big spike in these is now closing trees. Latest retriggers are all coming back green, so I'm hesitantly reopening trees to see how it goes.
And just like that, the failures start coming back in, even after jlal landed bug 1166073.
Depends on: 1156326
See Also: → 1165765
Summary: Intermittent TC build failures due to 500/502/503/404 from quay.io → Intermittent TC build failures due to 500/502/503/404 from wherever things are fetched
Comment number 303 is actually an upload error... We probably need to start splitting up this bug... We certainly have some errors related to pulling images (which I think was the majority of the original filings) but many of those issues have been resolved by moving to docker hub. NI myself here to dig through the list tomorrow.
Flags: needinfo?(jlal)
I've entered two tickets for the most common pull errors I've been seeing lately. https://bugzilla.mozilla.org/show_bug.cgi?id=1170997 https://bugzilla.mozilla.org/show_bug.cgi?id=1170999
Having better (read: usable) failure logs would go a long ways to avoiding dumping ground bugs like these.
I have started reviewing the reports here going from newest to oldest, and I'm back to about comment 168. The two bugs in comment 306 are the primary causes of most of these. Here is an ether pad where I'm tracking what comment is what issue: https://etherpad.mozilla.org/bug-1165759 I will triage new issues reported and begin working soon on the two bugs to try to find solutions and/or mitigate the issue a bit.
(In reply to Treeherder Robot from comment #318) > log: > https://treeherder.mozilla.org/logviewer.html#?repo=b2g- > inbound&job_id=2088394 > repository: b2g-inbound > start_time: 2015-06-04T01:56:47 > who: rvandermeulen[at]mozilla[dot]com > machine: unknown > revision: abeb01323218 Failed mulet build: *** No rule to make target `/home/worker/workspace/gecko/intl/icu/source/common/norm2allmodes.h', needed by `normalizer2.o'. Stop.
Looks like comments 313 to 318 are caused by build errors.
(In reply to Treeherder Robot from comment #326) > log: > https://treeherder.mozilla.org/logviewer.html#?repo=mozilla- > inbound&job_id=10438003 > repository: mozilla-inbound > start_time: 2015-06-04T10:28:30 > who: rvandermeulen[at]mozilla[dot]com > machine: unknown > revision: b03bb71b757f Build failure. .deps/host_dump_symbols.o.pp:191: *** missing separator. Stop. *** [toolkit/crashreporter/google-breakpad/src/common/linux/target] Error 2
Blocks: 1080265
(In reply to Treeherder Robot from comment #340) > log: > https://treeherder.mozilla.org/logviewer.html#?repo=fx-team&job_id=3357879 > repository: fx-team > start_time: 2015-06-06T21:25:42 > who: philringnalda[at]gmail[dot]com > machine: t-w732-ix-116 > buildname: Windows 7 32-bit fx-team pgo test web-platform-tests-1 > revision: c0177adc8763 > > TEST-UNEXPECTED-FAIL | > /content-security-policy/script-src/script-src-1_2.html | Violation report > status OK. - assert_equals: No report sent. expected "" but got "false" > Return code: 1 Test failure INFO - TEST-UNEXPECTED-FAIL | /content-security-policy/script-src/script-src-1_2.html | Violation report status OK. - assert_equals: No report sent. expected "" but got "false"
(In reply to Treeherder Robot from comment #339) > log: > https://treeherder.mozilla.org/logviewer.html#?repo=fx-team&job_id=3357863 > repository: fx-team > start_time: 2015-06-06T21:25:41 > who: philringnalda[at]gmail[dot]com > machine: t-w732-ix-134 > buildname: Windows 7 32-bit fx-team pgo test reftest-no-accel > revision: c0177adc8763 > > REFTEST TEST-UNEXPECTED-FAIL | > file:///C:/slave/test/build/tests/reftest/tests/layout/reftests/writing-mode/ > abspos/s71-abs-pos-non-replaced-vrl-094-ref.xht | load failed: timed out > waiting for test to complete (waiting for onload scripts to complete) > Return code: 1 Test Failure
(In reply to Treeherder Robot from comment #338) > log: > https://treeherder.mozilla.org/logviewer.html#?repo=mozilla- > inbound&job_id=10516941 > repository: mozilla-inbound > start_time: 2015-06-06T22:59:33 > who: philringnalda[at]gmail[dot]com > machine: t-w732-ix-072 > buildname: Windows 7 32-bit mozilla-inbound pgo test reftest-no-accel > revision: 6eaeb7343ac6 > > REFTEST TEST-UNEXPECTED-FAIL | > file:///C:/slave/test/build/tests/reftest/tests/layout/reftests/writing-mode/ > abspos/s71-abs-pos-non-replaced-vrl-068-ref.xht | load failed: timed out > waiting for test to complete (waiting for onload scripts to complete) > Return code: 1 Test failure