Intermittent [taskcluster:error] Pulling docker image has failed.

NEW
Unassigned

Status

Taskcluster
Docker Images
--
blocker
a year ago
5 days ago

People

(Reporter: Treeherder Bug Filer, Unassigned)

Tracking

({intermittent-failure})

Details

(Whiteboard: [stockwell infra])

Duplicate of this bug: 1297731
Duplicate of this bug: 1298488
Duplicate of this bug: 1299025
Duplicate of this bug: 1299416
Duplicate of this bug: 1300001
Duplicate of this bug: 1300009
Duplicate of this bug: 1300243
Duplicate of this bug: 1300317
Duplicate of this bug: 1300350
Duplicate of this bug: 1296659
29 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 12
* mozilla-inbound: 10
* try: 3
* mozilla-central: 2
* fx-team: 2

Platform breakdown:
* linux64: 26
* android-4-3-armv7-api15: 2
* android-api-15-gradle: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-14&endday=2016-09-14&tree=all
36 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 14
* autoland: 10
* mozilla-central: 7
* fx-team: 4
* try: 1

Platform breakdown:
* linux64: 30
* android-4-3-armv7-api15: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-15&endday=2016-09-15&tree=all
23 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-release: 12
* mozilla-inbound: 5
* autoland: 4
* mozilla-central: 2

Platform breakdown:
* linux64: 21
* android-4-3-armv7-api15: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-16&endday=2016-09-16&tree=all
95 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 32
* autoland: 29
* mozilla-release: 12
* mozilla-central: 12
* fx-team: 6
* try: 4

Platform breakdown:
* linux64: 83
* android-4-3-armv7-api15: 11
* android-api-15-gradle: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-12&endday=2016-09-18&tree=all
62 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 34
* mozilla-inbound: 17
* mozilla-central: 7
* fx-team: 4

Platform breakdown:
* linux64: 60
* android-4-3-armv7-api15: 1
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-21&endday=2016-09-21&tree=all
20 automation job failures were associated with this bug yesterday.

Repository breakdown:
* fx-team: 9
* autoland: 6
* mozilla-aurora: 4
* mozilla-inbound: 1

Platform breakdown:
* linux64: 17
* android-4-3-armv7-api15: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-22&endday=2016-09-22&tree=all
39 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 20
* mozilla-inbound: 18
* fx-team: 1

Platform breakdown:
* linux64: 39

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-23&endday=2016-09-23&tree=all
92 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 45
* try: 39
* mozilla-inbound: 8

Platform breakdown:
* linux64: 90
* android-4-3-armv7-api15: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-24&endday=2016-09-24&tree=all
229 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 110
* mozilla-inbound: 50
* try: 40
* fx-team: 15
* mozilla-central: 9
* mozilla-aurora: 4
* mozilla-beta: 1

Platform breakdown:
* linux64: 221
* android-4-3-armv7-api15: 7
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-19&endday=2016-09-25&tree=all
From Greg in https://bugzilla.mozilla.org/show_bug.cgi?id=1296659#c5

Looking back at the last dozen or so of these recorded, the workers are removing an image during a garbage collection cycle at the same time as the worker is importing an image.  I'm guessing that during import of an image tarball there are less locks on shared layers than when doing a pull from docker hub.
Depends on: 1305442
95 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 54
* mozilla-inbound: 21
* mozilla-central: 17
* fx-team: 3

Platform breakdown:
* linux64: 91
* android-4-3-armv7-api15: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-26&endday=2016-09-26&tree=all
59 automation job failures were associated with this bug yesterday.

Repository breakdown:
* try: 19
* mozilla-inbound: 18
* autoland: 14
* mozilla-central: 4
* mozilla-aurora: 2
* fx-team: 2

Platform breakdown:
* linux64: 58
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-27&endday=2016-09-27&tree=all
368 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 269
* mozilla-inbound: 46
* try: 21
* mozilla-central: 21
* fx-team: 6
* mozilla-aurora: 5

Platform breakdown:
* linux64: 357
* android-4-3-armv7-api15: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-09-26&endday=2016-10-02&tree=all
19 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 8
* autoland: 7
* mozilla-aurora: 3
* try: 1

Platform breakdown:
* linux64: 16
* android-4-3-armv7-api15: 2
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-10-03&endday=2016-10-09&tree=all
26 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 14
* autoland: 6
* mozilla-release: 3
* mozilla-aurora: 2
* mozilla-central: 1

Platform breakdown:
* linux64: 24
* android-4-3-armv7-api15: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-10-10&endday=2016-10-16&tree=all
35 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 14
* mozilla-inbound: 10
* mozilla-aurora: 5
* mozilla-central: 3
* try: 1
* mozilla-beta: 1
* fx-team: 1

Platform breakdown:
* linux64: 27
* android-4-3-armv7-api15: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-10-17&endday=2016-10-23&tree=all
369 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 279
* try: 73
* mozilla-inbound: 13
* autoland: 2
* mozilla-release: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 367
* osx-10-7: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-10-27&endday=2016-10-27&tree=all
39 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 36
* mozilla-inbound: 1
* mozilla-central: 1
* mozilla-aurora: 1

Platform breakdown:
* osx-10-7: 35
* linux64: 3
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-10-28&endday=2016-10-28&tree=all
433 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 282
* try: 77
* autoland: 45
* mozilla-inbound: 22
* mozilla-central: 5
* mozilla-release: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 389
* osx-10-7: 37
* android-4-3-armv7-api15: 5
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-10-24&endday=2016-10-30&tree=all
27 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 24
* mozilla-inbound: 2
* mozilla-aurora: 1

Platform breakdown:
* lint: 16
* linux64: 10
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-11-03&endday=2016-11-03&tree=all
this problem is increasing as of yesterday- either a fix was backed out, or we have a new problem.

what I see as issues in the last couple of days is primarily build tasks- where as before we had other *lint* and unittest tasks hitting this issue.

As of now, this is the #3 intermittent orange on the trees (ignoring 2 others with fixes landed)

:garndt, can you make sure this gets on the radar?
Flags: needinfo?(garndt)
aurora was running about ~ 300 times into this problem here and according to dustin this is now affecting all branches, so closing trees and bumping to blocker
Severity: normal → blocker
Depends on: 1314284
(In reply to Carsten Book [:Tomcat] from comment #32)
> aurora was running about ~ 300 times into this problem here and according to
> dustin this is now affecting all branches, so closing trees and bumping to
> blocker

note that trees are reopen since awhile now, also no other trees were affected
> :garndt, can you make sure this gets on the radar?

This has been an ongoing issue we've been looking into.  Changes have been deployed this morning that we hope will eliminate these errors.  We will be monitoring.

More information is in https://bugzilla.mozilla.org/show_bug.cgi?id=1314284
Flags: needinfo?(garndt)
727 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 700
* mozilla-inbound: 17
* autoland: 8
* try: 1
* mozilla-release: 1

Platform breakdown:
* linux64: 644
* android-4-3-armv7-api15: 64
* lint: 6
* linux32: 5
* android-4-0-armv7-api15: 3
* windows7-32-vm: 1
* osx-10-7: 1
* osx-10-10: 1
* android-4-4-armv7-api15: 1
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-11-04&endday=2016-11-04&tree=all
772 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 705
* autoland: 33
* mozilla-inbound: 25
* try: 5
* mozilla-central: 3
* mozilla-release: 1

Platform breakdown:
* linux64: 671
* android-4-3-armv7-api15: 65
* lint: 22
* linux32: 6
* android-4-0-armv7-api15: 3
* windows7-32-vm: 1
* osx-10-7: 1
* osx-10-10: 1
* android-4-4-armv7-api15: 1
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-10-31&endday=2016-11-06&tree=all
37 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 17
* mozilla-inbound: 12
* try: 3
* mozilla-release: 2
* mozilla-central: 2
* mozilla-aurora: 1

Platform breakdown:
* linux64: 36
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-11-07&endday=2016-11-13&tree=all
39 failures in 715 pushes (0.055 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 17
* mozilla-inbound: 10
* try: 4
* graphics: 3
* mozilla-central: 2
* mozilla-aurora: 2
* mozilla-release: 1

Platform breakdown:
* linux64: 36
* android-4-3-armv7-api15: 2
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-11-14&endday=2016-11-20&tree=all
20 failures in 124 pushes (0.161 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 10
* mozilla-inbound: 6
* try: 2
* mozilla-aurora: 2

Platform breakdown:
* linux64: 19
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-11-23&endday=2016-11-23&tree=all
66 failures in 623 pushes (0.106 failures/push) were associated with this bug in the last 7 days. 

This is the #15 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* autoland: 31
* mozilla-inbound: 22
* try: 5
* mozilla-aurora: 5
* mozilla-beta: 2
* mozilla-central: 1

Platform breakdown:
* linux64: 60
* android-4-3-armv7-api15: 5
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-11-21&endday=2016-11-27&tree=all
29 failures in 694 pushes (0.042 failures/push) were associated with this bug in the last 7 days. 

This is the #45 most frequent failure this week. 

Repository breakdown:
* mozilla-inbound: 11
* autoland: 9
* mozilla-aurora: 4
* mozilla-release: 2
* try: 1
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 26
* android-4-3-armv7-api15: 2
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-11-28&endday=2016-12-04&tree=all
14 failures in 289 pushes (0.048 failures/push) were associated with this bug in the last 7 days. 

This is the #48 most frequent failure this week. 

Repository breakdown:
* mozilla-central: 5
* mozilla-inbound: 3
* autoland: 3
* try: 1
* mozilla-aurora: 1
* graphics: 1

Platform breakdown:
* linux64: 14

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-12-05&endday=2016-12-11&tree=all
30 failures in 526 pushes (0.057 failures/push) were associated with this bug in the last 7 days. 

This is the #21 most frequent failure this week. 

Repository breakdown:
* autoland: 11
* mozilla-inbound: 9
* try: 5
* mozilla-central: 2
* graphics: 2
* mozilla-aurora: 1

Platform breakdown:
* linux64: 26
* android-4-3-armv7-api15: 3
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-12-12&endday=2016-12-18&tree=all
:garndt, this is still ongoing at a pretty steady rate- the data isn't clustered, so this indicates we are dealing with something not related to a common outage (even if it happens once/day) instead this is happening 5-6 times/day and spread out evenly throughout the day.

Should we make this an auto retry case?
Flags: needinfo?(garndt)
45 failures in 609 pushes (0.074 failures/push) were associated with this bug in the last 7 days. 

This is the #9 most frequent failure this week. 

Repository breakdown:
* autoland: 14
* mozilla-inbound: 13
* try: 5
* mozilla-central: 4
* graphics: 4
* mozilla-aurora: 3
* mozilla-beta: 2

Platform breakdown:
* linux64: 41
* android-4-3-armv7-api15: 2
* linux64-qr: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-12-19&endday=2016-12-25&tree=all
18 failures in 305 pushes (0.059 failures/push) were associated with this bug in the last 7 days. 

This is the #44 most frequent failure this week. 

Repository breakdown:
* mozilla-inbound: 8
* mozilla-aurora: 4
* autoland: 4
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 17
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2016-12-26&endday=2017-01-01&tree=all
33 failures in 563 pushes (0.059 failures/push) were associated with this bug in the last 7 days. 

This is the #41 most frequent failure this week. 

Repository breakdown:
* autoland: 15
* mozilla-inbound: 7
* mozilla-beta: 6
* mozilla-aurora: 3
* try: 1
* mozilla-release: 1

Platform breakdown:
* linux64: 31
* android-4-3-armv7-api15: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-02&endday=2017-01-08&tree=all
15 failures in 128 pushes (0.117 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 7
* autoland: 5
* mozilla-central: 2
* mozilla-beta: 1

Platform breakdown:
* linux64: 13
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-10&endday=2017-01-10&tree=all
17 failures in 155 pushes (0.11 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 8
* autoland: 7
* try: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 12
* linux32: 4
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-11&endday=2017-01-11&tree=all
21 failures in 118 pushes (0.178 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 13
* mozilla-inbound: 5
* try: 2
* mozilla-aurora: 1

Platform breakdown:
* linux64: 15
* linux32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-13&endday=2017-01-13&tree=all
86 failures in 722 pushes (0.119 failures/push) were associated with this bug in the last 7 days. 

This is the #16 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* autoland: 39
* mozilla-inbound: 28
* mozilla-aurora: 6
* try: 5
* mozilla-central: 5
* graphics: 2
* mozilla-beta: 1

Platform breakdown:
* linux64: 68
* linux32: 16
* android-4-3-armv7-api15: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-09&endday=2017-01-15&tree=all
30 failures in 165 pushes (0.182 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 14
* autoland: 6
* mozilla-central: 4
* graphics: 4
* try: 1
* mozilla-aurora: 1

Platform breakdown:
* linux32: 17
* linux64: 12
* linux64-qr: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-18&endday=2017-01-18&tree=all
:garndt, I am not sure if this will be fixed by the sidecar work (bug 1307493), but I wanted to make sure this is on your radar for the intermittent work in this quarter.
23 failures in 143 pushes (0.161 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 10
* autoland: 10
* graphics: 2
* mozilla-central: 1

Platform breakdown:
* linux64: 17
* linux32: 5
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-20&endday=2017-01-20&tree=all
103 failures in 690 pushes (0.149 failures/push) were associated with this bug in the last 7 days. 

This is the #8 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* mozilla-inbound: 46
* autoland: 34
* graphics: 7
* try: 6
* mozilla-central: 6
* mozilla-aurora: 4

Platform breakdown:
* linux64: 65
* linux32: 36
* linux64-qr: 1
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-16&endday=2017-01-22&tree=all
19 failures in 127 pushes (0.15 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 10
* autoland: 8
* try: 1

Platform breakdown:
* linux64: 14
* linux32: 4
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-23&endday=2017-01-23&tree=all
27 failures in 152 pushes (0.178 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 12
* mozilla-inbound: 7
* mozilla-central: 3
* mozilla-beta: 2
* mozilla-aurora: 2
* try: 1

Platform breakdown:
* linux64: 17
* linux32: 9
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-24&endday=2017-01-24&tree=all
23 failures in 153 pushes (0.15 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 12
* autoland: 9
* try: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 16
* linux32: 5
* android-4-3-armv7-api15: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-25&endday=2017-01-25&tree=all
16 failures in 108 pushes (0.148 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 9
* autoland: 5
* try: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 9
* linux32: 6
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-26&endday=2017-01-26&tree=all
now that sidecar is landed, we still have a very high failure rate for this bug, :garndt, who can take this work and figure out a solution?
27 failures in 141 pushes (0.191 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 11
* mozilla-inbound: 8
* mozilla-beta: 3
* try: 2
* mozilla-central: 1
* mozilla-aurora: 1
* graphics: 1

Platform breakdown:
* linux64: 13
* linux32: 12
* linux64-qr: 1
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-27&endday=2017-01-27&tree=all
120 failures in 749 pushes (0.16 failures/push) were associated with this bug in the last 7 days. 

This is the #9 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* mozilla-inbound: 48
* autoland: 48
* try: 7
* mozilla-beta: 7
* mozilla-central: 5
* mozilla-aurora: 3
* graphics: 2

Platform breakdown:
* linux64: 75
* linux32: 37
* android-4-3-armv7-api15: 6
* linux64-qr: 1
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-23&endday=2017-01-29&tree=all
21 failures in 733 pushes (0.029 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 9
* mozilla-inbound: 5
* mozilla-aurora: 3
* try: 2
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 12
* linux32: 5
* android-4-3-armv7-api15: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-01-30&endday=2017-02-05&tree=all

Comment 64

11 months ago
I think for the most part this error has become much less because we are now bundling the images inside the AMI.  There are still issues sometimes with linking the containers, but I think those are being captured under other bugs.
Flags: needinfo?(garndt)
Whiteboard: [stockwell infra]

Comment 65

11 months ago
14 failures in 812 pushes (0.017 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* mozilla-aurora: 7
* mozilla-inbound: 2
* mozilla-beta: 2
* mozilla-esr52: 1
* mozilla-central: 1
* graphics: 1

Platform breakdown:
* linux64: 9
* linux32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-02-20&endday=2017-02-26&tree=all

Comment 66

11 months ago
8 failures in 783 pushes (0.01 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* autoland: 5
* mozilla-release: 1
* mozilla-inbound: 1
* mozilla-aurora: 1

Platform breakdown:
* android-4-3-armv7-api15: 4
* linux64: 2
* linux64-stylo: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-02-27&endday=2017-03-05&tree=all

Comment 67

10 months ago
10 failures in 777 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 4
* mozilla-inbound: 2
* mozilla-aurora: 2
* try: 1
* mozilla-esr52: 1

Platform breakdown:
* linux64: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-03-13&endday=2017-03-19&tree=all

Comment 68

10 months ago
13 failures in 845 pushes (0.015 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 6
* graphics: 4
* mozilla-inbound: 3

Platform breakdown:
* linux64: 7
* linux32: 3
* android-4-3-armv7-api15: 2
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-03-27&endday=2017-04-02&tree=all

Comment 69

10 months ago
17 failures in 35 pushes (0.486 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 8
* mozilla-inbound: 4
* try: 3
* mozilla-aurora: 2

Platform breakdown:
* linux64: 13
* linux32: 2
* android-4-3-armv7-api15: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-04-08&endday=2017-04-08&tree=all

Comment 70

10 months ago
37 failures in 867 pushes (0.043 failures/push) were associated with this bug in the last 7 days. 

This is the #48 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 18
* mozilla-inbound: 10
* try: 3
* mozilla-aurora: 3
* oak: 1
* mozilla-esr52: 1
* graphics: 1

Platform breakdown:
* linux64: 20
* android-4-3-armv7-api15: 10
* linux32: 6
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-04-03&endday=2017-04-09&tree=all
this had a spike on April 7th, otherwise we are having a slow and steady rate still

Comment 72

9 months ago
19 failures in 894 pushes (0.021 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 8
* mozilla-inbound: 6
* mozilla-release: 3
* try: 1
* mozilla-esr52: 1

Platform breakdown:
* linux64: 8
* linux32: 6
* osx-10-7: 2
* linux64-stylo: 2
* linux64-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-04-10&endday=2017-04-16&tree=all

Comment 73

9 months ago
20 failures in 162 pushes (0.123 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-aurora: 18
* graphics: 2

Platform breakdown:
* linux64: 8
* linux32: 6
* osx-10-10: 3
* windowsxp: 1
* windows8-64: 1
* linux64-qr: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-04-21&endday=2017-04-21&tree=all

Comment 74

9 months ago
36 failures in 817 pushes (0.044 failures/push) were associated with this bug in the last 7 days. 

This is the #24 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-aurora: 18
* autoland: 7
* graphics: 4
* mozilla-inbound: 3
* try: 2
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 16
* linux32: 11
* osx-10-10: 3
* osx-10-7: 2
* windowsxp: 1
* windows8-64: 1
* linux64-stylo: 1
* linux64-qr: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-04-17&endday=2017-04-23&tree=all

Comment 75

9 months ago
25 failures in 135 pushes (0.185 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 11
* autoland: 7
* try: 3
* mozilla-central: 2
* mozilla-beta: 2

Platform breakdown:
* android-4-3-armv7-api15: 13
* linux64: 8
* linux32: 3
* android-api-15-gradle: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-04-26&endday=2017-04-26&tree=all

Comment 76

9 months ago
33 failures in 883 pushes (0.037 failures/push) were associated with this bug in the last 7 days. 

This is the #33 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 12
* mozilla-central: 8
* autoland: 7
* try: 4
* mozilla-beta: 2

Platform breakdown:
* android-4-3-armv7-api15: 13
* linux64: 12
* linux32: 5
* windows8-64: 2
* android-api-15-gradle: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-04-24&endday=2017-04-30&tree=all

Comment 77

9 months ago
The spike of these appear to be from a single machine having an issue, which is a rare occurrence.  Mostly with this in the log:
"Error: 140511475332992:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac:../deps/openssl/openssl/ssl/s3_pkt.c:532"

Comment 78

9 months ago
(In reply to Greg Arndt [:garndt] from comment #77)
> The spike of these appear to be from a single machine having an issue, which
> is a rare occurrence.  Mostly with this in the log:
> "Error: 140511475332992:error:1408F119:SSL
> routines:SSL3_GET_RECORD:decryption failed or bad record
> mac:../deps/openssl/openssl/ssl/s3_pkt.c:532"

If a single machine is responsible for a bulk of the reports, then I wouldn't be surprised if that machine has bad hardware, most likely memory or a NIC.

Comment 79

9 months ago
5 failures in 770 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 3
* graphics: 1
* autoland: 1

Platform breakdown:
* osx-10-10: 3
* linux64: 1
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-05-01&endday=2017-05-07&tree=all

Comment 80

8 months ago
57 failures in 879 pushes (0.065 failures/push) were associated with this bug in the last 7 days. 

This is the #17 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 17
* mozilla-inbound: 16
* mozilla-central: 13
* try: 6
* mozilla-beta: 4
* mozilla-esr52: 1

Platform breakdown:
* android-4-3-armv7-api15: 28
* linux64: 14
* linux32: 11
* osx-10-7: 1
* linux64-nightly: 1
* android-4-2-x86: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-05-08&endday=2017-05-14&tree=all

Comment 81

8 months ago
This was all caused by 1 worker, i-0efc6eabb37ba822d.  Currently none of our workers have health checks to see if they should stop claiming tasks.  In the case of pulling a docker image and loading, perhaps if the last N jobs have failed, we should have the work self-terminate.

Comment 82

8 months ago
15 failures in 777 pushes (0.019 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 5
* mozilla-esr52: 2
* mozilla-central: 2
* mozilla-beta: 2
* autoland: 2
* mozilla-release: 1
* graphics: 1

Platform breakdown:
* linux64: 6
* android-4-3-armv7-api15: 6
* linux32: 2
* linux32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-05-15&endday=2017-05-21&tree=all

Comment 83

8 months ago
18 failures in 147 pushes (0.122 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-beta: 12
* mozilla-inbound: 2
* mozilla-central: 2
* autoland: 2

Platform breakdown:
* android-4-3-armv7-api15: 5
* linux32: 4
* toolchains: 3
* osx-10-7: 3
* linux64: 2
* osx-10-7-add-on-devel: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-05-22&endday=2017-05-22&tree=all

Comment 84

8 months ago
20 failures in 891 pushes (0.022 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 12
* autoland: 4
* mozilla-inbound: 2
* mozilla-central: 2

Platform breakdown:
* android-4-3-armv7-api15: 5
* osx-10-7: 4
* linux32: 4
* toolchains: 3
* linux64: 3
* osx-10-7-add-on-devel: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-05-22&endday=2017-05-28&tree=all

Comment 85

8 months ago
5 failures in 820 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 3
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* linux32: 2
* android-4-3-armv7-api15: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-05-29&endday=2017-06-04&tree=all

Comment 86

7 months ago
2 failures in 814 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* linux64: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 87

7 months ago
1 failures in 892 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* linux64-stylo: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 88

7 months ago
8 failures in 718 pushes (0.011 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 4
* mozilla-beta: 2
* autoland: 2

Platform breakdown:
* linux64: 3
* linux32: 2
* osx-10-7: 1
* osx-10-10: 1
* linux64-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-06-26&endday=2017-07-02&tree=all

Comment 89

6 months ago
2 failures in 656 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* pine: 1
* mozilla-inbound: 1

Platform breakdown:
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-07-03&endday=2017-07-09&tree=all

Comment 90

6 months ago
4 failures in 720 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 2
* mozilla-beta: 1
* autoland: 1

Platform breakdown:
* linux64: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 91

6 months ago
3 failures in 822 pushes (0.004 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 3

Platform breakdown:
* linux64-stylo: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-07-17&endday=2017-07-23&tree=all

Comment 92

6 months ago
2 failures in 1008 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1
* mozilla-beta: 1

Platform breakdown:
* windowsxp: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-07-24&endday=2017-07-30&tree=all

Comment 93

6 months ago
1 failures in 888 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 1

Platform breakdown:
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 94

5 months ago
1 failures in 908 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-08-21&endday=2017-08-27&tree=all

Comment 95

5 months ago
24 failures in 939 pushes (0.026 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 12
* try: 6
* mozilla-inbound: 4
* mozilla-central: 2

Platform breakdown:
* linux64: 12
* linux32-stylo: 4
* linux32: 4
* linux64-stylo-sequential: 2
* linux64-qr: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-08-28&endday=2017-09-03&tree=all

Comment 96

4 months ago
21 failures in 924 pushes (0.023 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 10
* try: 6
* mozilla-inbound: 4
* mozilla-beta: 1

Platform breakdown:
* linux64: 6
* linux32-stylo: 4
* linux32: 3
* linux64-stylo-sequential: 2
* linux32-stylo-disabled: 2
* linux64-stylo-disabled: 1
* linux64-stylo: 1
* linux64-qr: 1
* linux64-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-09-04&endday=2017-09-10&tree=all

Comment 97

4 months ago
23 failures in 247 pushes (0.093 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* try: 6
* mozilla-inbound: 5
* mozilla-central: 2

Platform breakdown:
* linux64: 11
* linux32: 7
* linux64-qr: 2
* linux64-stylo-disabled: 1
* linux64-nightly: 1
* linux32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-09-13&endday=2017-09-13&tree=all

Comment 98

4 months ago
37 failures in 1032 pushes (0.036 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 13
* try: 11
* mozilla-inbound: 8
* mozilla-central: 4
* mozilla-beta: 1

Platform breakdown:
* linux64: 18
* linux32: 10
* linux64-qr: 2
* lint: 2
* linux64-stylo-disabled: 1
* linux64-stylo: 1
* linux64-nightly: 1
* linux64-devedition: 1
* linux32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 99

4 months ago
Mixture of corrupted tarballs that corrected itself and s3 slow down errors from when AWS had an outtage.

Comment 100

4 months ago
16 failures in 885 pushes (0.018 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 7
* mozilla-central: 4
* mozilla-inbound: 3
* try: 1
* mozilla-beta: 1

Platform breakdown:
* gecko-decision: 6
* linux64: 5
* linux32: 2
* linux32-stylo-disabled: 1
* linux32-nightly: 1
* gecko-3-decision: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-09-25&endday=2017-10-01&tree=all

Comment 101

3 months ago
34 failures in 824 pushes (0.041 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 16
* mozilla-inbound: 8
* try: 3
* mozilla-esr52: 3
* mozilla-central: 2
* mozilla-beta: 2

Platform breakdown:
* linux64: 16
* linux32: 9
* linux64-stylo-disabled: 4
* linux32-stylo-disabled: 4
* linux64-jsdcov: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-10-02&endday=2017-10-08&tree=all
Duplicate of this bug: 1406846

Comment 103

3 months ago
15 failures in 188 pushes (0.08 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 5
* mozilla-beta: 5
* mozilla-central: 3
* autoland: 2

Platform breakdown:
* linux64: 6
* linux64-stylo-disabled: 3
* linux32-stylo-disabled: 3
* linux64-nightly: 1
* linux32-devedition: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-10-11&endday=2017-10-11&tree=all

Comment 104

3 months ago
75 failures in 947 pushes (0.079 failures/push) were associated with this bug in the last 7 days. 

This is the #21 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* mozilla-inbound: 28
* autoland: 22
* mozilla-central: 14
* try: 7
* mozilla-beta: 4

Platform breakdown:
* linux64: 43
* linux32-stylo-disabled: 16
* linux64-stylo-disabled: 8
* linux32: 6
* linux64-qr: 1
* linux32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-10-09&endday=2017-10-15&tree=all
:garndt, this has picked up a bit in the last week, can you look to see if there is a new failure?
Flags: needinfo?(garndt)
Sorry, should have commented yesterday when I looked into it.  These failures seem to be much of the same that we've been seeing.  In this case it was some machines reporting the issue multiple times which made it worse.  So far it looks like a corrupted download on some of the machines.  We are working towards having content hashes for artifacts this quarter that will prevent something like this from happening.  Or at least make it so that that particular reason is no longer an issue.
Flags: needinfo?(garndt)

Comment 107

3 months ago
24 failures in 221 pushes (0.109 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 11
* mozilla-inbound: 10
* mozilla-esr52: 2
* mozilla-beta: 1

Platform breakdown:
* linux64: 12
* linux32: 8
* linux64-stylo-disabled: 2
* linux32-stylo-disabled: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-10-17&endday=2017-10-17&tree=all

Comment 108

3 months ago
16 failures in 162 pushes (0.099 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 9
* mozilla-inbound: 3
* mozilla-beta: 3
* mozilla-central: 1

Platform breakdown:
* linux64: 8
* linux64-stylo-disabled: 3
* linux32: 3
* linux64-nightly: 1
* linux32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-10-19&endday=2017-10-19&tree=all
So far what I've determined for the failures that occur when extracting the downloaded image and result in "Error 39 : Read error : premature end":

1. not related to artifact transfers between regions, failures have been noted in us-west-2.  Looking at most of the recent failures this month, it's been machines only in us-west-2.  This is not surprising, over 90% of the gecko-t-linux-medium machines are provisioned in us-west-2 so it's more likely to see errors there.
2. usually isolated to particular machines
3. once those machines are bad, they are always bad [1][2][3] and will fail at downloading the file (so retrying within a single task won't help since claiming additional tasks and downloading again is the same outcome as retrying)
4. it gets to different points in the tar extract and fails (not consistent)
5. Downloaded file size is always the same between failed and successful loads.  Perhaps something with the content is corrupted resulting in the same size, but bad content.  Work going on now to have headers defining the content hash for s3 objects will help answer this and make it not a possibility
6. always m1.medium machines



[1] https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/gecko-t-linux-medium/workers/us-west-2/i-09f11ee7e67b88535
[2] https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/gecko-t-linux-medium/workers/us-west-2/i-0ed128d92620dd676
[3] https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/gecko-t-linux-medium/workers/us-west-2/i-0a6b06f9b6969a7ff
At least for the 3 machines referenced in comment 109 this is a result in a bad volume.  I imagine this is the root cause for most of these machines.

Errors are filled with things like:
end_request: I/O error, dev xvdb, sector 48349928

Comment 111

3 months ago
17 failures in 146 pushes (0.116 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 8
* mozilla-central: 5
* mozilla-beta: 3
* mozilla-inbound: 1

Platform breakdown:
* linux64: 6
* linux64-stylo-disabled: 3
* linux32-stylo-disabled: 2
* linux32: 2
* windows2012-32: 1
* osx-cross: 1
* linux64-devedition: 1
* linux32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-10-20&endday=2017-10-20&tree=all

Comment 112

3 months ago
63 failures in 864 pushes (0.073 failures/push) were associated with this bug in the last 7 days. 

This is the #17 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 32
* mozilla-inbound: 17
* mozilla-beta: 7
* mozilla-central: 4
* mozilla-esr52: 3

Platform breakdown:
* linux64: 32
* linux32: 15
* linux64-stylo-disabled: 8
* linux32-stylo-disabled: 4
* osx-cross: 1
* linux64-nightly: 1
* linux64-devedition: 1
* linux32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-10-16&endday=2017-10-22&tree=all

Comment 113

3 months ago
24 failures in 163 pushes (0.147 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-beta: 9
* autoland: 6
* mozilla-inbound: 5
* try: 2
* mozilla-central: 2

Platform breakdown:
* linux64: 11
* linux32-stylo-disabled: 5
* linux32: 5
* linux32-nightly: 2
* linux64-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-10-24&endday=2017-10-24&tree=all

Comment 114

3 months ago
49 failures in 912 pushes (0.054 failures/push) were associated with this bug in the last 7 days. 

This is the #28 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-beta: 15
* autoland: 10
* mozilla-inbound: 8
* mozilla-release: 6
* mozilla-central: 6
* try: 3
* mozilla-esr52: 1

Platform breakdown:
* linux64: 23
* linux32: 9
* linux32-stylo-disabled: 6
* linux64-nightly: 4
* linux32-nightly: 3
* linux64-stylo-disabled: 2
* osx-cross: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-10-23&endday=2017-10-29&tree=all

Comment 115

3 months ago
23 failures in 857 pushes (0.027 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 12
* mozilla-beta: 6
* try: 2
* mozilla-release: 2
* autoland: 1

Platform breakdown:
* linux64: 11
* linux64-stylo-disabled: 5
* linux64-qr: 3
* linux64-nightly: 1
* linux32-nightly: 1
* linux32-devedition: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-10-30&endday=2017-11-05&tree=all

Comment 116

2 months ago
1 failures in 849 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-11-06&endday=2017-11-12&tree=all

Comment 117

2 months ago
3 failures in 762 pushes (0.004 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2
* autoland: 1

Platform breakdown:
* linux32-stylo-disabled: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-11-13&endday=2017-11-19&tree=all

Comment 118

2 months ago
16 failures in 744 pushes (0.022 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 6
* autoland: 5
* mozilla-inbound: 4
* mozilla-beta: 1

Platform breakdown:
* linux64: 9
* osx-cross: 3
* linux32: 3
* linux64-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-11-20&endday=2017-11-26&tree=all

Comment 119

2 months ago
4 failures in 792 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 3
* mozilla-central: 1

Platform breakdown:
* osx-cross-noopt: 2
* osx-cross: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-11-27&endday=2017-12-03&tree=all

Comment 120

a month ago
12 failures in 889 pushes (0.013 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 4
* autoland: 4
* mozilla-inbound: 2
* try: 1
* mozilla-release: 1

Platform breakdown:
* linux64: 8
* linux32: 3
* linux32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-12-04&endday=2017-12-10&tree=all

Comment 121

a month ago
7 failures in 423 pushes (0.017 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 3
* mozilla-inbound: 2
* try: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 4
* osx-cross: 1
* linux32: 1
* android-4-3-armv7-api16: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-12-11&endday=2017-12-17&tree=all

Comment 122

26 days ago
15 failures in 590 pushes (0.025 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 5
* mozilla-release: 4
* mozilla-central: 4
* try: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 7
* android-4-0-armv7-api16: 3
* windows-mingw32-32: 1
* osx-cross: 1
* linux64-nightly: 1
* linux32-nightly: 1
* android-4-2-x86-old-id: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2017-12-18&endday=2017-12-24&tree=all
4 failures in 788 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 4

Platform breakdown:
* linux64: 3
* android-4-0-armv7-api16: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1302596&startday=2018-01-08&endday=2018-01-14&tree=all
You need to log in before you can comment on or make changes to this bug.