1447695 - Intermittent build failure IOError: [Errno 28] No space left on device

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=169425510&repo=mozilla-central

https://queue.taskcluster.net/v1/task/AhoCr1W9QfSaPNa3wdoNQA/runs/0/artifacts/public/logs/live_backing.log

[task 2018-03-21T13:08:29.019Z] Traceback (most recent call last):
[task 2018-03-21T13:08:29.019Z]   File "/usr/lib/python2.7/logging/__init__.py", line 883, in emit
[task 2018-03-21T13:08:29.024Z]     self.flush()
[task 2018-03-21T13:08:29.024Z]   File "/usr/lib/python2.7/logging/__init__.py", line 843, in flush
[task 2018-03-21T13:08:29.024Z]     self.stream.flush()
[task 2018-03-21T13:08:29.024Z] IOError: [Errno 28] No space left on device
[task 2018-03-21T13:08:29.024Z] Logged from file log.py, line 585
[task 2018-03-21T13:08:29.024Z] 13:08:29     INFO -      self._data.write(content)
[task 2018-03-21T13:08:29.024Z] Traceback (most recent call last):
[task 2018-03-21T13:08:29.024Z]   File "/usr/lib/python2.7/logging/__init__.py", line 883, in emit
[task 2018-03-21T13:08:29.024Z]     self.flush()
[task 2018-03-21T13:08:29.024Z]   File "/usr/lib/python2.7/logging/__init__.py", line 843, in flush
[task 2018-03-21T13:08:29.024Z]     self.stream.flush()
[task 2018-03-21T13:08:29.024Z] IOError: [Errno 28] No space left on device
[task 2018-03-21T13:08:29.024Z] Logged from file log.py, line 585
[task 2018-03-21T13:08:29.024Z] 13:08:29     INFO -  IOError: [Errno 28] No space left on device
[task 2018-03-21T13:08:29.024Z] Traceback (most recent call last):
[task 2018-03-21T13:08:29.024Z]   File "/usr/lib/python2.7/logging/__init__.py", line 883, in emit
[task 2018-03-21T13:08:29.024Z]     self.flush()
[task 2018-03-21T13:08:29.024Z]   File "/usr/lib/python2.7/logging/__init__.py", line 843, in flush
[task 2018-03-21T13:08:29.024Z]     self.stream.flush()
[task 2018-03-21T13:08:29.024Z] IOError: [Errno 28] No space left on device
[task 2018-03-21T13:08:29.024Z] Logged from file log.py, line 585
[task 2018-03-21T13:08:29.024Z] 13:08:29     INFO -  /builds/worker/workspace/build/src/testing/testsuite-targets.mk:158: recipe for target 'package-tests-gtest' failed
[task 2018-03-21T13:08:29.024Z] Traceback (most recent call last):
[task 2018-03-21T13:08:29.024Z]   File "/usr/lib/python2.7/logging/__init__.py", line 883, in emit
[task 2018-03-21T13:08:29.024Z]     self.flush()
[task 2018-03-21T13:08:29.024Z]   File "/usr/lib/python2.7/logging/__init__.py", line 843, in flush
[task 2018-03-21T13:08:29.024Z]     self.stream.flush()
[task 2018-03-21T13:08:29.024Z] IOError: [Errno 28] No space left on device
[task 2018-03-21T13:08:29.024Z] Logged from file log.py, line 585
[task 2018-03-21T13:08:29.024Z] 13:08:29     INFO -  make[2]: *** [package-tests-gtest] Error 1
[task 2018-03-21T13:08:29.024Z] Traceback (most recent call last):
[task 2018-03-21T13:08:29.025Z]   File "/usr/lib/python2.7/logging/__init__.py", line 883, in emit
[task 2018-03-21T13:08:29.025Z]     self.flush()
[task 2018-03-21T13:08:29.025Z]   File "/usr/lib/python2.7/logging/__init__.py", line 843, in flush
[task 2018-03-21T13:08:29.025Z]     self.stream.flush()
[task 2018-03-21T13:08:29.025Z] IOError: [Errno 28] No space left on device
[task 2018-03-21T13:08:29.025Z] Logged from file log.py, line 585
[task 2018-03-21T13:08:29.025Z] 13:08:29     INFO -  make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-21T13:08:29.025Z] Traceback (most recent call last):
[task 2018-03-21T13:08:29.025Z]   File "/usr/lib/python2.7/logging/__init__.py", line 883, in emit
[task 2018-03-21T13:08:29.025Z]     self.flush()
[task 2018-03-21T13:08:29.025Z]   File "/usr/lib/python2.7/logging/__init__.py", line 843, in flush
[task 2018-03-21T13:08:29.025Z]     self.stream.flush()
[task 2018-03-21T13:08:29.025Z] IOError: [Errno 28] No space left on device
[task 2018-03-21T13:08:29.025Z] Logged from file log.py, line 585
[task 2018-03-21T13:08:29.025Z] 13:08:29     INFO -  make[2]: *** Waiting for unfinished jobs....
[task 2018-03-21T13:08:29.025Z] Traceback (most recent call last):
[task 2018-03-21T13:08:29.025Z]   File "/usr/lib/python2.7/logging/__init__.py", line 883, in emit
[task 2018-03-21T13:08:29.025Z]     self.flush()
[task 2018-03-21T13:08:29.025Z]   File "/usr/lib/python2.7/logging/__init__.py", line 843, in flush
[task 2018-03-21T13:08:29.025Z]     self.stream.flush()
[task 2018-03-21T13:08:29.025Z] IOError: [Errno 28] No space left on device
[task 2018-03-21T13:08:29.025Z] Logged from file log.py, line 585
[task 2018-03-21T13:08:29.160Z] 13:08:29     INFO -  make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-03-21T13:08:29.160Z] Traceback (most recent call last):
[task 2018-03-21T13:08:29.160Z]   File "/usr/lib/python2.7/logging/__init__.py", line 883, in emit
[task 2018-03-21T13:08:29.160Z]     self.flush()
[task 2018-03-21T13:08:29.161Z]   File "/usr/lib/python2.7/logging/__init__.py", line 843, in flush
[task 2018-03-21T13:08:29.161Z]     self.stream.flush()
[task 2018-03-21T13:08:29.161Z] IOError: [Errno 28] No space left on device
[task 2018-03-21T13:08:29.161Z] Logged from file log.py, line 585
[task 2018-03-21T13:08:29.161Z] 13:08:29     INFO -  /builds/worker/workspace/build/src/obj-firefox/_virtualenv/bin/python -m mozbuild.action.test_archive  web-platform '/builds/worker/workspace/build/src/obj-firefox/dist//target.web-platform.tests.tar.gz'
[task 2018-03-21T13:08:29.161Z] Traceback (most recent call last):
[task 2018-03-21T13:08:29.161Z]   File "/usr/lib/python2.7/logging/__init__.py", line 883, in emit
[task 2018-03-21T13:08:29.161Z]     self.flush()
[task 2018-03-21T13:08:29.161Z]   File "/usr/lib/python2.7/logging/__init__.py", line 843, in flush
[task 2018-03-21T13:08:29.161Z]     self.stream.flush()
[task 2018-03-21T13:08:29.161Z] IOError: [Errno 28] No space left on device

Comment 1

[Intermittent Failures Robot]

1 failures in 822 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* linux64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2018-03-20&endday=2018-03-26&tree=trunk

Comment 2

[Intermittent Failures Robot]

2 failures in 718 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* linux64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2018-03-26&endday=2018-04-01&tree=trunk

Comment 3

[Intermittent Failures Robot]

1 failures in 792 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* linux64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2018-04-09&endday=2018-04-15&tree=trunk

Comment 4

[Intermittent Failures Robot]

2 failures in 782 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* linux64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2018-04-16&endday=2018-04-22&tree=trunk

Comment 5

[Intermittent Failures Robot]

2 failures in 660 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* linux64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2018-05-07&endday=2018-05-13&tree=trunk

Comment 6

[Intermittent Failures Robot]

1 failures in 632 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* linux64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2018-05-21&endday=2018-05-27&tree=trunk

Comment 7

[Intermittent Failures Robot]

1 failures in 669 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* linux64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2018-06-04&endday=2018-06-10&tree=trunk

Comment 8

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Comment 9

[Daniel Varga [:dvarga]]

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=225920386&repo=mozilla-inbound&lineNumber=109044

[task 2019-02-04T19:08:20.245Z] IOError: [Errno 28] No space left on device
[task 2019-02-04T19:08:20.245Z] Logged from file log.py, line 585
[task 2019-02-04T19:08:20.245Z] 19:08:20 INFO - Starting Mac pre-processing on file: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest/XUL
[task 2019-02-04T19:08:20.245Z] Traceback (most recent call last):
[task 2019-02-04T19:08:20.245Z] File "/usr/lib/python2.7/logging/init.py", line 883, in emit
[task 2019-02-04T19:08:20.245Z] self.flush()
[task 2019-02-04T19:08:20.245Z] File "/usr/lib/python2.7/logging/init.py", line 843, in flush
[task 2019-02-04T19:08:20.245Z] self.stream.flush()
[task 2019-02-04T19:08:20.246Z] IOError: [Errno 28] No space left on device
[task 2019-02-04T19:08:20.246Z] Logged from file log.py, line 585
[task 2019-02-04T19:08:20.246Z] Traceback (most recent call last):
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 883, in emit
[task 2019-02-04T19:08:20.246Z] self.flush()
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 843, in flush
[task 2019-02-04T19:08:20.246Z] self.stream.flush()
[task 2019-02-04T19:08:20.246Z] IOError: [Errno 28] No space left on device
[task 2019-02-04T19:08:20.246Z] Logged from file log.py, line 585
[task 2019-02-04T19:08:20.246Z] 19:08:20 INFO - Running Mac pre-processing on file: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest/XUL
[task 2019-02-04T19:08:20.246Z] Traceback (most recent call last):
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 883, in emit
[task 2019-02-04T19:08:20.246Z] self.flush()
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 843, in flush
[task 2019-02-04T19:08:20.246Z] self.stream.flush()
[task 2019-02-04T19:08:20.246Z] IOError: [Errno 28] No space left on device
[task 2019-02-04T19:08:20.246Z] Logged from file log.py, line 585
[task 2019-02-04T19:08:20.246Z] Traceback (most recent call last):
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 883, in emit
[task 2019-02-04T19:08:20.246Z] self.flush()
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 843, in flush
[task 2019-02-04T19:08:20.246Z] self.stream.flush()
[task 2019-02-04T19:08:20.246Z] IOError: [Errno 28] No space left on device
[task 2019-02-04T19:08:20.246Z] Logged from file log.py, line 585
[task 2019-02-04T19:08:20.246Z] 19:08:20 INFO - /builds/worker/workspace/build/src/build/macosx/llvm-dsymutil --arch=x86_64 /builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest/XUL
[task 2019-02-04T19:08:20.246Z] Traceback (most recent call last):
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 883, in emit
[task 2019-02-04T19:08:20.246Z] self.flush()
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 843, in flush
[task 2019-02-04T19:08:20.246Z] self.stream.flush()
[task 2019-02-04T19:08:20.246Z] IOError: [Errno 28] No space left on device
[task 2019-02-04T19:08:20.246Z] Logged from file log.py, line 585
[task 2019-02-04T19:08:20.246Z] Traceback (most recent call last):
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 883, in emit
[task 2019-02-04T19:08:20.246Z] self.flush()
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 843, in flush
[task 2019-02-04T19:08:20.246Z] self.stream.flush()
[task 2019-02-04T19:08:20.246Z] IOError: [Errno 28] No space left on device
[task 2019-02-04T19:08:20.246Z] Logged from file log.py, line 585
[task 2019-02-04T19:08:20.246Z] 19:08:20 INFO - warning: could not find referenced DIE
[task 2019-02-04T19:08:20.246Z] Traceback (most recent call last):
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 883, in emit
[task 2019-02-04T19:08:20.246Z] self.flush()
[task 2019-02-04T19:08:20.246Z] File "/usr/lib/python2.7/logging/init.py", line 843, in flush
[task 2019-02-04T19:08:20.246Z] self.stream.flush()
[task 2019-02-04T19:08:20.246Z] IOError: [Errno 28] No space left on device
[task 2019-02-04T19:08:20.246Z] Logged from file log.py, line 585
[task 2019-02-04T19:08:20.247Z] Traceback (most recent call last):
[task 2019-02-04T19:08:20.247Z] File "/usr/lib/python2.7/logging/init.py", line 883, in emit
[task 2019-02-04T19:08:20.247Z] self.flush()
[task 2019-02-04T19:08:20.247Z] File "/usr/lib/python2.7/logging/init.py", line 843, in flush
[task 2019-02-04T19:08:20.247Z] self.stream.flush()
[task 2019-02-04T19:08:20.247Z] IOError: [Errno 28] No space left on device
[task 2019-02-04T19:08:20.247Z] Logged from file log.py, line 585
[task 2019-02-04T19:08:20.247Z] Traceback (most recent call last):
[task 2019-02-04T19:08:20.247Z] File "/usr/lib/python2.7/logging/init.py", line 883, in emit
[task 2019-02-04T19:08:20.247Z] self.flush()
[task 2019-02-04T19:08:20.247Z] File "/usr/lib/python2.7/logging/init.py", line 843, in flush
[task 2019-02-04T19:08:20.247Z] self.stream.flush()
[task 2019-02-04T19:08:20.247Z] IOError: [Errno 28] No space left on device
[task 2019-02-04T19:08:20.247Z] Logged from file log.py, line 585

Comment 10

[Intermittent Failures Robot]

2 failures in 2658 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• mozilla-inbound: 1

Platform breakdown:

• osx-cross: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-02-04&endday=2019-02-10&tree=all

Comment 11

[Ryan VanderMeulen [:RyanVM]]

This seems to be happening more regularly lately with OSX builds, including today on mozilla-release. Can we please try to find someone to investigate?

Comment 12

[Chris AtLee [:catlee]]

Redirecting to fubar, keeper of the workers...

Ryan, is this happening more often than is being starred? I only see 2 occurrences of this in the past week according to comment #10 above.

Comment 13

[Kendall Libby [:fubar] (he/him)]

The latest log there is from a macOS cross-compile build, so it's a docker host in AWS; Wander, any ideas? It's not clear to me if it's an issue within the docker image or the underlying host..

Comment 14

[Wander Lairson Costa]

(In reply to Kendall Libby [:fubar] (he/him) from comment #13)

The latest log there is from a macOS cross-compile build, so it's a docker
host in AWS; Wander, any ideas? It's not clear to me if it's an issue within
the docker image or the underlying host..

I reran the task a kept measuring disk usage, it kept around 50-65%, and the worker capacity is set to one. No clue what is going on.

Comment 15

[Intermittent Failures Robot]

5 failures in 2832 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1
• mozilla-release: 2

Platform breakdown:

• osx-cross: 4
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-02-11&endday=2019-02-17&tree=all

Comment 16

[Cosmin Sabou [:CosminS]]

The following workers have been terminated because of this failure hitting today on beta:

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/gecko-3-b-linux/workers/us-west-1/i-096ad97909fa7e400

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/gecko-3-b-android/workers/us-east-1/i-0a675ab5a9aa6b75a

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/gecko-3-b-linux/workers/us-east-1/i-0d93ac233d6533558

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/gecko-3-b-linux/workers/us-west-2/i-0dd362434c4106361

Th push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&resultStatus=busted&searchStr=nightly&fromchange=bb9e3868a7bf0e1eeca0209d9943732ffb855958&tochange=bce0092f646c52d0402531a5b5a860905dfe7ad8&selectedJob=229014156

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=229014156&repo=mozilla-beta&lineNumber=109081

Could someone look into this issue? Thank you.

Comment 17

[Intermittent Failures Robot]

5 failures in 2602 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 4
• autoland: 1

Platform breakdown:

• osx-cross: 3
• android-4-2-x86: 1
• osx-cross-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-02-18&endday=2019-02-24&tree=all

Comment 18

[Wander Lairson Costa]

(In reply to Cosmin Sabou [:CosminS] from comment #16)

The following workers have been terminated because of this failure hitting
today on beta:

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-linux/workers/us-west-1/i-096ad97909fa7e400

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-android/workers/us-east-1/i-0a675ab5a9aa6b75a

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-linux/workers/us-east-1/i-0d93ac233d6533558

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-linux/workers/us-west-2/i-0dd362434c4106361

Th push:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-
beta&resultStatus=busted&searchStr=nightly&fromchange=bb9e3868a7bf0e1eeca0209
d9943732ffb855958&tochange=bce0092f646c52d0402531a5b5a860905dfe7ad8&selectedJ
ob=229014156

Failure log:
https://treeherder.mozilla.org/logviewer.html#/
jobs?job_id=229014156&repo=mozilla-beta&lineNumber=109081

Could someone look into this issue? Thank you.

I am investigating this.

Comment 19

[Wander Lairson Costa]

(In reply to Cosmin Sabou [:CosminS] from comment #16)

The following workers have been terminated because of this failure hitting
today on beta:

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-linux/workers/us-west-1/i-096ad97909fa7e400

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-android/workers/us-east-1/i-0a675ab5a9aa6b75a

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-linux/workers/us-east-1/i-0d93ac233d6533558

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-linux/workers/us-west-2/i-0dd362434c4106361

Th push:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-
beta&resultStatus=busted&searchStr=nightly&fromchange=bb9e3868a7bf0e1eeca0209
d9943732ffb855958&tochange=bce0092f646c52d0402531a5b5a860905dfe7ad8&selectedJ
ob=229014156

Failure log:
https://treeherder.mozilla.org/logviewer.html#/
jobs?job_id=229014156&repo=mozilla-beta&lineNumber=109081

Could someone look into this issue? Thank you.

I investigated this particular one [1]. When I click on "Inpect Task", it shows the task with status "Completed" and no sign of "No space left" in the logs. I am quite confused now :/

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&resultStatus=busted&searchStr=nightly&selectedJob=229014156&revision=bce0092f646c52d0402531a5b5a860905dfe7ad8
[2] https://tools.taskcluster.net/groups/TbzbfqqYSVu7yoVp-U7kLg/tasks/AIC4rVRkS32dTgsFePIjTA/runs/1/logs/public%2Flogs%2Flive.log

Comment 20

[Wander Lairson Costa]

(In reply to Wander Lairson Costa [:wcosta] from comment #19)

(In reply to Cosmin Sabou [:CosminS] from comment #16)

The following workers have been terminated because of this failure hitting
today on beta:

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-linux/workers/us-west-1/i-096ad97909fa7e400

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-android/workers/us-east-1/i-0a675ab5a9aa6b75a

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-linux/workers/us-east-1/i-0d93ac233d6533558

https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types/
gecko-3-b-linux/workers/us-west-2/i-0dd362434c4106361

Th push:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-
beta&resultStatus=busted&searchStr=nightly&fromchange=bb9e3868a7bf0e1eeca0209
d9943732ffb855958&tochange=bce0092f646c52d0402531a5b5a860905dfe7ad8&selectedJ
ob=229014156

Failure log:
https://treeherder.mozilla.org/logviewer.html#/
jobs?job_id=229014156&repo=mozilla-beta&lineNumber=109081

Could someone look into this issue? Thank you.

I investigated this particular one [1]. When I click on "Inpect Task", it
shows the task with status "Completed" and no sign of "No space left" in the
logs. I am quite confused now :/

[1]
https://treeherder.mozilla.org/#/jobs?repo=mozilla-
beta&resultStatus=busted&searchStr=nightly&selectedJob=229014156&revision=bce
0092f646c52d0402531a5b5a860905dfe7ad8
[2]
https://tools.taskcluster.net/groups/TbzbfqqYSVu7yoVp-U7kLg/tasks/
AIC4rVRkS32dTgsFePIjTA/runs/1/logs/public%2Flogs%2Flive.log

Update: shame on me, I was looking at Run 1 instead of Run 0.

Comment 22

[Johan Lorenzo [:jlorenzo]]

We've had the same problem, but perma-failing in bug 1530682. In that case, there's a difference between level 1 workers (which do work) and level 3 ones. More details in there.

Comment 23

[Johan Lorenzo [:jlorenzo]]

Upgrading importance because the situation has changed over the past 2 weeks. Please change it again, if I misinterpreted the thread.

Comment 24

[Intermittent Failures Robot]

6 failures in 2981 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 6

Platform breakdown:

• osx-cross-devedition: 1
• osx-cross: 4
• android-4-0-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-02-25&endday=2019-03-03&tree=all

Comment 25

[Intermittent Failures Robot]

3 failures in 2818 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2
• mozilla-central: 1

Platform breakdown:

• osx-cross-devedition: 1
• linux64: 1
• osx-cross-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-03-04&endday=2019-03-10&tree=all

Comment 26

[Intermittent Failures Robot]

2 failures in 2771 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 2

Platform breakdown:

• linux64: 1
• osx-cross: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-03-11&endday=2019-03-17&tree=all

Comment 27

[Intermittent Failures Robot]

1 failures in 3910 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• osx-cross-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-03-25&endday=2019-03-31&tree=all

Comment 28

[Intermittent Failures Robot]

1 failures in 3031 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• osx-cross: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-04-22&endday=2019-04-28&tree=all

Comment 29

[Intermittent Failures Robot]

1 failures in 4131 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• osx-cross-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-04-29&endday=2019-05-05&tree=all

Comment 30

[Intermittent Failures Robot]

3 failures in 4705 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3

Platform breakdown:

• osx-cross: 2
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-05-06&endday=2019-05-12&tree=all

Comment 31

[Intermittent Failures Robot]

2 failures in 4658 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• mozilla-release: 1

Platform breakdown:

• osx-cross: 1
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-05-13&endday=2019-05-19&tree=all

Comment 32

[Intermittent Failures Robot]

3 failures in 4730 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 1
• mozilla-inbound: 1

Platform breakdown:

• osx-cross-devedition: 1
• osx-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-05-20&endday=2019-05-26&tree=all

Comment 33

[Wander Lairson Costa]

Attachment: Bug 1447695: Dump disk usage on ENOSPC errors r=glandium

Bug 1447695 reports an intermittent "No space left" error
in CI. Given that the job runs in a docker container and it
is destroy as soon as the task finishes, it makes difficult
to diagnose the root cause of the bug. We then dump the disk
usage to logs so we can perform a post mortem analysis.

Comment 34

[Intermittent Failures Robot]

2 failures in 4337 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform breakdown:

• android-5-0-x86_64: 1
• osx-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-05-27&endday=2019-06-02&tree=all

Comment 35

[Intermittent Failures Robot]

1 failures in 5197 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• osx-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-06-03&endday=2019-06-09&tree=all

Comment 36

[Intermittent Failures Robot]

1 failures in 4753 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr60: 1

Platform breakdown:

• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-06-10&endday=2019-06-16&tree=all

Comment 37

[Intermittent Failures Robot]

1 failures in 4409 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• osx-cross: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-06-24&endday=2019-06-30&tree=all

Comment 38

[Intermittent Failures Robot]

3 failures in 4710 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-beta: 1
• mozilla-central: 1

Platform breakdown:

• android-5-0-x86_64: 1
• osx-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 39

[Intermittent Failures Robot]

2 failures in 4298 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• osx-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-07-15&endday=2019-07-21&tree=all

Comment 40

[Intermittent Failures Robot]

3 failures in 4608 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-beta: 1

Platform breakdown:

• linux64: 1
• osx-shippable: 1
• osx-cross-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-07-22&endday=2019-07-28&tree=all

Comment 41

[Intermittent Failures Robot]

39 failures in 657 pushes (0.059 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 24
• mozilla-beta: 10
• mozilla-inbound: 2
• mozilla-central: 3

Platform breakdown:

• windows-mingw32: 2
• android-4-0-armv7-api16: 5
• android-5-0-aarch64: 6
• android-4-2-x86: 6
• linux64-shippable: 2
• osx-shippable: 3
• android-4-0-geckoview-fat-aar: 2
• linux64: 6
• android-5-0-x86_64: 3
• osx-cross-noopt: 1
• osx-cross-devedition: 1
• linux64-noopt: 1
• osx-cross: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-07-31&endday=2019-07-31&tree=all

Comment 42

[Andrei Ciure[:aciure]]

There are 31 failures associated to this bug in the last 7 days. These are occurring on: android-4-0-armv7-api16, android-4-2-x86, android-5-0-aarch64, linux64, linux64-shippable, windows-mingw32, osx-shippable all builds.

Comment 43

[Intermittent Failures Robot]

19 failures in 628 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-esr68: 6
• mozilla-beta: 1

Platform breakdown:

• android-4-0-armv7-api16: 1
• osx-shippable: 2
• android-4-2-x86: 1
• android-5-0-aarch64: 2
• linux64: 4
• android-5-0-x86_64-beta: 2
• osx-cross-noopt: 1
• android-4-0-armv7-api16-beta: 1
• android-5-0-x86_64: 1
• linux32: 1
• windows-mingw32: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-08-01&endday=2019-08-01&tree=all

Comment 44

[Intermittent Failures Robot]

62 failures in 4439 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #25 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 38
• mozilla-beta: 12
• mozilla-inbound: 2
• mozilla-central: 3
• mozilla-esr68: 6
• mozilla-release: 1

Platform breakdown:

• linux64: 11
• android-5-0-aarch64: 9
• windows-mingw32: 5
• android-4-0-armv7-api16: 7
• android-4-2-x86: 7
• osx-shippable: 5
• linux64-shippable: 2
• android-5-0-x86_64: 5
• osx-cross-noopt: 2
• osx-cross-devedition: 1
• android-4-0-geckoview-fat-aar: 2
• linux64-noopt: 1
• osx-cross: 1
• android-5-0-x86_64-beta: 2
• android-4-0-armv7-api16-beta: 1
• linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-07-29&endday=2019-08-04&tree=all

Comment 45

[Intermittent Failures Robot]

8 failures in 4666 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 1
• mozilla-esr68: 1

Platform breakdown:

• android-5-0-x86_64: 4
• android-5-0-x86_64-beta: 1
• osx-shippable: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 46

[Intermittent Failures Robot]

3 failures in 4839 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 1

Platform breakdown:

• osx-cross: 1
• osx-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-08-12&endday=2019-08-18&tree=all

Comment 49

[Intermittent Failures Robot]

5 failures in 5073 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 2
• mozilla-inbound: 1
• mozilla-central: 1

Platform breakdown:

• android-4-0-armv7-api16: 1
• osx-shippable: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-08-19&endday=2019-08-25&tree=all

Comment 50

[Intermittent Failures Robot]

2 failures in 5047 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• osx-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-08-26&endday=2019-09-01&tree=all

Comment 51

[Intermittent Failures Robot]

6 failures in 4408 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6

Platform breakdown:

• osx-shippable: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-09-02&endday=2019-09-08&tree=all

Comment 52

[Intermittent Failures Robot]

1 failures in 4637 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-09-09&endday=2019-09-15&tree=all

Comment 53

[Intermittent Failures Robot]

2 failures in 4350 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• android-5-0-x86_64: 1
• osx-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-09-16&endday=2019-09-22&tree=all

Comment 55

[Intermittent Failures Robot]

7 failures in 4530 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1
• autoland: 4
• mozilla-central: 1
• mozilla-beta: 1

Platform breakdown:

• osx-shippable: 5
• osx-cross: 1
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-09-23&endday=2019-09-29&tree=all

Comment 56

[Intermittent Failures Robot]

3 failures in 3801 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform breakdown:

• linux64: 2
• android-5-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-10-07&endday=2019-10-13&tree=all

Comment 57

[Intermittent Failures Robot]

3 failures in 3964 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform breakdown:

• linux64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-10-14&endday=2019-10-20&tree=all

Comment 58

[Intermittent Failures Robot]

5 failures in 4182 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 3

Platform breakdown:

• osx-cross: 2
• osx-shippable: 1
• android-5-0-x86_64: 1
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-10-21&endday=2019-10-27&tree=all

Comment 59

[Intermittent Failures Robot]

2 failures in 4314 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• osx-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-10-28&endday=2019-11-03&tree=all

Comment 60

[Intermittent Failures Robot]

3 failures in 4850 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-esr68: 1

Platform breakdown:

• linux64: 2
• osx-cross: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 61

[Intermittent Failures Robot]

6 failures in 4207 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• autoland: 5

Platform breakdown:

• android-4-0-armv7-api16: 1
• linux64: 2
• osx-shippable: 2
• android-5-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-11-11&endday=2019-11-17&tree=all

Comment 62

[Intermittent Failures Robot]

1 failures in 5444 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-11-18&endday=2019-11-24&tree=all

Comment 63

[Intermittent Failures Robot]

4 failures in 4331 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform breakdown:

• osx-shippable: 3
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-11-25&endday=2019-12-01&tree=all

Comment 64

[Noemi Erli[:noemi_erli]]

This started to fail quite frequently today:
https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&searchStr=c93018f1e173e285a117f34a528343cb8536c34f&fromchange=d58db9c67aae38c26425a0a70de1a5df1a64f721&tochange=d07675191e795e0d3b41535d6e0a5b01fc47c702&selectedJob=279762373

Comment 65

[Wander Lairson Costa]

Attachment: Bug 1447695: Increase volume threshold for build worker r=dustin

Comment 66

[Steve Fink [:sfink] [:s:]]

(In reply to Noemi Erli[:noemi_erli] from comment #64)

This started to fail quite frequently today:
https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&searchStr=c93018f1e173e285a117f34a528343cb8536c34f&fromchange=d58db9c67aae38c26425a0a70de1a5df1a64f721&tochange=d07675191e795e0d3b41535d6e0a5b01fc47c702&selectedJob=279762373

This is a different bug. It could have been bug 1570522, but that one got duped here because the first log posted there happened to get the same symptom (error compiling gdb-tests) from a different cause (out of space). The above build, and the majority of the ones that are now getting classified as this bug, are different and do not appear to be related to "No space left on device." The relevant portion of the log is:

[task 2019-12-05T12:00:13.605Z] 12:00:13     INFO -  /builds/worker/workspace/build/src/obj-firefox/x86_64-linux-android/release/libjsrust.a(wrappers.o): In function `AnnotateMozCrashReason':
[task 2019-12-05T12:00:13.606Z] 12:00:13     INFO -  /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/Assertions.h:42: undefined reference to `__asan_report_store8'
[task 2019-12-05T12:00:13.606Z] 12:00:13     INFO -  /builds/worker/workspace/build/src/obj-firefox/x86_64-linux-android/release/libjsrust.a(wrappers.o): In function `MOZ_Crash':
[task 2019-12-05T12:00:13.607Z] 12:00:13     INFO -  /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/Assertions.h:332: undefined reference to `__asan_report_store4'
[task 2019-12-05T12:00:13.607Z] 12:00:13     INFO -  /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/Assertions.h:332: undefined reference to `__asan_handle_no_return'
[task 2019-12-05T12:00:13.607Z] 12:00:13     INFO -  /builds/worker/workspace/build/src/obj-firefox/x86_64-linux-android/release/libjsrust.a(wrappers.o): In function `asan.module_ctor':
[task 2019-12-05T12:00:13.608Z] 12:00:13     INFO -  wrappers.cpp:(.text.asan.module_ctor+0x2): undefined reference to `__asan_init'
[task 2019-12-05T12:00:13.608Z] 12:00:13     INFO -  wrappers.cpp:(.text.asan.module_ctor+0x7): undefined reference to `__asan_version_mismatch_check_v8'
[task 2019-12-05T12:00:13.608Z] 12:00:13     INFO -  clang-9: error: linker command failed with exit code 1 (use -v to see invocation)
[task 2019-12-05T12:00:13.609Z] 12:00:13     INFO -  /builds/worker/workspace/build/src/config/rules.mk:522: recipe for target '../../../dist/bin/gdb-tests' failed
[task 2019-12-05T12:00:13.609Z] 12:00:13    ERROR -  make[4]: *** [../../../dist/bin/gdb-tests] Error 1

(edit: removed first line of pasted log with an LLVM gold plugin:...IDs have conflicting values message from log snippet above, since it also happens in successful builds.)

Comment 67

[Steve Fink [:sfink] [:s:]]

I filed bug 1601704 for this problem. Hopefully it will start to get the classifications now.

Comment 68

[Intermittent Failures Robot]

29 failures in 1048 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 29

Platform breakdown:

• android-5-0-x86_64: 29

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-12-05&endday=2019-12-05&tree=all

Comment 69

[Intermittent Failures Robot]

36 failures in 5144 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #29 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 36

Platform breakdown:

• osx-shippable: 2
• linux64: 5
• android-5-0-x86_64: 29

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-12-02&endday=2019-12-08&tree=all

Comment 70

[Intermittent Failures Robot]

1 failures in 4582 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• symbols: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-12-16&endday=2019-12-22&tree=all

Comment 71

[Intermittent Failures Robot]

1 failures in 1862 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• symbols: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2019-12-30&endday=2020-01-05&tree=all

Comment 72

[Intermittent Failures Robot]

2 failures in 4472 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• symbols: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2020-01-06&endday=2020-01-12&tree=all

Comment 73

[Intermittent Failures Robot]

3 failures in 3773 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform breakdown:

• symbols: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447695&startday=2020-01-13&endday=2020-01-19&tree=all

Comment 74

[Wander Lairson Costa]

No failure in 2 months, closing.