1457452 - Intermittent Error copying results /home/cltbld/workspace/build/local.json to upload dir /home/cltbld/workspace/build/blobber_upload_dir/perfherder-data.json

Status: RESOLVED FIXED

(Testing :: Talos, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=175914124&repo=autoland

https://queue.taskcluster.net/v1/task/Z08qwmiOSG27N9rm4EV57Q/runs/0/artifacts/public/logs/live_backing.log

00:11:35     INFO -  Test is using firstNonBlankPaint, browser pref will be turned on
00:11:35     INFO -  Test is using firstNonBlankPaint, browser pref will be turned on
00:11:35     INFO -  Test is using firstNonBlankPaint, browser pref will be turned on
00:11:35     INFO -  mozversion application_buildid: 20180427062038
00:11:35     INFO -  mozversion application_changeset: 8229360171458d5e2a36fd562e9ba1da6af9b6ba
00:11:35     INFO -  mozversion application_display_name: Firefox Nightly
00:11:35     INFO -  mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
00:11:35     INFO -  mozversion application_name: Firefox
00:11:35     INFO -  mozversion application_remotingname: firefox
00:11:35     INFO -  mozversion application_repository: https://hg.mozilla.org/integration/autoland
00:11:35     INFO -  mozversion application_vendor: Mozilla
00:11:35     INFO -  mozversion application_version: 61.0a1
00:11:35     INFO -  mozversion platform_buildid: 20180427062038
00:11:35     INFO -  mozversion platform_changeset: 8229360171458d5e2a36fd562e9ba1da6af9b6ba
00:11:35     INFO -  mozversion platform_repository: https://hg.mozilla.org/integration/autoland
00:11:35     INFO -  mozversion platform_version: 61.0a1
00:11:35     INFO -  using testdate: 1524813095
00:11:35     INFO -  actual date: 1524813095
00:11:35     INFO -  starting webserver on 'localhost:46359'
00:11:35     INFO -  Starting mitmproxy playback using env path: /home/cltbld/workspace/build/application/firefox
00:11:35     INFO -  Starting mitmproxy playback using command: /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmdump -k -s /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/alternate-server-replay.py /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-google.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-youtube.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-amazon.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-facebook.mp
00:11:35     INFO -  Error starting proxy server: OSError(98, 'Address already in use')
00:11:45     INFO -  Aborting: mitmproxy playback process failed to start, poll returned: 1
00:11:45     INFO - Return code: 0
00:11:45 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
00:11:45 CRITICAL - Error copying results /home/cltbld/workspace/build/local.json to upload dir /home/cltbld/workspace/build/blobber_upload_dir/perfherder-data.json
00:11:45     INFO - Running post-action listener: _package_coverage_data
00:11:45     INFO - Running post-action listener: _resource_record_post_action
00:11:45     INFO - [mozharness: 2018-04-27 07:11:45.648690Z] Finished run-tests step (failed)
00:11:45    FATAL - Uncaught exception: Traceback (most recent call last):
00:11:45    FATAL -   File "/home/cltbld/workspace/mozharness/mozharness/base/script.py", line 2076, in run
00:11:45    FATAL -     self.run_action(action)
00:11:45    FATAL -   File "/home/cltbld/workspace/mozharness/mozharness/base/script.py", line 2015, in run_action
00:11:45    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
00:11:45    FATAL -   File "/home/cltbld/workspace/mozharness/mozharness/base/script.py", line 1955, in _possibly_run_method
00:11:45    FATAL -     return getattr(self, method_name)()
00:11:45    FATAL -   File "/home/cltbld/workspace/mozharness/mozharness/mozilla/testing/talos.py", line 768, in run_tests
00:11:45    FATAL -     self._artifact_perf_data(dest)
00:11:45    FATAL -   File "/home/cltbld/workspace/mozharness/mozharness/mozilla/testing/talos.py", line 663, in _artifact_perf_data
00:11:45    FATAL -     parser.update_worst_log_and_tbpl_levels(CRITICAL, TBPL_FAILURE)
00:11:45    FATAL - NameError: global name 'parser' is not defined
00:11:45    FATAL - Running post_fatal callback...
00:11:45    FATAL - Exiting -1
00:11:45     INFO - Running post-run listener: _resource_record_post_run
00:11:45     INFO - Total resource usage - Wall time: 23s; CPU: 8.0%; Read bytes: 0; Write bytes: 282296320; Read time: 0; Write time: 38552
00:11:45     INFO - TinderboxPrint: CPU usage<br/>7.6%
00:11:45     INFO - TinderboxPrint: I/O read bytes / time<br/>0 / 0
00:11:45     INFO - TinderboxPrint: I/O write bytes / time<br/>282,296,320 / 38,552
00:11:45     INFO - TinderboxPrint: CPU idle<br/>169.8 (92.4%)
00:11:45     INFO - TinderboxPrint: CPU user<br/>13.4 (7.3%)
00:11:45     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
00:11:45     INFO - install - Wall time: 11s; CPU: 13.0%; Read bytes: 0; Write bytes: 275693568; Read time: 0; Write time: 37384
00:11:45     INFO - setup-mitmproxy - Wall time: 1s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
00:11:45     INFO - run-tests - Wall time: 11s; CPU: 1.0%; Read bytes: 0; Write bytes: 6602752; Read time: 0; Write time: 1168
00:11:45     INFO - Running post-run listener: _upload_blobber_files
00:11:45  WARNING - Blob upload gear skipped. Missing cmdline options.
00:11:45     INFO - Running post-run listener: copy_logs_to_upload_dir
00:11:45     INFO - Copying logs to upload dir...
00:11:45     INFO - mkdir: /home/cltbld/workspace/build/upload/logs
cleanup
+ cleanup
+ local rv=255
+ [[ -s /home/cltbld/.xsession-errors ]]
+ cp /home/cltbld/.xsession-errors /home/cltbld/artifacts/public/xsession-errors.log
+ false
+ exit 255

Comment 1

[Eliza Balazs [:ebalazs_]]

Retriggered this and found that the failure rate is 10%.

The push that started this seems to be:
https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=opt-talos-h2-e10s&tochange=64583d3e0f6fd6ec9049bea1039948ce871f4209&fromchange=3a10be9bb6bd3b90594b4afe2896f9279f339e9a&selectedJob=176193875

Comment 2

[Intermittent Failures Robot]

20 failures in 780 pushes (0.026 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 8
* mozilla-inbound: 7
* mozilla-central: 4
* mozilla-beta: 1

Platform breakdown:
* linux64: 19
* linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-04-23&endday=2018-04-29&tree=trunk

Comment 3

[Robert Wood [:rwood]]

Wow, never seen anything like this before.  I don't get how that change could break mitmproxy:

04:02:42     INFO -  Starting mitmproxy playback using command: /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmdump -k -s /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/alternate-server-replay.py /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-google.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-youtube.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-amazon.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-facebook.mp
04:02:43     INFO -  Error starting proxy server: OSError(98, 'Address already in use')

It almost looks as though mitmproxy was still running from a previous test - like the machine hadn't been cleaned or reset after a previous run or something... or something else has been introduced that is grabbing the mitmproxy port or something.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

this is linux specific and it looks to be related to a merge from inbound:
https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=f551d1f94952bd23073a75244fba738e6e3080c9

I did some retriggers on mozilla-inbound in case this helps us figure it out:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=47db02b5cdc9&tochange=1472f261db23&filter-searchStr=linux%20h2%20talos&selectedJob=175784623

in response to rwood, possibly there is something that changed in our server creation code or cleanup code- maybe this is infrastructure related?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

oh, these all seem to be  t-linux64-ms-239:
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-linux-talos/workers/mdc1/t-linux64-ms-239

seems like a pattern.

:markco, can you take a look at this machine? I have asked CIduty to remove it from the working pool.

Comment 6

[Mark Cornmesser [:markco]]

Dhouse is more familiar with the linux nodes. Passing the NI to hum.

Comment 7

[:dhouse]

I checked over the system logs and I don't see any failures or warning around the mitmproxy startups. I have rebooted and un-quarantined it to see if the next job succeeds.

Comment 8

[Phil Ringnalda (:philor)]

I'd describe the pattern of https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1457452&entireHistory=true&tree=all as less "it's all about 239" and more "something breaks them and then they stay broken until something." Given how 060 stayed broken from the evening of the 27th to midday on the 29th, if the until something is a reboot, uh, do we really only reboot them every two days? And I'm really missing the way buildbot would let us see history for an arbitrary number of jobs, since the something breaks them could quite easily be what it has been in the past for similar things, "John does a try push with his patch for that one bug, and whatever machine does the run of the foo suite is then broken on every run of talos h2 and tp6 it does until it is next rebooted."

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

:coop, is there a similar tool in the new land of taskcluster that allows us to see a larger history of jobs run/status for a given machine?  Please see :philor's concern in comment 8

Comment 10

[Intermittent Failures Robot]

44 failures in 551 pushes (0.08 failures/push) were associated with this bug in the last 7 days. 

This is the #32 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 24
* try: 6
* autoland: 6
* mozilla-central: 4
* mozilla-beta: 4

Platform breakdown:
* linux64: 42
* linux64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-04-30&endday=2018-05-06&tree=trunk

Comment 11

[Chris Cooper [:coop] (he/him)]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #9)
> :coop, is there a similar tool in the new land of taskcluster that allows us
> to see a larger history of jobs run/status for a given machine?  Please see
> :philor's concern in comment 8

Just the worker type explorer that you mentioned already in comment #5.

If you need a deeper history there for hardware workers in particular, we could consider adding more and paginating as we do elsewhere in the tool already.

Let's get a separate tools bug on file if you do want this.

Comment 12

[Intermittent Failures Robot]

1 failures in 660 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-05-07&endday=2018-05-13&tree=trunk

Comment 13

[Intermittent Failures Robot]

13 failures in 639 pushes (0.02 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 7
* mozilla-inbound: 4
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 10
* linux64-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-05-14&endday=2018-05-20&tree=trunk

Comment 14

[Intermittent Failures Robot]

15 failures in 632 pushes (0.024 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 8
* mozilla-beta: 3
* try: 2
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 9
* linux64-qr: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-05-21&endday=2018-05-27&tree=trunk

Comment 15

[Intermittent Failures Robot]

14 failures in 655 pushes (0.021 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 6
* try: 4
* mozilla-inbound: 2
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 10
* linux64-qr: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-05-28&endday=2018-06-03&tree=trunk

Comment 16

[Intermittent Failures Robot]

14 failures in 669 pushes (0.021 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 5
* try: 4
* mozilla-inbound: 3
* mozilla-central: 2

Platform breakdown:
* linux64: 9
* linux64-qr: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-06-04&endday=2018-06-10&tree=trunk

Comment 17

[Intermittent Failures Robot]

7 failures in 521 pushes (0.013 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 4
* try: 1
* mozilla-inbound: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 5
* linux64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-06-18&endday=2018-06-24&tree=trunk

Comment 18

[Intermittent Failures Robot]

8 failures in 718 pushes (0.011 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 3
* try: 2
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* linux64: 6
* linux64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-06-25&endday=2018-07-01&tree=trunk

Comment 19

[Bogdan Tara[:bogdan_tara | bogdant]]

In the last 7 days, there have been 30 failures on Linux 64 pgo and opt.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=186948997&repo=autoland&lineNumber=834

18:50:22     INFO -  Starting mitmproxy playback using env path: /home/cltbld/workspace/build/application/firefox
18:50:22     INFO -  Starting mitmproxy playback using command: /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmdump -k -s /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/alternate-server-replay.py /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-google.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-youtube.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-amazon.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-facebook.mp
18:50:23     INFO -  Error starting proxy server: OSError(98, 'Address already in use')
18:50:32     INFO -  Aborting: mitmproxy playback process failed to start, poll returned: 1
18:50:33     INFO - Return code: 0
18:50:33 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
18:50:33 CRITICAL - Error copying results /home/cltbld/workspace/build/local.json to upload dir /home/cltbld/workspace/build/blobber_upload_dir/perfherder-data.json
18:50:33 CRITICAL - # TBPL FAILURE #

Comment 20

[Intermittent Failures Robot]

35 failures in 622 pushes (0.056 failures/push) were associated with this bug in the last 7 days. 

This is the #36 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 13
* mozilla-inbound: 9
* mozilla-central: 7
* try: 3
* mozilla-beta: 3

Platform breakdown:
* linux64: 22
* linux64-qr: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-07-02&endday=2018-07-08&tree=trunk

Comment 21

[Intermittent Failures Robot]

14 failures in 623 pushes (0.022 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 5
* autoland: 5
* try: 3
* mozilla-beta: 1

Platform breakdown:
* linux64: 13
* linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-07-09&endday=2018-07-15&tree=trunk

Comment 22

[Intermittent Failures Robot]

17 failures in 653 pushes (0.026 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 6
* autoland: 5
* try: 3
* mozilla-central: 3

Platform breakdown:
* linux64: 12
* linux64-qr: 4
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-07-16&endday=2018-07-22&tree=trunk

Comment 23

[Intermittent Failures Robot]

45 failures in 138 pushes (0.326 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 21
* mozilla-central: 14
* mozilla-inbound: 9
* mozilla-beta: 1

Platform breakdown:
* linux64: 28
* linux64-qr: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-07-25&endday=2018-07-25&tree=trunk

Comment 24

[Narcis Beleuzu [:NarcisB]]

In the last 7 days, there have been 63 failures on Linux 64 and Linux64-qr pgo and opt.

Recent failure log:
https://treeherder.mozilla.org/logviewer.html#?job_id=190184340&repo=autoland&lineNumber=835

22:59:56     INFO -  starting webserver on 'localhost:57678'
22:59:56     INFO -  Starting mitmproxy playback using env path: /home/cltbld/workspace/build/application/firefox
22:59:56     INFO -  Starting mitmproxy playback using command: /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmdump -k -s /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/alternate-server-replay.py /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-google.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-youtube.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-amazon.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-facebook.mp
22:59:57     INFO -  Error starting proxy server: OSError(98, 'Address already in use')
23:00:06     INFO -  Aborting: mitmproxy playback process failed to start, poll returned: 1
23:00:07     INFO - Return code: 0
23:00:07 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
23:00:07 CRITICAL - Error copying results /home/cltbld/workspace/build/local.json to upload dir /home/cltbld/workspace/build/blobber_upload_dir/perfherder-data.json
23:00:07 CRITICAL - # TBPL FAILURE #
23:00:07  WARNING - setting return code to 2

rwood, can you please take a look?

Comment 25

[Intermittent Failures Robot]

16 failures in 142 pushes (0.113 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 7
* mozilla-inbound: 5
* mozilla-beta: 3
* try: 1

Platform breakdown:
* linux64: 10
* linux64-qr: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-07-26&endday=2018-07-26&tree=trunk

Comment 26

[Intermittent Failures Robot]

70 failures in 760 pushes (0.092 failures/push) were associated with this bug in the last 7 days. 

This is the #21 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 32
* mozilla-inbound: 16
* mozilla-central: 15
* mozilla-beta: 6
* try: 1

Platform breakdown:
* linux64: 45
* linux64-qr: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-07-23&endday=2018-07-29&tree=trunk

Comment 27

[Intermittent Failures Robot]

26 failures in 549 pushes (0.047 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 11
* autoland: 8
* mozilla-central: 3
* mozilla-beta: 3
* try: 1

Platform breakdown:
* linux64-qr: 9
* linux64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-09&endday=2018-08-09&tree=all

Comment 28

[Intermittent Failures Robot]

20 failures in 502 pushes (0.04 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 7
* mozilla-central: 3
* mozilla-beta: 2
* try: 1
* autoland: 7

Platform breakdown:
* linux64: 15
* linux64-qr: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-10&endday=2018-08-10&tree=all

Comment 29

[Intermittent Failures Robot]

20 failures in 98 pushes (0.204 failures/push) were associated with this bug yesterday.

Repository breakdown:
* autoland: 4
* mozilla-central: 5
* mozilla-beta: 2
* try: 1
* mozilla-inbound: 8

Platform breakdown:
* linux64-qr: 5
* linux64: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-12&endday=2018-08-12&tree=all

Comment 30

[Eliza Balazs [:ebalazs_]]

There are 70 failures in the last 7 days.
They occur on linux64 (opt) and linux64-qr (opt and pgo).
Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=193589337&repo=autoland&lineNumber=835

Comment 31

[Intermittent Failures Robot]

75 failures in 2579 pushes (0.029 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 24
* mozilla-beta: 8
* try: 3
* mozilla-central: 12
* mozilla-inbound: 28

Platform breakdown:
* linux64-qr: 21
* linux64: 54

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-06&endday=2018-08-12&tree=all

Comment 32

[Intermittent Failures Robot]

23 failures in 594 pushes (0.039 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-inbound: 8
* autoland: 8
* mozilla-beta: 1
* mozilla-central: 3
* try: 3

Platform breakdown:
* linux64-qr: 7
* linux64: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-14&endday=2018-08-14&tree=all

Comment 33

[Intermittent Failures Robot]

24 failures in 494 pushes (0.049 failures/push) were associated with this bug yesterday.

Repository breakdown:
* try: 1
* mozilla-central: 4
* autoland: 4
* mozilla-beta: 3
* mozilla-inbound: 12

Platform breakdown:
* linux64-qr: 12
* linux64: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-15&endday=2018-08-15&tree=all

Comment 34

[Intermittent Failures Robot]

41 failures in 527 pushes (0.078 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 3
* autoland: 15
* mozilla-beta: 4
* mozilla-inbound: 19

Platform breakdown:
* linux64-qr: 15
* linux64: 26

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-16&endday=2018-08-16&tree=all

Comment 35

[Intermittent Failures Robot]

18 failures in 115 pushes (0.157 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 4
* autoland: 1
* mozilla-inbound: 13

Platform breakdown:
* linux64-qr: 6
* linux64: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-18&endday=2018-08-18&tree=all

Comment 36

[Intermittent Failures Robot]

183 failures in 2672 pushes (0.068 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 29
* try: 6
* mozilla-beta: 12
* autoland: 48
* mozilla-inbound: 88

Platform breakdown:
* linux64: 112
* linux64-qr: 71

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-13&endday=2018-08-19&tree=all

Comment 37

[Natalia Csoregi [:nataliaCs]]

Update: there have been 172 failures in the last 7 days on linux x64 and Linux x64 QR.

Comment 38

[Joel Maher ( :jmaher ) (UTC -8)]

raptor is running tp6 as tier-2 on mozilla-central still.  We were delayed the last 3 weeks working on Android, so I think we are looking at end of September before we make raptor tier-1; it is possible we could do this sooner and get tp6 for firefox only as tier-1 and then turn off tp6 talos.  It is worth considering.

Comment 39

[Intermittent Failures Robot]

17 failures in 492 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:
* autoland: 7
* mozilla-inbound: 7
* try: 1
* mozilla-central: 2

Platform breakdown:
* linux64-qr: 6
* linux64: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-21&endday=2018-08-21&tree=all

Comment 40

[Intermittent Failures Robot]

68 failures in 2568 pushes (0.026 failures/push) were associated with this bug in the last 7 days.

This is the #26 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 15
* mozilla-beta: 5
* try: 1
* mozilla-inbound: 27
* autoland: 20

Platform breakdown:
* linux64: 40
* linux64-qr: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-20&endday=2018-08-26&tree=all

Comment 41

[Intermittent Failures Robot]

63 failures in 537 pushes (0.117 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 22
* autoland: 17
* mozilla-central: 19
* try: 5

Platform breakdown:
* linux64-qr: 17
* linux64: 46

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-29&endday=2018-08-29&tree=all

Comment 42

[Stefan Hindli [:stefan_hindli]]

Over the last 7 days there are 66 failures on this bug. These happen on linux64 and linux64-qr 
This is the most recent log example: https://treeherder.mozilla.org/logviewer.html#?job_id=196589055&repo=autoland&lineNumber=837

23:19:03 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
23:19:03 CRITICAL - Error copying results /home/cltbld/workspace/build/local.json to upload dir /home/cltbld/workspace/build/blobber_upload_dir/perfherder-data.json
23:19:03 CRITICAL - # TBPL FAILURE #

Comment 43

[Intermittent Failures Robot]

40 failures in 538 pushes (0.074 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 1
* autoland: 14
* mozilla-inbound: 16
* mozilla-central: 9

Platform breakdown:
* linux64-qr: 13
* linux64: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-30&endday=2018-08-30&tree=all

Comment 44

[Intermittent Failures Robot]

119 failures in 2545 pushes (0.047 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 29
* mozilla-beta: 1
* autoland: 39
* try: 6
* mozilla-inbound: 44

Platform breakdown:
* linux64: 85
* linux64-qr: 34

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-08-27&endday=2018-09-02&tree=all

Comment 45

[Intermittent Failures Robot]

32 failures in 2667 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 6
* try: 2
* mozilla-inbound: 11
* autoland: 13

Platform breakdown:
* linux64: 25
* linux64-qr: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-09-03&endday=2018-09-09&tree=all

Comment 46

[Tiberius Oros[:tiberius_oros]]

There have been 36 failures in the last week on linux64 and linux64-qr platforms.
Affected builds: opt & pgo
Summary: Intermittent Error copying results /home/cltbld/workspace/build/local.json to upload dir /home/cltbld/workspace/build/blobber_upload_dir/perfherder-data.json

Recent relevant log file:
https://treeherder.mozilla.org/logviewer.html#?job_id=198502576&repo=mozilla-inbound&lineNumber=831

Comment 47

[Intermittent Failures Robot]

15 failures in 2684 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 7
* mozilla-inbound: 4
* try: 1
* mozilla-central: 3

Platform breakdown:
* linux64-qr: 6
* linux64: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-09-10&endday=2018-09-16&tree=all

Comment 48

[Intermittent Failures Robot]

22 failures in 571 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 1
* autoland: 10
* mozilla-central: 2
* mozilla-inbound: 9

Platform breakdown:
* linux64: 13
* linux64-qr: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-09-18&endday=2018-09-18&tree=all

Comment 49

[Intermittent Failures Robot]

43 failures in 543 pushes (0.079 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-beta: 4
* autoland: 8
* mozilla-inbound: 15
* try: 8
* mozilla-central: 8

Platform breakdown:
* linux64-qr: 15
* linux64: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-09-19&endday=2018-09-19&tree=all

Comment 50

[Dorel Luca [:dluca]]

This bug failed 60 times in the last 7 days. Failures occur on linux64 and linux64-qr on pgo and opt build types.

Recent log:
https://treeherder.mozilla.org/logviewer.html#?job_id=200328196&repo=autoland&lineNumber=833

  INFO - Calling ['/home/cltbld/workspace/build/venv/bin/python', '/home/cltbld/workspace/build/tests/talos/talos/run_tests.py', '--title', 't-linux64-ms-278', '--symbolsPath', 'https://queue.taskcluster.net/v1/task/UIZK47OCTJmnk14GEVkdFw/artifacts/public/build/target.crashreporter-symbols.zip', '--mitmproxy', u'mitmproxy-recording-google.mp mitmproxy-recording-youtube.mp mitmproxy-recording-amazon.mp mitmproxy-recording-facebook.mp', '--suite', 'tp6-e10s', '--executablePath', '/home/cltbld/workspace/build/application/firefox/firefox', '--mitmdumpPath', '/home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmdump', '--log-tbpl-level=debug', '--log-errorsummary=/home/cltbld/workspace/build/blobber_upload_dir/tp6-e10s_errorsummary.log', '--log-raw=/home/cltbld/workspace/build/blobber_upload_dir/tp6-e10s_raw.log'] with output_timeout 3600
18:04:22     INFO -  Test is using firstNonBlankPaint, browser pref will be turned on
18:04:22     INFO -  Test is using firstNonBlankPaint, browser pref will be turned on
18:04:22     INFO -  Test is using firstNonBlankPaint, browser pref will be turned on
18:04:22     INFO -  mozversion application_buildid: 20180919232655
18:04:22     INFO -  mozversion application_changeset: 8e1af99ab1adb64ba631d81f67942dc9b6c48df5
18:04:22     INFO -  mozversion application_display_name: Firefox Nightly
18:04:22     INFO -  mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
18:04:22     INFO -  mozversion application_name: Firefox
18:04:22     INFO -  mozversion application_remotingname: firefox
18:04:22     INFO -  mozversion application_repository: https://hg.mozilla.org/integration/autoland
18:04:22     INFO -  mozversion application_vendor: Mozilla
18:04:22     INFO -  mozversion application_version: 64.0a1
18:04:22     INFO -  mozversion platform_buildid: 20180919232655
18:04:22     INFO -  mozversion platform_changeset: 8e1af99ab1adb64ba631d81f67942dc9b6c48df5
18:04:22     INFO -  mozversion platform_repository: https://hg.mozilla.org/integration/autoland
18:04:22     INFO -  mozversion platform_version: 64.0a1
18:04:22     INFO -  using testdate: 1537405462
18:04:22     INFO -  actual date: 1537405462
18:04:22     INFO -  starting webserver on 'localhost:34400'
18:04:22     INFO -  Starting mitmproxy playback using env path: /home/cltbld/workspace/build/application/firefox
18:04:22     INFO -  Starting mitmproxy playback using command: /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmdump -k -s /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/alternate-server-replay.py /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-google.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-youtube.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-amazon.mp /home/cltbld/workspace/build/tests/talos/talos/mitmproxy/mitmproxy-recording-facebook.mp
18:04:22     INFO -  Error starting proxy server: OSError(98, 'Address already in use')
18:04:32     INFO -  Aborting: mitmproxy playback process failed to start, poll returned: 1
18:04:32     INFO - Return code: 0
18:04:32 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
18:04:32 CRITICAL - Error copying results /home/cltbld/workspace/build/local.json to upload dir /home/cltbld/workspace/build/blobber_upload_dir/perfherder-data.json
18:04:32 CRITICAL - # TBPL FAILURE #
18:04:32  WARNING - setting return code to 2
18:04:32     INFO - Running post-action listener: _package_coverage_data
18:04:32     INFO - Running post-action listener: _resource_record_post_action
18:04:32     INFO - Running post-action listener: process_java_coverage_data
18:04:32     INFO - [mozharness: 2018-09-20 01:04:32.645815Z] Finished run-tests step (success)
18:04:32     INFO - Running post-run listener: _resource_record_post_run
18:04:32     INFO - Total resource usage - Wall time: 24s; CPU: 8.0%; Read bytes: 0; Write bytes: 3694592; Read time: 0; Write time: 24
18:04:32     INFO - TinderboxPrint: CPU usage<br/>7.9%
18:04:32     INFO - TinderboxPrint: I/O read bytes / time<br/>0 / 0
18:04:32     INFO - TinderboxPrint: I/O write bytes / time<br/>3,694,592 / 24
18:04:32     INFO - TinderboxPrint: CPU idle<br/>176.6 (92.1%)
18:04:32     INFO - TinderboxPrint: CPU user<br/>14.7 (7.7%)
18:04:32     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
18:04:32     INFO - install - Wall time: 13s; CPU: 13.0%; Read bytes: 0; Write bytes: 1687552; Read time: 0; Write time: 16
18:04:32     INFO - setup-mitmproxy - Wall time: 1s; CPU: 12.0%; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
18:04:32     INFO - run-tests - Wall time: 11s; CPU: 0.0%; Read bytes: 0; Write bytes: 2007040; Read time: 0; Write time: 8
18:04:32     INFO - Running post-run listener: copy_logs_to_upload_dir
18:04:32     INFO - Copying logs to upload dir...
18:04:32     INFO - mkdir: /home/cltbld/workspace/build/upload/logs
18:04:32     INFO - Copying logs to upload dir...
18:04:32  WARNING - returning nonzero exit status 2

rwood: Can you please take a look at this bug?

Comment 51

[Intermittent Failures Robot]

76 failures in 2902 pushes (0.026 failures/push) were associated with this bug in the last 7 days.

This is the #14 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 23
* mozilla-inbound: 26
* try: 9
* mozilla-beta: 7
* mozilla-central: 11

Platform breakdown:
* linux64-qr: 31
* linux64: 45

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-09-17&endday=2018-09-23&tree=all

Comment 52

[Intermittent Failures Robot]

1 failures in 2773 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-09-24&endday=2018-09-30&tree=all

Comment 53

[Eliza Balazs [:ebalazs_]]

There are 10 failures in the last 2 weeks. jmaher: what would be the right whiteboard tag for this (Comment 38)?
Clearing the 2 months ago ni for rwood.

Comment 54

[Joel Maher ( :jmaher ) (UTC -8)]

ok, marking this as fixed:other

Comment 55

[Intermittent Failures Robot]

9 failures in 2708 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 6
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* linux64-qr: 2
* linux64: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-10-01&endday=2018-10-07&tree=all

Comment 56

[Intermittent Failures Robot]

2 failures in 2759 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1
* try: 1

Platform breakdown:
* linux64-qr: 1
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-10-15&endday=2018-10-21&tree=all

Comment 57

[Intermittent Failures Robot]

1 failures in 2584 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1457452&startday=2018-10-22&endday=2018-10-28&tree=all