1440995 - Intermittent talos OS X Aborting task - max run time exceeded! followed by Exit Code: -1

Status: RESOLVED DUPLICATE of bug 1445580

(Testing :: Talos, defect)

Description

[Andreea Pavel [:apavel]]

Revision: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=bfe62272d2a21f9d10d45e5aaa680f5c735604ae

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=164190404&repo=mozilla-central


02:59:47     INFO -  TEST-INFO | started process 2511 (/Users/cltbld/tasks/task_1519524388/build/application/Firefox Nightly.app/Contents/MacOS/firefox -foreground -profile /var/folders/9g/1jk_l9qn7pn68yzgn2443qzc00000w/T/tmpQmZUGh/profile http://localhost:49225/startup_test/tresize/addon/content/tresize-test.html?gecko_profile_entries=1000000&gecko_profile_threads=GeckoMain%2CCompositor&gecko_profile_dir=%2Fvar%2Ffolders%2F9g%2F1jk_l9qn7pn68yzgn2443qzc00000w%2FT%2Ftmp1WLxYm&gecko_profile_interval=2 http://localhost:49225/startup_test/tresize/addon/content/tresize-test.html?gecko_profile_entries=1000000&gecko_profile_threads=GeckoMain%2CCompositor&gecko_profile_dir=%2Fvar%2Ffolders%2F9g%2F1jk_l9qn7pn68yzgn2443qzc00000w%2FT%2Ftmp1WLxYm&gecko_profile_interval=2)
02:59:47     INFO -  PID 2511 | Unable to read VR Path Registry from /Users/cltbld/Library/Application Support/OpenVR/.openvr/openvrpaths.vrpath
02:59:48     INFO -  PID 2511 | Unable to read VR Path Registry from /Users/cltbld/Library/Application Support/OpenVR/.openvr/openvrpaths.vrpath
02:59:49     INFO -  PID 2511 | Unable to read VR Path Registry from /Users/cltbld/Library/Application Support/OpenVR/.openvr/openvrpaths.vrpath
02:59:49     INFO -  PID 2511 | 2018-02-25 02:59:49.392 plugin-container[2512:33809] *** CFMessagePort: bootstrap_register(): failed 1100 (0x44c) 'Permission denied', port = 0x9d3b, name = 'com.apple.tsm.portname'
02:59:49     INFO -  PID 2511 | See /usr/include/servers/bootstrap_defs.h for the error codes.
02:59:49     INFO -  PID 2511 | 2018-02-25 02:59:49.421 plugin-container[2513:33831] *** CFMessagePort: bootstrap_register(): failed 1100 (0x44c) 'Permission denied', port = 0x924b, name = 'com.apple.tsm.portname'
02:59:49     INFO -  PID 2511 | See /usr/include/servers/bootstrap_defs.h for the error codes.
02:59:50     INFO -  PID 2511 | Unable to read VR Path Registry from /Users/cltbld/Library/Application Support/OpenVR/.openvr/openvrpaths.vrpath
02:59:50     INFO -  PID 2511 | 2018-02-25 02:59:50.758 plugin-container[2514:33941] *** CFMessagePort: bootstrap_register(): failed 1100 (0x44c) 'Permission denied', port = 0x963f, name = 'com.apple.tsm.portname'
02:59:50     INFO -  PID 2511 | See /usr/include/servers/bootstrap_defs.h for the error codes.
03:00:03     INFO -  PID 2511 | 2018-02-25 03:00:02.999 firefox[2511:33804] Persistent UI failed to open file file:///Users/cltbld/Library/Saved%20Application%20State/org.mozilla.nightly.savedState/window_1.data: No such file or directory (2)
03:00:05     INFO -  PID 2511 | __start_report28.62__end_report
03:00:05     INFO -  PID 2511 | __startTimestamp1519556405340__endTimestamp
03:00:06     INFO -  PID 2511 | *** UTM:SVC TimerManager:registerTimer called after profile-before-change notification. Ignoring timer registration for id: telemetry_modules_ping
03:00:06     INFO -  TEST-INFO | 2511: exit 0
[taskcluster 2018-02-25T11:00:17.346Z] Aborting task - max run time exceeded!
[taskcluster 2018-02-25T11:00:17.350Z] Exit Code: -1
[taskcluster 2018-02-25T11:00:17.350Z] === Task Finished ===
[taskcluster 2018-02-25T11:00:17.350Z] Task Duration: 14m58.809909888s
[taskcluster 2018-02-25T11:00:17.980Z] Uploading artifact public/logs/localconfig.json from file logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:18.439Z] Uploading artifact public/logs/talos_critical.log from file logs/talos_critical.log with content encoding "gzip", mime type "text/plain" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:18.797Z] Uploading artifact public/logs/talos_error.log from file logs/talos_error.log with content encoding "gzip", mime type "text/plain" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:19.176Z] Uploading artifact public/logs/talos_fatal.log from file logs/talos_fatal.log with content encoding "gzip", mime type "text/plain" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:19.569Z] Uploading artifact public/logs/talos_info.log from file logs/talos_info.log with content encoding "gzip", mime type "text/plain" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:19.993Z] Uploading artifact public/logs/talos_raw.log from file logs/talos_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:20.539Z] Uploading artifact public/logs/talos_warning.log from file logs/talos_warning.log with content encoding "gzip", mime type "text/plain" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:20.908Z] Uploading artifact public/test_info/chromez-e10s_errorsummary.log from file build/blobber_upload_dir/chromez-e10s_errorsummary.log with content encoding "gzip", mime type "text/plain" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:21.295Z] Uploading artifact public/test_info/chromez-e10s_raw.log from file build/blobber_upload_dir/chromez-e10s_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:21.697Z] Uploading artifact public/test_info/missingsymbols.zip from file build/blobber_upload_dir/missingsymbols.zip with content encoding "", mime type "application/zip" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:22.578Z] Uploading artifact public/test_info/profile_about_preferences_basic.zip from file build/blobber_upload_dir/profile_about_preferences_basic.zip with content encoding "", mime type "application/zip" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:23.241Z] Uploading artifact public/test_info/profile_tresize.zip from file build/blobber_upload_dir/profile_tresize.zip with content encoding "", mime type "application/zip" and expiry 2019-02-25T10:03:27.189Z
[taskcluster 2018-02-25T11:00:23.574Z] Task not successful due to following exception(s):
[taskcluster 2018-02-25T11:00:23.574Z] Exception 1)
[taskcluster 2018-02-25T11:00:23.574Z] signal: killed

Comment 1

[Intermittent Failures Robot]

10 failures in 675 pushes (0.015 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 10

Platform breakdown:
* macosx64-nightly: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1440995&startday=2018-02-19&endday=2018-02-25&tree=all

Comment 2

[Intermittent Failures Robot]

25 failures in 831 pushes (0.03 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 23
* mozilla-inbound: 1
* autoland: 1

Platform breakdown:
* macosx64-nightly: 23
* osx-10-10: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1440995&startday=2018-02-26&endday=2018-03-04&tree=all

Comment 3

[Stefan Hindli [:stefan_hindli]]

Over the last 7 days this bug has 30 failures. These happen on OS X 10.10 and macosx64-nightly 

Here is the most recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=165946894

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

these are all on profiling jobs- I suspect there is something broken there- possibly a quick fix.

Comment 5

[Intermittent Failures Robot]

16 failures in 747 pushes (0.021 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 6
* mozilla-inbound: 5
* autoland: 5

Platform breakdown:
* osx-10-10: 10
* macosx64-nightly: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1440995&startday=2018-03-05&endday=2018-03-11&tree=all

Comment 6

[Intermittent Failures Robot]

35 failures in 132 pushes (0.265 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 19
* mozilla-inbound: 12
* mozilla-central: 4

Platform breakdown:
* osx-10-10: 29
* macosx64-nightly: 4
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1440995&startday=2018-03-12&endday=2018-03-12&tree=all

Comment 7

[Narcis Beleuzu [:NarcisB]]

Since 11.03 this fail spikes up and now there are 75 failures in the last 3 days.
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1440995&startday=2018-03-11&endday=2018-03-13&tree=all

Most of them are on OS X 10.10 and only 6 on macosx64-nightly. There are 2 on Linux but those are starred wrong
:rwood, can you please take a look?

Comment 8

[Intermittent Failures Robot]

65 failures in 171 pushes (0.38 failures/push) were associated with this bug yesterday.    

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-inbound: 50
* autoland: 10
* mozilla-central: 5

Platform breakdown:
* osx-10-10: 63
* macosx64-nightly: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1440995&startday=2018-03-13&endday=2018-03-13&tree=all

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

this seems to be mostly related to the time to download artifact [1]:
3:14:41     INFO -  u'xpcshell': [u'target.common.tests.zip', u'target.xpcshell.tests.zip']}
23:14:41     INFO - Downloading packages: [u'target.common.tests.zip', u'target.mochitest.tests.zip'] for test suite categories: ['mochitest']
23:14:41     INFO - Downloading and extracting to /Users/cltbld/tasks/task_1520920450/build/tests these dirs * from https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.common.tests.zip
23:14:41     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.common.tests.zip'}, attempt #1
23:14:41     INFO - Fetch https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.common.tests.zip into memory
23:15:01     INFO - Content-Length response header: 36551587
23:15:01     INFO - Bytes received: 36551587
23:15:18     INFO - Downloading and extracting to /Users/cltbld/tasks/task_1520920450/build/tests these dirs * from https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.mochitest.tests.zip
23:15:18     INFO - retry: Calling fetch_url_into_memory with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.mochitest.tests.zip'}, attempt #1
23:15:18     INFO - Fetch https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.mochitest.tests.zip into memory
23:52:34     INFO - Content-Length response header: 78551031
23:52:34     INFO - Bytes received: 78551031
23:52:46     INFO - Downloading https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.dmg to /Users/cltbld/tasks/task_1520920450/installer.dmg
23:52:46     INFO - retry: Calling _download_file with args: (), kwargs: {'url': 'https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.dmg', 'file_name': '/Users/cltbld/tasks/task_1520920450/installer.dmg'}, attempt #1
23:53:00     INFO - Downloaded 73040179 bytes.
23:53:00     INFO - Setting buildbot property build_url to https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.dmg
23:53:00     INFO - mkdir: /Users/cltbld/tasks/task_1520920450/properties
23:53:00     INFO - Writing buildbot properties ['build_url'] to /Users/cltbld/tasks/task_1520920450/properties/build_url
23:53:00     INFO - Writing to file /Users/cltbld/tasks/task_1520920450/properties/build_url


that is almost 30 minutes to download target.common.tests.zip which is 78MB.

In other cases [2], [3] we time out before finishing the download of a .zip or python package.  Here is a snippet from [4]:
15:04:59     INFO -  Collecting simplejson
15:06:15     INFO -    Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x1098fbd10>: Failed to establish a new connection: [Errno 60] Operation timed out',)': /pub
15:07:30     INFO -    Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x1098fb1d0>: Failed to establish a new connection: [Errno 60] Operation timed out',)': /pub
15:08:46     INFO -    Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x109a1d090>: Failed to establish a new connection: [Errno 60] Operation timed out',)': /pub


I am tempted to make this a duplicate of bug 1443130, needinfo :dragrom as that is the next step on bug 1443130 as well.

[1] https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=167596727
[2] https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=167596731
[3] https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=167793330
[4] https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=167775711

Comment 10

[Dragos Crisan [:dragrom]]

Tested on t-yosemite-r7-423.test.releng.mdc1.mozilla.com:

[[e-mail] ~]# wget https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.common.tests.zip; rm target.common.tests.zip
--2018-03-14 05:24:24--  https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.common.tests.zip
Resolving queue.taskcluster.net... 50.16.228.78, 107.22.197.53, 50.17.218.87
Connecting to queue.taskcluster.net|50.16.228.78|:443... connected.
HTTP request sent, awaiting response... 303 See Other
Location: https://taskcluster-artifacts.net/aLf3dLO8T2eepBczSorNug/0/public/build/target.common.tests.zip [following]
--2018-03-14 05:24:25--  https://taskcluster-artifacts.net/aLf3dLO8T2eepBczSorNug/0/public/build/target.common.tests.zip
Resolving taskcluster-artifacts.net... 54.192.212.56
Connecting to taskcluster-artifacts.net|54.192.212.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 36551587 (35M) [application/zip]
Saving to: 'target.common.tests.zip'

100%[==================================================================================================================================================================>] 36,551,587  6.30MB/s   in 11s    

2018-03-14 05:24:37 (3.26 MB/s) - 'target.common.tests.zip' saved [36551587/36551587]

[[e-mail] ~]# wget https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.common.tests.zip; rm target.common.tests.zip
--2018-03-14 05:24:43--  https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.common.tests.zip
Resolving queue.taskcluster.net... 50.16.228.78, 107.22.197.53, 50.17.218.87
Connecting to queue.taskcluster.net|50.16.228.78|:443... connected.
HTTP request sent, awaiting response... 303 See Other
Location: https://taskcluster-artifacts.net/aLf3dLO8T2eepBczSorNug/0/public/build/target.common.tests.zip [following]
--2018-03-14 05:24:44--  https://taskcluster-artifacts.net/aLf3dLO8T2eepBczSorNug/0/public/build/target.common.tests.zip
Resolving taskcluster-artifacts.net... 54.192.212.56
Connecting to taskcluster-artifacts.net|54.192.212.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 36551587 (35M) [application/zip]
Saving to: 'target.common.tests.zip'

100%[==================================================================================================================================================================>] 36,551,587  5.32MB/s   in 7.7s   

2018-03-14 05:24:52 (4.53 MB/s) - 'target.common.tests.zip' saved [36551587/36551587]

[[e-mail] ~]# wget https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.mochitest.tests.zip; rm target.mochitest.tests.zip
--2018-03-14 05:26:14--  https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.mochitest.tests.zip
Resolving queue.taskcluster.net... 50.16.228.78, 107.22.197.53, 50.17.218.87
Connecting to queue.taskcluster.net|50.16.228.78|:443... connected.
HTTP request sent, awaiting response... 303 See Other
Location: https://taskcluster-artifacts.net/aLf3dLO8T2eepBczSorNug/0/public/build/target.mochitest.tests.zip [following]
--2018-03-14 05:26:15--  https://taskcluster-artifacts.net/aLf3dLO8T2eepBczSorNug/0/public/build/target.mochitest.tests.zip
Resolving taskcluster-artifacts.net... 54.192.212.56
Connecting to taskcluster-artifacts.net|54.192.212.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 78551031 (75M) [application/zip]
Saving to: 'target.mochitest.tests.zip'

100%[==================================================================================================================================================================>] 78,551,031  5.71MB/s   in 18s    

2018-03-14 05:26:35 (4.16 MB/s) - 'target.mochitest.tests.zip' saved [78551031/78551031]

I also used curl instead of wget:
[[e-mail] ~]# curl -L -O  https://queue.taskcluster.net/v1/task/aLf3dLO8T2eepBczSorNug/artifacts/public/build/target.mochitest.tests.zip; rm target.mochitest.tests.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   124  100   124    0     0    104      0  0:00:01  0:00:01 --:--:--   104
100 74.9M  100 74.9M    0     0  4813k      0  0:00:15  0:00:15 --:--:-- 6135k

looks like the average speed is around 4,5 MB/s

If I run wget or curl on individual hosts or on the max 3 hosts in the same time, the download speed looks OK.(I tested 3 downloads in the same time from 3 different hosts). Will be interesting to see the network load if more workers will try to download the artifacts in the same time..... 

:dividehex Who can give us a network load for the last 24 hours? I suspect a network bottleneck, when many worker try to download the artifacts in the same time....

Comment 11

[Intermittent Failures Robot]

27 failures in 168 pushes (0.161 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 27

Platform breakdown:
* osx-10-10: 27

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1440995&startday=2018-03-16&endday=2018-03-16&tree=all

Comment 12

[Intermittent Failures Robot]

21 failures in 46 pushes (0.457 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 16
* mozilla-inbound: 5

Platform breakdown:
* osx-10-10: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1440995&startday=2018-03-17&endday=2018-03-17&tree=all

Comment 13

[Intermittent Failures Robot]

166 failures in 814 pushes (0.204 failures/push) were associated with this bug in the last 7 days. 

This is the #5 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* autoland: 80
* mozilla-inbound: 75
* mozilla-central: 11

Platform breakdown:
* osx-10-10: 157
* macosx64-nightly: 7
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1440995&startday=2018-03-12&endday=2018-03-18&tree=all

Comment 15

[Intermittent Failures Robot]

19 failures in 822 pushes (0.023 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 8
* mozilla-central: 7
* mozilla-inbound: 4

Platform breakdown:
* osx-10-10: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1440995&startday=2018-03-20&endday=2018-03-26&tree=trunk

Comment 16

[Intermittent Failures Robot]

1 failures in 718 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1440995&startday=2018-03-26&endday=2018-04-01&tree=trunk

Comment 17

[Intermittent Failures Robot]

1 failures in 765 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1440995&startday=2018-04-02&endday=2018-04-08&tree=trunk

Comment 18

[Intermittent Failures Robot]

5 failures in 782 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 5

Platform breakdown:
* osx-10-10: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1440995&startday=2018-04-16&endday=2018-04-22&tree=trunk

Comment 19

[Intermittent Failures Robot]

8 failures in 780 pushes (0.01 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 6
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* osx-10-10: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1440995&startday=2018-04-23&endday=2018-04-29&tree=trunk

Comment 20

[Intermittent Failures Robot]

5 failures in 551 pushes (0.009 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 3
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 4
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1440995&startday=2018-04-30&endday=2018-05-06&tree=trunk

Comment 21

[Intermittent Failures Robot]

8 failures in 660 pushes (0.012 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 5
* mozilla-inbound: 3

Platform breakdown:
* osx-10-10: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1440995&startday=2018-05-07&endday=2018-05-13&tree=trunk

Comment 22

[Intermittent Failures Robot]

18 failures in 119 pushes (0.151 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 18

Platform breakdown:
* osx-10-10: 10
* macosx64-qr: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1440995&startday=2018-05-17&endday=2018-05-17&tree=trunk

Comment 23

[Intermittent Failures Robot]

28 failures in 639 pushes (0.044 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 22
* mozilla-inbound: 5
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 19
* macosx64-qr: 8
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1440995&startday=2018-05-14&endday=2018-05-20&tree=trunk