1527134 - Intermittent [taskcluster:error] exit status 1073807364

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: OpenCloudConfig, task, P5)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: btara [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=227748563&repo=mozilla-release

https://queue.taskcluster.net/v1/task/Mv77YdcwQEeBu6gOHqlHnA/runs/0/artifacts/public/logs/live_backing.log

The failure summary is the same as bug 1485628 but the log looks different.

21:59:59 INFO - TEST-START | dom/tests/mochitest/fetch/test_fetch_observer.html
22:00:00 INFO - GECKO(192) | ++DOMWINDOW == 14 (0000024D93CA9C00) [pid = 11184] [serial = 64] [outer = 0000024D932B8400]
22:00:00 INFO - GECKO(192) | ++DOCSHELL 0000024D8B941800 == 4 [pid = 11184] [id = {f5b3d361-98eb-4a90-a6ec-2d48b8313f70}]
22:00:00 INFO - GECKO(192) | ++DOMWINDOW == 15 (0000024D93CA5800) [pid = 11184] [serial = 65] [outer = 0000000000000000]
22:00:00 INFO - GECKO(192) | ++DOMWINDOW == 16 (0000024D95329800) [pid = 11184] [serial = 66] [outer = 0000024D93CA5800]
22:00:00 INFO - GECKO(192) | JavaScript error: http://mochi.test:8888/tests/dom/tests/mochitest/fetch/file_fetch_observer.html, line 24: AbortError: The operation was aborted.
[taskcluster 2019-02-11T22:00:00.846Z] Exit Code: 1073807364
[taskcluster 2019-02-11T22:00:00.846Z] User Time: 15.625ms
[taskcluster 2019-02-11T22:00:00.846Z] Kernel Time: 0s
[taskcluster 2019-02-11T22:00:00.846Z] Wall Time: 30m49.6471419s
[taskcluster 2019-02-11T22:00:00.846Z] Result: FAILED
[taskcluster 2019-02-11T22:00:00.846Z] === Task Finished ===
[taskcluster 2019-02-11T22:00:00.846Z] Task Duration: 30m49.6481212s
[taskcluster 2019-02-11T22:00:01.323Z] Uploading artifact public/logs/localconfig.json from file logs\localconfig.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:01.940Z] Uploading artifact public/logs/log_critical.log from file logs\log_critical.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:02.280Z] Uploading artifact public/logs/log_error.log from file logs\log_error.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:02.630Z] Uploading artifact public/logs/log_fatal.log from file logs\log_fatal.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:03.018Z] Uploading artifact public/logs/log_info.log from file logs\log_info.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:04.036Z] Uploading artifact public/logs/log_raw.log from file logs\log_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:04.677Z] Uploading artifact public/logs/log_warning.log from file logs\log_warning.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:05.009Z] Uploading artifact public/test_info/manifests.list from file build\blobber_upload_dir\manifests.list with content encoding "gzip", mime type "application/octet-stream" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:05.422Z] Uploading artifact public/test_info/plain-chunked_errorsummary.log from file build\blobber_upload_dir\plain-chunked_errorsummary.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:05.794Z] Uploading artifact public/test_info/plain-chunked_raw.log from file build\blobber_upload_dir\plain-chunked_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:06.402Z] Uploading artifact public/test_info/system-info.log from file build\blobber_upload_dir\system-info.log with content encoding "gzip", mime type "text/plain" and expiry 2020-02-11T20:29:50.088Z
[taskcluster 2019-02-11T22:00:06.993Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/Mv77YdcwQEeBu6gOHqlHnA/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-02-11T20:29:50.088Z
[taskcluster:error] exit status 1073807364

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

 22:00:00 INFO - GECKO(192) | JavaScript error: http://mochi.test:8888/tests/dom/tests/mochitest/fetch/file_fetch_observer.html, line 24: AbortError: The operation was aborted.

looks like a test failed -- not a worker issue.

Comment 2

[Bogdan Tara[:bogdan_tara | bogdant]]

:dustin
There are more failures like this one (same exit status code) on other trees too.

eg: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=227749381&repo=autoland&lineNumber=9075

Comment 3

[Bogdan Tara[:bogdan_tara | bogdant]]

Another example:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=227751244&repo=autoland&lineNumber=86662

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Pete, some google suggests this is Windows-ese for a host shutting down. Is this spot termination, by chance?

Comment 5

[Intermittent Failures Robot]

64 failures in 533 pushes (0.12 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18
• mozilla-inbound: 4
• try: 1
• mozilla-beta: 5
• mozilla-central: 30
• mozilla-release: 6

Platform breakdown:

• windows10-64: 34
• windows7-32: 2
• windows10-64-nightly: 1
• linux64: 1
• windows10-64-ccov: 26

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1527134&startday=2019-02-11&endday=2019-02-11&tree=all

Comment 6

[Rob Thijssen [:grenade (EET/UTC+0300)]]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #4)

Pete, some google suggests this is Windows-ese for a host shutting down. Is this spot termination, by chance?

it is indeed a system reboot and looks to be triggered by windows update (which is supposed to be disabled. we've seen this before, bug 1485628)

i am investigating now...

Comment 7

[Rob Thijssen [:grenade (EET/UTC+0300)]]

argh! there was some configuration missing from the gecko-t-win10-64 manifest which handles the setting of the registry keys that disable updates on windows 10.

i've reinstated the missing config and redeployed the worker type. it will be 40 minutes or so before the rebooting machines are purged and replaced with better behaving ones.

Comment 8

[Rob Thijssen [:grenade (EET/UTC+0300)]]

just a note that the patch succeeded. last occurrence of "1073807364" in the event logs is at 2019-02-12 06:14:13 UTC (5 hours ago)

https://papertrailapp.com/groups/2488493/events?q=1073807364

Comment 10

[Intermittent Failures Robot]

17 failures in 564 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17

Platform breakdown:

• windows10-64: 16
• windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1527134&startday=2019-02-12&endday=2019-02-12&tree=all

Comment 11

[Intermittent Failures Robot]

81 failures in 2832 pushes (0.029 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 35
• mozilla-central: 30
• mozilla-release: 6
• try: 1
• mozilla-inbound: 4
• mozilla-beta: 5

Platform breakdown:

• windows10-64-nightly: 1
• windows7-32: 2
• windows10-64: 50
• windows10-64-qr: 1
• linux64: 1
• windows10-64-ccov: 26

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1527134&startday=2019-02-11&endday=2019-02-17&tree=all

Comment 12

[Narcis Beleuzu [:NarcisB]]

Fail reappeared on beta.
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=231957028&repo=mozilla-beta&lineNumber=749

Comment 13

[Rob Thijssen [:grenade (EET/UTC+0300)]]

these failures are most likely triggered by our upgrade to generic worker 13 (see bug 1524592, comment 19).
during this particular upgrade, we've had to manually terminate workers running older gw. we don't normally need to do this and i don't foresee circumstances that would call for this again, but on this occasion, the upgrade path calls for manual terminations. under normal circumstances, workers terminate themselves after completing any tasks in progress which prevents errors like those mentioned here. apologies for the inconvenience. aborted tasks should complete normally when retriggered.

Comment 14

[Intermittent Failures Robot]

10 failures in 2818 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 9

Platform breakdown:

• windows10-64-ux: 1
• windows2012-32-noopt: 1
• windows2012-64: 5
• windows2012-64-noopt: 2
• windows2012-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1527134&startday=2019-03-04&endday=2019-03-10&tree=all

Comment 15

[Intermittent Failures Robot]

1 failures in 2561 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1527134&startday=2019-03-18&endday=2019-03-24&tree=all

Comment 16

[Intermittent Failures Robot]

2 failures in 2673 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1
• try: 1

Platform breakdown:

• windows10-64-qr: 1
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1527134&startday=2019-04-15&endday=2019-04-21&tree=all

Comment 17

[Intermittent Failures Robot]

1 failures in 4298 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• macosx1010-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1527134&startday=2019-07-15&endday=2019-07-21&tree=all