1659965 - Intermittent [Tier2] Linux CCOV TEST-UNEXPECTED-FAIL | automation.py | application terminated with exit code -11

Status: RESOLVED FIXED

(Testing :: Code Coverage, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nbeleuzu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=313428139&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RIdID35dSmeDZJGCTau7YQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-08-19T11:17:50.782Z] 11:17:50 INFO - runtests.py | Running tests: start.
[task 2020-08-19T11:17:50.782Z] 11:17:50 INFO -
[task 2020-08-19T11:17:50.797Z] 11:17:50 INFO - Application command: /builds/worker/workspace/build/application/firefox/firefox -marionette -foreground -profile /tmp/tmpsGC8bF.mozrunner
[task 2020-08-19T11:17:50.797Z] 11:17:50 INFO - runtests.py | Application pid: 1517
[task 2020-08-19T11:17:50.797Z] 11:17:50 INFO - TEST-INFO | started process GECKO(1517)
[task 2020-08-19T11:20:50.887Z] 11:20:50 INFO - runtests.py | Waiting for browser...
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - TEST-INFO | Main app process: killed by SIGSEGV
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - Buffered messages finished
[task 2020-08-19T11:20:50.889Z] 11:20:50 ERROR - TEST-UNEXPECTED-FAIL | automation.py | application terminated with exit code -11
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - runtests.py | Application ran for: 0:03:00.090588
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - zombiecheck | Reading PID log: /tmp/tmprh5K__pidlog
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - Traceback (most recent call last):
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 2979, in doTests
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - e10s=options.e10s
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 2442, in runApp
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - six.reraise(exc, value, tb)
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 2355, in runApp
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - self.marionette.start_session()
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - m._handle_socket_failure()
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 654, in _handle_socket_failure
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - reraise(exc_cls, exc, tb)
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 26, in _
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - return func(*args, **kwargs)
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 1112, in start_session
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - self.raise_for_port(timeout=timeout)
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 573, in raise_for_port
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - self.host, self.port))
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - timeout: Timed out waiting for connection on 127.0.0.1:2828!
[task 2020-08-19T11:20:50.889Z] 11:20:50 ERROR - Automation Error: Received unexpected exception while running application
[task 2020-08-19T11:20:50.889Z] 11:20:50 ERROR -
[task 2020-08-19T11:20:50.889Z] 11:20:50 INFO - Stopping web server
[task 2020-08-19T11:20:50.896Z] 11:20:50 INFO - Stopping web socket server
[task 2020-08-19T11:20:50.917Z] 11:20:50 INFO - Stopping ssltunnel
[task 2020-08-19T11:20:50.944Z] 11:20:50 INFO - websocket/process bridge listening on port 8191
[task 2020-08-19T11:20:50.972Z] 11:20:50 INFO - Stopping websocket/process bridge
[task 2020-08-19T11:20:50.972Z] 11:20:50 WARNING - leakcheck | refcount logging is off, so leaks can't be detected!
[task 2020-08-19T11:20:50.973Z] 11:20:50 INFO - runtests.py | Running tests: end.
[task 2020-08-19T11:20:50.973Z] 11:20:50 INFO - Buffered messages finished

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Likely a build issue because all tests which launch the application fail: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=157db696462d8a98905d0f8697088aa97cb6e08f&selectedTaskRun=RIdID35dSmeDZJGCTau7YQ.0

Comment 2

[(Away)]

Is it possible to backfill these jobs on autoland to get a smaller regression range?

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

We (sheriffs) plan to check if the issue affects the next Linux ccov build and backfill in that case. If the build and tests succeed, shall this issue still be investigated (rerunning the failed build + a test task and backfilling if the rerun also failed)?

Comment 4

[(Away)]

If the jobs go back to green then I wouldn't worry about it IMO.

Comment 5

[Cosmin Sabou [:CosminS]]

Looks like the failures are a fallout from Bug 1658847 - https://hg.mozilla.org/integration/autoland/rev/4a008bd49ea4c445c9b3bfe947e4288427daf342
Here are the backfills.
Andrew, could you have a look over it as Linux ccov tests are failing massively because of this? TH link. Thank you.

Comment 6

[Intermittent Failures Robot]

118 failures in 934 pushes (0.126 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 118

Platform breakdown:

• linux1804-64-ccov: 118

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1659965&startday=2020-08-19&endday=2020-08-19&tree=all

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1658847

Comment 9

[Andrew Osmond [:aosmond] (he/him)]

I suspect the profiler needs to be initialized before code coverage.

Comment 10

[Andrew Osmond [:aosmond] (he/him)]

Try came back green:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=620190362e02700f3f1b61b9dfa1341199b92565

Comment 11

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1659965 - Initialize the profiler before CodeCoverageHandler.

CodeCoverageHandler relies upon CrossProcessMutex. On Linux this is
implemented using shared memory. In bug 1658847, we used to the
profiler's "thread sleep" mechanism to resolve the signal interrupts in
posix_fallocate, which requires the profiler to be initialized. As such,
CodeCoverageHandler now needs to be initialized after.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a341ca5f7ec5
Initialize the profiler before CodeCoverageHandler. r=mstange

Comment 13

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/a341ca5f7ec5

Comment 14

[Intermittent Failures Robot]

116 failures in 643 pushes (0.18 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 116

Platform breakdown:

• linux1804-64-ccov: 116

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1659965&startday=2020-08-21&endday=2020-08-21&tree=all

Comment 15

[Intermittent Failures Robot]

236 failures in 4100 pushes (0.058 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 236

Platform breakdown:

• linux1804-64-ccov: 236

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1659965&startday=2020-08-17&endday=2020-08-23&tree=all