1658084 - Perma testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_leak TEST-UNEXPECTED-FAIL

Status: RESOLVED FIXED

(Testing :: Mochitest, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=312458006&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/c0eDAA0kS5apdB6XhDXq9Q/runs/0/artifacts/public/logs/live_backing.log

---

task 2020-08-08T00:12:09.777Z] 2:10.03 /builds/worker/checkouts/gecko/testing/mochitest/tests/python/test_basic_mochitest_plain.py
[task 2020-08-08T00:12:09.778Z] 2:10.03 ============================= test session starts ==============================
[task 2020-08-08T00:12:09.778Z] 2:10.03 platform linux2 -- Python 2.7.17, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/gecko-QUm8IANb-2/bin/python
[task 2020-08-08T00:12:09.778Z] 2:10.03 rootdir: /builds/worker/checkouts/gecko, inifile: /builds/worker/checkouts/gecko/config/mozunit/mozunit/pytest.ini
[task 2020-08-08T00:12:09.778Z] 2:10.03 collecting ... collected 6 items
[task 2020-08-08T00:12:09.779Z] 2:10.03
[task 2020-08-08T00:12:09.779Z] 2:10.03 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_pass PASSED
[task 2020-08-08T00:12:09.779Z] 2:10.03 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_fail PASSED
[task 2020-08-08T00:12:09.779Z] 2:10.03 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_crash PASSED
[task 2020-08-08T00:12:09.779Z] 2:10.03 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_asan SKIPPED
[task 2020-08-08T00:12:09.780Z] 2:10.03 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_assertion PASSED
[task 2020-08-08T00:12:09.780Z] 2:10.03 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_leak TEST-UNEXPECTED-FAIL
[task 2020-08-08T00:12:09.781Z] 2:10.03
[task 2020-08-08T00:12:09.781Z] 2:10.03 =================================== FAILURES ===================================
[task 2020-08-08T00:12:09.782Z] 2:10.03 _______________________________ test_output_leak _______________________________
[task 2020-08-08T00:12:09.782Z] 2:10.03
[task 2020-08-08T00:12:09.782Z] 2:10.03 runtests = <function inner at 0x7fcc643447d0>
[task 2020-08-08T00:12:09.783Z] 2:10.03
[task 2020-08-08T00:12:09.783Z] 2:10.03 @pytest.mark.skip_mozinfo("!debug")
[task 2020-08-08T00:12:09.783Z] 2:10.03 def test_output_leak(runtests):
[task 2020-08-08T00:12:09.783Z] 2:10.03 status, lines = runtests('test_leak.html')
[task 2020-08-08T00:12:09.783Z] 2:10.03 # TODO: mochitest should return non-zero here
[task 2020-08-08T00:12:09.783Z] 2:10.03 assert status == 0
[task 2020-08-08T00:12:09.783Z] 2:10.03
[task 2020-08-08T00:12:09.783Z] 2:10.03 tbpl_status, log_level, summary = get_mozharness_status(lines, status)
[task 2020-08-08T00:12:09.784Z] 2:10.03 > assert tbpl_status == TBPL_WARNING
[task 2020-08-08T00:12:09.784Z] 2:10.03 E AssertionError: assert 'SUCCESS' == 'WARNING'
[task 2020-08-08T00:12:09.784Z] 2:10.03 E - SUCCESS
[task 2020-08-08T00:12:09.784Z] 2:10.03 E + WARNING
[task 2020-08-08T00:12:09.784Z] 2:10.03
[task 2020-08-08T00:12:09.784Z] 2:10.03 testing/mochitest/tests/python/test_basic_mochitest_plain.py:110: AssertionError

Comment 1

[Natalia Csoregi [:nataliaCs]]

Retriggered/backfilled down to: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=Linux%2C18.04%2Cx64%2Cdebug%2CPython%2C2%2Cunit%2Ctests%2Csource-test-python-mochitest-harness-linux1804-64%2Fdebug%2Cmch&tochange=0fa4371f24bf9108eba4ddd4ca0815a4b668b217&fromchange=86c7c0bb421e5b2eb11e264bb5e35cc834c81fd1&selectedTaskRun=ROBE32SKTWmyYyfhHlLiaA.0

Comment 2

[Intermittent Failures Robot]

9 failures in 4618 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 2

Platform breakdown:

• linux1804-64: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-08-03&endday=2020-08-09&tree=all

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Only bug 1657363 is a reasonable candidate as regressor. The other changesets enable a test our only touch the devtools folder.

Comment 4

[Emilio Cobos Álvarez (:emilio)]

This seems some kind of mochitest/reftest log parser unit test, which somehow is no longer detected a leak. I have zero idea how my patch could've possibly caused this.

Bug 1657650 and bug 1657948 seem more related.

Comment 5

[Andrew Halberstadt [:ahal]]

This task only runs when certain files are changed, so if it's truly intermittent it could have been regressed awhile back. I'll take a look tomorrow.

Comment 6

[Andrew Halberstadt [:ahal]]

Due to recent events, this has fallen down the list of priorities a little :/

Comment 7

[Intermittent Failures Robot]

6 failures in 3601 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 3

Platform breakdown:

• linux1804-64: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-08-10&endday=2020-08-16&tree=all

Comment 8

[Intermittent Failures Robot]

12 failures in 4100 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 5

Platform breakdown:

• linux1804-64: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-08-17&endday=2020-08-23&tree=all

Comment 9

[Intermittent Failures Robot]

8 failures in 4046 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 4
• mozilla-central: 3

Platform breakdown:

• linux1804-64: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-08-24&endday=2020-08-30&tree=all

Comment 10

[Intermittent Failures Robot]

13 failures in 2984 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• autoland: 7
• mozilla-beta: 1

Platform breakdown:

• linux1804-64: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-08-31&endday=2020-09-06&tree=all

Comment 11

[Andrew Halberstadt [:ahal]]

The bisection does look like this started happening with bug 1657363:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2C18.04%2Cx64%2Cdebug%2Cpython%2C2%2Cunit%2Ctests%2Csource-test-python-mochitest-harness-linux1804-64%2Fdebug%2Cmch&tochange=e5eb146b8ba12453a79126ff9b59b7fa965ec333&fromchange=80852dff4c7fb99a30c72de3fd2180b7983d87f1

However because of the build bustage, it could also have been one of:

• Bug 1642769
• Bug 1653457

Though those look even less relevant than bug 1657363. I don't think this is caused by any of these bugs though, I'd be surprised if the fix here isn't in the test itself.

Comment 12

[Andrew Halberstadt [:ahal]]

I backed out both bug 1642769 and bug 1653457 and pushed to try, but the task still failed. So that leads me to believe that bug 1657363 was indeed the start of the regression.

Note: I'm not suggesting that you are on the hook for fixing this :emilio, I agree that change looks unrelated and this is likely a test issue. Just adding the regression relationship back in for tracking purposes.

Comment 13

[Intermittent Failures Robot]

11 failures in 3410 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 5

Platform breakdown:

• linux1804-64: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-09-07&endday=2020-09-13&tree=all

Comment 14

[Intermittent Failures Robot]

14 failures in 3760 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• mozilla-central: 5
• autoland: 8

Platform breakdown:

• linux1804-64: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-09-14&endday=2020-09-20&tree=all

Comment 16

[Intermittent Failures Robot]

4 failures in 4019 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 1
• try: 2

Platform breakdown:

• linux1804-64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-09-21&endday=2020-09-27&tree=all

Comment 17

[Intermittent Failures Robot]

21 failures in 4195 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 14
• mozilla-central: 6

Platform breakdown:

• linux1804-64: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-09-28&endday=2020-10-04&tree=all

Comment 18

[Intermittent Failures Robot]

19 failures in 4130 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• try: 5
• mozilla-central: 6

Platform breakdown:

• linux1804-64: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-10-05&endday=2020-10-11&tree=all

Comment 19

[Intermittent Failures Robot]

5 failures in 3694 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• try: 1
• autoland: 2
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-10-12&endday=2020-10-18&tree=all

Comment 20

[Intermittent Failures Robot]

20 failures in 4230 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13
• mozilla-central: 7

Platform breakdown:

• linux1804-64: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-10-19&endday=2020-10-25&tree=all

Comment 21

[Intermittent Failures Robot]

18 failures in 4275 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-central: 7

Platform breakdown:

• linux1804-64: 14
• windows10-64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-10-26&endday=2020-11-01&tree=all

Comment 22

[Andrew McCreight [:mccr8]]

(In reply to Andrew Halberstadt [:ahal] from comment #12)

Note: I'm not suggesting that you are on the hook for fixing this :emilio, I agree that change looks unrelated and this is likely a test issue. Just adding the regression relationship back in for tracking purposes.

It looks like this was actually an issue. It seems like we haven't had leak checking on Linux for a few months. Nika happened upon a similar failure in this test, and did some investigation, and it looks like some printing changes added a new static initializer which broke leak checking.

Comment 23

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1658084 - Avoid creating a mozilla::OffTheBooksMutex in a static initializer,

Comment 24

[Andrew McCreight [:mccr8]]

[Tracking Requested - why for this release]: Leak checking is not working on Linux. We should backport it to the relevant branches. It looks like the regressing bug landed in 81.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2e6309c1cdbd
Avoid creating a mozilla::OffTheBooksMutex in a static initializer, r=emilio,mccr8

Comment 26

[Razvan Maries]

Backed out for perma failures.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=QCx8t7M5RIqTjqE61AQi6g.0&resultStatus=testfailed%2Cbusted%2Cexception&revision=2e6309c1cdbd39f755a8e65977d29d89e10c86e0

Logs:
https://treeherder.mozilla.org/logviewer?job_id=320669522&repo=autoland&lineNumber=3554
https://treeherder.mozilla.org/logviewer?job_id=320669497&repo=autoland&lineNumber=2759

TV failures:
https://treeherder.mozilla.org/logviewer?job_id=320669159&repo=autoland&lineNumber=11047
https://treeherder.mozilla.org/logviewer?job_id=320669030&repo=autoland&lineNumber=12179

Backout: https://hg.mozilla.org/integration/autoland/rev/2caabcdb4753d0fbc5c95d5e6b2a64e0eee59d2d

Comment 27

[BugBot [:suhaib / :marco/ :calixte]]

Changing the priority to p1 as the bug is tracked by a release manager for the current beta.
See What Do You Triage for more information

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/addf7bdc4c56
Avoid creating a mozilla::OffTheBooksMutex in a static initializer, r=emilio,mccr8

Comment 29

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/addf7bdc4c56

Comment 30

[Ryan VanderMeulen [:RyanVM]]

Please nominate this for Beta approval when you get a chance.

Comment 31

[Nika Layzell [:nika] (ni? for response)]

Comment on attachment 9185378 [details]
Bug 1658084 - Avoid creating a mozilla::OffTheBooksMutex in a static initializer,

Beta/Release Uplift Approval Request

• User impact if declined: Linux leak-check builds broken for content processes
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Fairly straightforward patch to fix major testing issues.

Does impact startup, but should be fairly consistent across devices.

• String changes made/needed: None

Comment 32

[Intermittent Failures Robot]

6 failures in 4128 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 3

Platform breakdown:

• linux1804-64: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1658084&startday=2020-11-02&endday=2020-11-08&tree=all

Comment 33

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9185378 [details]
Bug 1658084 - Avoid creating a mozilla::OffTheBooksMutex in a static initializer,

Not essential for our release users and will be in our 84 beta next week, thanks.

Comment 34

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:nika, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 35

[Andrew McCreight [:mccr8]]

Comment on attachment 9185378 [details]
Bug 1658084 - Avoid creating a mozilla::OffTheBooksMutex in a static initializer,

Beta/Release Uplift Approval Request

• User impact if declined: see comment 31
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
• String changes made/needed:

Comment 36

[Andrew McCreight [:mccr8]]

Oh wait, I got confused.