1531789 - High frequency toolkit/xre/test/browser_checkdllblockliststate.js | Windows dll blocklist status should be true, indicating it is running properly. A failure in this test is considered a release blocker. -

Status: RESOLVED FIXED

(Toolkit :: Startup and Profile System, defect, P5)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=231227015&repo=mozilla-central

https://queue.taskcluster.net/v1/task/GEs0nm5nSemsL8IPHHvT5Q/runs/0/artifacts/public/logs/live_backing.log

12:55:59 INFO - TEST-START | toolkit/xre/test/browser_checkdllblockliststate.js
12:55:59 INFO - GECKO(5176) | ### XPCOM_MEM_BLOAT_LOG defined -- logChrome file dging bloat/leaks to c:\users\oesn't exist: Z:\task_15514task38841\_build\tests\mochitest\browser\toolki1551438841\appdata\local\temp\tmpqzmswk.mt\xre\test\head.jso
12:55:59 INFO - GECKO(5176) | zrunner\runtests_leaks_tab_pid2976.log
12:55:59 INFO - GECKO(5176) | DLL blocklist was unable to intercept AppInit DLLs.
12:55:59 INFO - GECKO(5176) | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to c:\users\task_1551438841\appdata\local\temp\tmpqzmswk.mozrunner\runtests_leaks_tab_pid9796.log
12:55:59 INFO - GECKO(5176) | [Parent 6872, Main Thread] WARNING: '!parent', file z:/build/build/src/netwerk/ipc/NeckoParent.cpp, line 957
12:56:00 INFO - GECKO(5176) | [Parent 6872, Main Thread] WARNING: '!parent', file z:/build/build/src/netwerk/ipc/NeckoParent.cpp, line 957
12:56:00 INFO - GECKO(5176) | ++DOCSHELL 0000013CCF975800 == 1 [pid = 2976] [id = {96464d7d-3a1e-4889-b28b-cd64664066e3}]
12:56:00 INFO - GECKO(5176) | ++DOMWINDOW == 1 (0000013CD369E800) [pid = 2976] [serial = 1] [outer = 0000000000000000]
12:56:00 INFO - GECKO(5176) | ++DOCSHELL 0000021DAC675800 == 1 [pid = 9796] [id = {3b1332ae-7775-4c58-b121-618ef855d32e}]
12:56:00 INFO - GECKO(5176) | ++DOMWINDOW == 1 (0000021DB03A9800) [pid = 9796] [serial = 1] [outer = 0000000000000000]
12:56:00 INFO - GECKO(5176) | ++DOMWINDOW == 2 (0000013CD36A1800) [pid = 2976] [serial = 2] [outer = 0000013CD369E800]
12:56:00 INFO - GECKO(5176) | ++DOMWINDOW == 2 (0000021DB03AC800) [pid = 9796] [serial = 2] [outer = 0000021DB03A9800]
12:56:00 INFO - GECKO(5176) | ++DOMWINDOW == 3 (0000013CD4698C00) [pid = 2976] [serial = 3] [outer = 0000013CD369E800]
12:56:00 INFO - GECKO(5176) | ++DOMWINDOW == 3 (0000021DB1395C00) [pid = 9796] [serial = 3] [outer = 0000021DB03A9800]
12:56:00 INFO - TEST-INFO | started process screenshot
12:56:00 INFO - TEST-INFO | screenshot: exit 0
12:56:00 INFO - Buffered messages logged at 12:55:59
12:56:00 INFO - Entering test bound test
12:56:00 INFO - Buffered messages finished
12:56:00 INFO - TEST-UNEXPECTED-FAIL | toolkit/xre/test/browser_checkdllblockliststate.js | Windows dll blocklist status should be true, indicating it is running properly. A failure in this test is considered a release blocker. -
12:56:00 INFO - Stack trace:
12:56:00 INFO - chrome://mochikit/content/browser-test.js:test_ok:1304
12:56:00 INFO - chrome://mochitests/content/browser/toolkit/xre/test/browser_checkdllblockliststate.js:test/<:7
12:56:00 INFO - resource://testing-common/BrowserTestUtils.jsm:withNewTab:111
12:56:00 INFO - chrome://mochitests/content/browser/toolkit/xre/test/browser_checkdllblockliststate.js:test:6
12:56:00 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1106
12:56:00 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1134
12:56:00 INFO - chrome://mochikit/content/browser-test.js:nextTest/<:995
12:56:00 INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:803
12:56:00 INFO - Leaving test bound test
12:56:00 INFO - GECKO(5176) | MEMORY STAT | vsize 2103366MB | vsizeMaxContiguous 65141503MB | residentFast 372MB | heapAllocated 114MB
12:56:00 INFO - TEST-OK | toolkit/xre/test/browser_checkdllblockliststate.js | took 1041ms

Comment 1

[Intermittent Failures Robot]

17 failures in 2981 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 17

Platform breakdown:

• windows10-64-ccov: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-02-25&endday=2019-03-03&tree=all

Comment 2

[Intermittent Failures Robot]

35 failures in 2818 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 35

Platform breakdown:

• windows10-64-ccov: 35

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-03-04&endday=2019-03-10&tree=all

Comment 3

[Intermittent Failures Robot]

40 failures in 2771 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #47 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 40

Platform breakdown:

• windows10-64-ccov: 40

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-03-11&endday=2019-03-17&tree=all

Comment 4

[Intermittent Failures Robot]

20 failures in 2561 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 20

Platform breakdown:

• windows10-64-ccov: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-03-18&endday=2019-03-24&tree=all

Comment 5

[Intermittent Failures Robot]

35 failures in 3910 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 34
• try: 1

Platform breakdown:

• windows10-64-ccov: 35

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-03-25&endday=2019-03-31&tree=all

Comment 6

[Intermittent Failures Robot]

19 failures in 2843 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 19

Platform breakdown:

• windows10-64-ccov: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-04-01&endday=2019-04-07&tree=all

Comment 7

[Intermittent Failures Robot]

25 failures in 3025 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 25

Platform breakdown:

• windows10-64-ccov: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-04-08&endday=2019-04-14&tree=all

Comment 8

[:Gijs (he/him)]

Jim, this test says test failure should be a release blocker. Is that true for win64-ccov?

Comment 9

[Jim Mathies [:jimm]]

If the failure is clang ccov build only then yes I think we can disable it. I poked around clang docs for some information on what they do exactly, didn't find much. But I'm guessing this could slow down execution / much with timing and cause something to load before the dll interceptor kicks in, which would manifest as this failure. Normal builds should still keep this test.

Comment 10

[Intermittent Failures Robot]

23 failures in 2673 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 22
• try: 1

Platform breakdown:

• windows10-64-ccov: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-04-15&endday=2019-04-21&tree=all

Comment 11

[Julien Cristau [:jcristau]]

Attachment: Bug 1531789 - skip browser_checkdllblockliststate.js on win-ccov. r?jmathies

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e8b8985aa697
skip browser_checkdllblockliststate.js on win-ccov. r=jmathies

Comment 13

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/e8b8985aa697

Comment 14

[Intermittent Failures Robot]

14 failures in 3031 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 12
• try: 2

Platform breakdown:

• windows10-64-ccov: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-04-22&endday=2019-04-28&tree=all

Comment 15

[Natalia Csoregi [:nataliaCs]]

R/b on two bc chunks:

• https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cshippable%2Copt%2Cmochitests%2Ctest-windows7-32-shippable%2Fopt-mochitest-browser-chrome-e10s-6%2Cm%28bc6%29&tochange=505173c26e1af4afcb7ee13a567a5ae4d8149dc7&fromchange=58fced48cd5dcf0f87c1d8345e011f6961af78cf
• https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C7%2Cdebug%2Cmochitests%2Ctest-windows7-32%2Fdebug-mochitest-browser-chrome-e10s-7%2Cm%28bc7%29&tochange=2799cfb271ca258e9087a16601dd16561bd4a3f3&fromchange=ff23369d071869072c56646034cd0411cb346a85

Comment 16

[:Gijs (he/him)]

Re-pinging Jim based on this restarting, now on non-ccov builds (including pgo ones, cf. https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=255495055&resultStatus=testfailed%2Cbusted%2Cexception&revision=2799cfb271ca258e9087a16601dd16561bd4a3f3&searchStr=windows ) so timeouts/slowness seem less likely as a possible cause.

Comment 17

[Intermittent Failures Robot]

59 failures in 1090 pushes (0.054 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 52
• mozilla-central: 3
• try: 2
• mozilla-inbound: 2

Platform breakdown:

• windows7-32: 27
• windows7-32-shippable: 32

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-07-09&endday=2019-07-09&tree=all

Comment 18

[Andreea Pavel [:apavel]]

Based on retrigger results from Natalia's comment, this is a regression of bug 1557564

Comment 19

[Andreea Pavel [:apavel]]

Joel, do we backout here?

Comment 20

[Joel Maher ( :jmaher ) (UTC -8)]

I would back out, this is the path we want to take for all regressions that are perma fail or high frequency failures where a root cause is identified.

Comment 21

[Andreea Pavel [:apavel]]

The recent occurences here are fixed by https://hg.mozilla.org/integration/autoland/rev/07df7588275399c0008f811069f23f2c97c9b084

Comment 23

[Intermittent Failures Robot]

40 failures in 1024 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23
• oak: 2
• mozilla-central: 5
• try: 8
• mozilla-inbound: 2

Platform breakdown:

• windows7-32: 27
• windows7-32-shippable: 11
• windows7-32-devedition: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-07-10&endday=2019-07-10&tree=all

Comment 24

[Intermittent Failures Robot]

108 failures in 4710 pushes (0.023 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 75
• mozilla-central: 8
• try: 19
• mozilla-inbound: 4
• oak: 2

Platform breakdown:

• windows7-32: 59
• windows7-32-shippable: 47
• windows7-32-devedition: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1531789&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 25

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

I think Jim is a more appropriate owner of this one, as he approved Julien's fix in April. (I only retriggered it recently from bug 1557564, but that was backed-out over there.)