Open Bug 1430530 Opened 2 years ago Updated 4 months ago

Intermittent gfx/tests/mochitest/test_acceleration.html | Advanced Layers enabled on Windows; Direct3D 11:false - didn't expect +0, but got it

Categories

(Core :: Graphics, defect, P3)

Unspecified
Windows 10
defect

Tracking

()

People

(Reporter: intermittent-bug-filer, Unassigned, NeedInfo)

References

Details

(Keywords: intermittent-failure, leave-open, Whiteboard: [gfx-noted][stockwell disabled])

Attachments

(1 file, 4 obsolete files)

Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=156299928&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/BrCUizC5R9miM0u-5YMHKg/runs/0/artifacts/public/logs/live_backing.log

06:19:13     INFO -  71 INFO TEST-PASS | gfx/tests/mochitest/test_acceleration.html | DirectWrite enabled on Windows 7 or newer
3448
06:19:13     INFO -  Buffered messages finished
3449
06:19:13    ERROR -  72 INFO TEST-UNEXPECTED-FAIL | gfx/tests/mochitest/test_acceleration.html | Advanced Layers enabled on Windows; Direct3D 11:false - didn't expect +0, but got it
3450
06:19:13     INFO -      SimpleTest.isnot@SimpleTest/SimpleTest.js:324:5
3451
06:19:13     INFO -      runTest@gfx/tests/mochitest/test_acceleration.html:135:9
3452
06:19:13     INFO -      EventListener.handleEvent*@gfx/tests/mochitest/test_acceleration.html:25:1
Looks like Advanced Layers isn't being enabled for some reason, perhaps a problem initializing D3D11.

> 06:19:07     INFO -  GECKO(10788) | [GPU 220, Compositor] WARNING: No device or incompatible device.: file z:/build/build/src/gfx/layers/d3d11/TextureD3D11.cpp, line 840
OS: Unspecified → Windows 10
Whiteboard: [gfx-noted]
There are 33 failures in the past week.
Platforms: windows10-64 opt/pgo and one occurrence on windows10-64-nightly opt.

Recent log failure: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=176866429&lineNumber=2106

Relevant part of the log: 23:48:20    ERROR -  63 INFO TEST-UNEXPECTED-FAIL | gfx/tests/mochitest/test_acceleration.html | Advanced Layers enabled on Windows; Direct3D 11:false - didn't expect +0, but got it
23:48:20     INFO -      SimpleTest.isnot@SimpleTest/SimpleTest.js:324:5
23:48:20     INFO -      runTest@gfx/tests/mochitest/test_acceleration.html:136:9
23:48:20     INFO -      EventListener.handleEvent*@gfx/tests/mochitest/test_acceleration.html:25:1
23:48:20     INFO -  64 INFO TEST-PASS | gfx/tests/mochitest/test_acceleration.html | WebRender disabled on Windows
23:48:20     INFO -  GECKO(5248) | MEMORY STAT | vsize 2098644MB | vsizeMaxContiguous 128450632MB | residentFast 76MB | heapAllocated 14MB
23:48:20     INFO -  65 INFO TEST-OK | gfx/tests/mochitest/test_acceleration.html | took 1157ms
23:48:20     INFO -  66 INFO TEST-START | Shutdown
23:48:20     INFO -  67 INFO Passed:  4
23:48:20  WARNING -  68 INFO Failed:  1
23:48:20  WARNING -  One or more unittests failed.
23:48:20     INFO -  69 INFO Todo:    0
23:48:20     INFO -  70 INFO Mode:    e10s
23:48:20     INFO -  71 INFO Slowest: 1158ms - /tests/gfx/tests/mochitest/test_acceleration.html
23:48:20     INFO -  72 INFO SimpleTest FINISHED
23:48:20     INFO -  73 INFO TEST-INFO | Ran 1 Loops
23:48:20     INFO -  74 INFO SimpleTest FINISHED

:milan Can you please take a look at this?
Flags: needinfo?(milaninbugzilla)
I see this at the top:'gpu_required': 'False'; could that mean that we sometimes get an instance that doesn't have the GPU and then things go bad?
Flags: needinfo?(milaninbugzilla) → needinfo?(nerli)
Over the last 7 days there are 43 failures present on this bug. These happen only on windows10-64.

Here is the most recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=177610744&lineNumber=2114

06:53:53    ERROR -  63 INFO TEST-UNEXPECTED-FAIL | gfx/tests/mochitest/test_acceleration.html | Advanced Layers enabled on Windows; Direct3D 11:false - didn't expect +0, but got it
06:53:53     INFO -      SimpleTest.isnot@SimpleTest/SimpleTest.js:324:5
06:53:53     INFO -      runTest@gfx/tests/mochitest/test_acceleration.html:136:9
06:53:53     INFO -      EventListener.handleEvent*@gfx/tests/mochitest/test_acceleration.html:25:1
06:53:53     INFO -  64 INFO TEST-PASS | gfx/tests/mochitest/test_acceleration.html | WebRender disabled on Windows
06:53:53     INFO -  GECKO(5452) | MEMORY STAT | vsize 2098643MB | vsizeMaxContiguous 129138263MB | residentFast 76MB | heapAllocated 14MB
06:53:53     INFO -  65 INFO TEST-OK | gfx/tests/mochitest/test_acceleration.html | took 1180ms
06:53:53     INFO -  66 INFO TEST-START | Shutdown
Flags: needinfo?(dbolter)
Flags: needinfo?(dbolter)
Priority: P3 → P5
Over the last 7 days this bug has 45 failures. These happen on windows7-32, windows10-64-nightly, windows10-64.

Here is the most recent log example: https://treeherder.mozilla.org/logviewer.html#?job_id=178891564&repo=mozilla-inbound&lineNumber=2109

00:39:37     INFO -  58 INFO SimpleTest START
00:39:37     INFO -  59 INFO TEST-START | gfx/tests/mochitest/test_acceleration.html
00:39:38     INFO -  TEST-INFO | started process screenshot
00:39:38     INFO -  TEST-INFO | screenshot: exit 0
00:39:38     INFO -  Buffered messages logged at 00:39:38
00:39:38     INFO -  60 INFO TEST-PASS | gfx/tests/mochitest/test_acceleration.html | Acceleration enabled on Windows XP or newer
00:39:38     INFO -  61 INFO TEST-PASS | gfx/tests/mochitest/test_acceleration.html | Direct2D enabled on Windows 8 or newer
00:39:38     INFO -  62 INFO TEST-PASS | gfx/tests/mochitest/test_acceleration.html | DirectWrite enabled on Windows 7 or newer
00:39:38     INFO -  Buffered messages finished
00:39:38    ERROR -  63 INFO TEST-UNEXPECTED-FAIL | gfx/tests/mochitest/test_acceleration.html | Advanced Layers enabled on Windows; Direct3D 11:false - didn't expect +0, but got it
00:39:38     INFO -      SimpleTest.isnot@SimpleTest/SimpleTest.js:324:5
00:39:38     INFO -      runTest@gfx/tests/mochitest/test_acceleration.html:136:9
00:39:38     INFO -      EventListener.handleEvent*@gfx/tests/mochitest/test_acceleration.html:25:1
00:39:38     INFO -  64 INFO TEST-PASS | gfx/tests/mochitest/test_acceleration.html | WebRender disabled on Windows
00:39:38     INFO -  GECKO(6708) | MEMORY STAT | vsize 2098643MB | vsizeMaxContiguous 129965714MB | residentFast 76MB | heapAllocated 14MB
00:39:38     INFO -  65 INFO TEST-OK | gfx/tests/mochitest/test_acceleration.html | took 1162ms
00:39:38     INFO -  66 INFO TEST-START | Shutdown
Flags: needinfo?(dbolter)
I'm curious about Milan's comment 14. Could this be related to our test infra being flaky about reporting a GPU?
Flags: needinfo?(dbolter) → needinfo?(bas)
Hi guys! 
There are 38 failures in the last 7 days. Mostly on windows10-64, only 1 on windows7-32.
Do you have any updates on this, should we go ahead with disabling the test?
Flags: needinfo?(dbolter)
(In reply to Narcis Beleuzu [:NarcisB] from comment #22)
> Hi guys! 
> There are 38 failures in the last 7 days. Mostly on windows10-64, only 1 on
> windows7-32.
> Do you have any updates on this, should we go ahead with disabling the test?

This test is pretty important, we've had people accidentally 'switch off' acceleration on automation in the past, and that's why we introduced it. Disabling is probably risky.

(In reply to David Bolter [:davidb] (NeedInfo me for attention) from comment #19)
> I'm curious about Milan's comment 14. Could this be related to our test
> infra being flaky about reporting a GPU?

It's either that, or somehow some of our test hardware isn't getting acceleration, I wonder whether the drivers our testing hardware is running are inconsistent, and whether we've recently perhaps blacklisted a driver used on some 'subset' of the machines.

This seems to be only reporting a failure to get Advanced Layers, now we recently did blocklist an NVidia driver range for advanced layers specifically. I can't seem to find a device ID or driver version for https://perf-html.io/public/b7050df4af0bb9e4e093d1eae1886b371df22b5c/calltree/?range=12.8214_13.2717&thread=6&threadOrder=0-2-3-4-6-1-5&v=3. Nor am I sure how to get there, but something tells me it might be in the blacklisted range!
Flags: needinfo?(bas)
DBurns do you know who can look into comment 23 regarding gfx driver versions on test hardware?
Flags: needinfo?(dbolter) → needinfo?(dburns)
:fubar, can you help figure out what graphics drivers we have on windows 10.  These are GPU instance types, specifically g2.2xlarge.

on another note, the failures show up on try- why are people annotating failures on try?  we should remove that from our bugzilla magic reports.
Flags: needinfo?(dburns) → needinfo?(klibby)
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #25)
> why are people annotating
> failures on try?

I annotate most of my try pushes, it's the best way I've found to determine if an orange-laden push is just full of intermittents or actually exposing bugs in the patches.
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #25)
> :fubar, can you help figure out what graphics drivers we have on windows 10.
> These are GPU instance types, specifically g2.2xlarge.

Swapping NI to markco to dig that info up.
Flags: needinfo?(klibby) → needinfo?(mcornmesser)
Nvidia 369.95-quadro-grid-desktop-notebook-win10-64bit
Flags: needinfo?(mcornmesser)
Looks like we're not blacklisting Windows 10 anyway, but even if we were, that driver version is not in the blacklist. As far as I can tell the only option we'd have here is add more annotation to the Advanced Layers decision making process and seeing why on some runs it decides not to enable it.
Flags: needinfo?(jmaher)
Attachment #8981417 - Flags: review?(jmaher)
Comment on attachment 8981417 [details] [diff] [review]
Skipped test_acceleration.html for frequent failures on win10 !debug

Review of attachment 8981417 [details] [diff] [review]:
-----------------------------------------------------------------

::: gfx/tests/mochitest/mochitest.ini
@@ +5,2 @@
>  subsuite = gpu
>  fail-if = (os == "win" && os_version == "5.1" && e10s) # Bug 1253862

can you remove this fail-if line since you are in here?

1) I don't know if skip-if and fail-if can coexist in the same test.
2) we don't run winxp (os_version == "5.1") anymore
Attachment #8981417 - Flags: review?(jmaher) → review-
Flags: needinfo?(jmaher)
Attachment #8981417 - Attachment is obsolete: true
Attachment #8981435 - Flags: review?(jmaher)
Comment on attachment 8981435 [details] [diff] [review]
Skipped test_acceleration.html for frequent failures on win10 !debug

Review of attachment 8981435 [details] [diff] [review]:
-----------------------------------------------------------------

sorry for the next r-, each line is important!

::: gfx/tests/mochitest/mochitest.ini
@@ -2,5 @@
>  
>  [test_acceleration.html]
>  skip-if = (os == 'win' && os_version == '10.0' && !debug) # Bug 1430530
> -subsuite = gpu
> -fail-if = (os == "win" && os_version == "5.1" && e10s) # Bug 1253862

we still need |subsuite = gpu|
Attachment #8981435 - Flags: review?(jmaher) → review-
Attachment #8981435 - Attachment is obsolete: true
Attachment #8981439 - Flags: review?(jmaher)
Attachment #8981439 - Flags: review?(jmaher)
Attachment #8981439 - Attachment is obsolete: true
Attachment #8981475 - Flags: review?(jmaher)
Comment on attachment 8981475 [details] [diff] [review]
Skipped test_acceleration.html for frequent failures on win10 !debug

Review of attachment 8981475 [details] [diff] [review]:
-----------------------------------------------------------------

excellent
Attachment #8981475 - Flags: review?(jmaher) → review+
Whiteboard: [gfx-noted][stockwell disable-recommended] → [gfx-noted][stockwell disabled]
Pushed by csabou@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4e1b975f4897
Disable test_acceleration.html for frequent failures on windows 10 !debug. r=jmaher
Keywords: checkin-needed
Flags: needinfo?(nerli)
There are 110 failures in the last 7 days.
All of them are on Win7 & Win10 - debug

Latest log: https://treeherder.mozilla.org/logviewer.html#?job_id=194658379&repo=mozilla-inbound&lineNumber=2954

03:53:47     INFO - TEST-START | gfx/tests/mochitest/test_acceleration.html
03:53:47     INFO - GECKO(1172) | ++DOMWINDOW == 7 (0000025831C9B400) [pid = 10396] [serial = 7] [outer = 0000025834020000]
03:53:48     INFO - GECKO(1172) | ++DOMWINDOW == 8 (00000258340A6000) [pid = 10396] [serial = 8] [outer = 0000025834020000]
03:53:48     INFO - GECKO(1172) | [Child 10396, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file z:/build/build/src/xpcom/base/nsSystemInfo.cpp, line 119
03:53:48     INFO - TEST-INFO | started process screenshot
03:53:48     INFO - TEST-INFO | screenshot: exit 0
03:53:48     INFO - Buffered messages logged at 03:53:48
03:53:48     INFO - TEST-PASS | gfx/tests/mochitest/test_acceleration.html | Acceleration enabled on Windows XP or newer 
03:53:48     INFO - TEST-PASS | gfx/tests/mochitest/test_acceleration.html | Direct2D enabled on Windows 8 or newer 
03:53:48     INFO - TEST-PASS | gfx/tests/mochitest/test_acceleration.html | DirectWrite enabled on Windows 7 or newer 
03:53:48     INFO - Buffered messages finished
03:53:48     INFO - TEST-UNEXPECTED-FAIL | gfx/tests/mochitest/test_acceleration.html | Advanced Layers enabled on Windows; Direct3D 11:false - didn't expect +0, but got it
03:53:48     INFO -     SimpleTest.isnot@SimpleTest/SimpleTest.js:324:5
03:53:48     INFO -     runTest@gfx/tests/mochitest/test_acceleration.html:135:9
03:53:48     INFO -     EventListener.handleEvent*@gfx/tests/mochitest/test_acceleration.html:25:1
03:53:48     INFO - TEST-PASS | gfx/tests/mochitest/test_acceleration.html | WebRender disabled on Windows 
03:53:48     INFO - GECKO(1172) | MEMORY STAT | vsize 1533MB | vsizeMaxContiguous 131720051MB | residentFast 85MB | heapAllocated 12MB
03:53:48     INFO - TEST-OK | gfx/tests/mochitest/test_acceleration.html | took 1407ms
Flags: needinfo?(dbolter)
Whiteboard: [gfx-noted][stockwell disabled] → [gfx-noted][stockwell disabled][stockwell needswork:owner]
Duplicate of this bug: 1484568
Due to the high frequency of failures let's disable this on all windows.
Attachment #8981475 - Attachment is obsolete: true
Attachment #9003146 - Flags: review?(jmaher)
Comment on attachment 9003146 [details] [diff] [review]
disable_on_Win.patch

Review of attachment 9003146 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks Cosmin
Attachment #9003146 - Flags: review?(jmaher) → review+
Pushed by csabou@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/349025ae0ac8
Disable gfx/tests/mochitest/test_acceleration.html on windows for frequent failures. r=jmaher
Jeff should we worry about the tests being disabled?
Flags: needinfo?(dbolter) → needinfo?(jmuizelaar)
Yes. We should worry about this. We want this to in place to ensure that we're testing the right things. Someone should probably investigate.
Flags: needinfo?(jmuizelaar) → needinfo?(dbolter)
NI Matt for awareness. It feels like we have higher priorities but tests are important.
Flags: needinfo?(dbolter) → needinfo?(matt.woodrow)
Whiteboard: [gfx-noted][stockwell disabled][stockwell needswork:owner] → [gfx-noted][stockwell disabled]
You need to log in before you can comment on or make changes to this bug.