1763680 - No video decode hardware acceleration [on twitch] since ~100.0b2

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P2)

Description

[Tomasz Sobczyk]

Attachment: mozregression bisection. Also see https://bugzilla.mozilla.org/show_bug.cgi?id=1763311#c5

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0

Steps to reproduce:

Only h264 hardware decoding accelerator present (Nvidia GTX 750). Watch any twitch.tv stream. (media.hardware-video-decoding.enabled=true, media.hardware-video-decoding.failed=false, media.hardware-video-decoding.force-enabled=false (also checked true), media.av1.enabled=true (also tried false), media.av1.use-dav1d=true (also tried false), media.wmf.av1.enabled=true (also tried false))

Actual results:

GPU's video engine not being used (as reported by GPU-Z); CPU usage much higher than normal. Is used in other browsers (opera) and was fine before 100.0b update.

Expected results:

Video decode engine should be used.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Tomasz Sobczyk]

Attachment: video playback stats (confirming the video is h264)

Comment 3

[Alastor Wu [:alwu]]

Could you help me try the latest Nightly to see if this issue still exists? Bug 1760804 might solve this issue already.
Thank you.

Comment 4

[Tomasz Sobczyk]

Still wrong behaviour in 101.0a in Firefox Nightly. Was unable to find a good build with mozregression.

Comment 5

[Tomasz Sobczyk]

https://bugzilla.mozilla.org/show_bug.cgi?id=1762125 might be related

Comment 6

[Jeff Muizelaar [:jrmuizel]]

Can you attach the graphics section of about:support?

Comment 7

[Jeff Muizelaar [:jrmuizel]]

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1758605

Comment 9

[Tomasz Sobczyk]

Attachment: about:support Graphics section (Polish)

(In reply to Jeff Muizelaar [:jrmuizel] from comment #6)
> Can you attach the graphics section of about:support?

Comment 10

[Tomasz Sobczyk]

Attachment: about:support, graphics

(In reply to Jeff Muizelaar [:jrmuizel] from comment #6)
> Can you attach the graphics section of about:support?

Comment 11

[Tomasz Sobczyk]

Not sure what the best way of getting these in english is, hopefully language is not a problem.

Comment 12

[Jeff Muizelaar [:jrmuizel]]

Zaggy1024, what are the implications of backing out bug 1758605 from beta?

Comment 13

[Zaggy1024]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #12)

Zaggy1024, what are the implications of backing out bug 1758605 from beta?

There should be no issue with backing that out, but I doubt that's the cause of the issue. I would have to guess it was due to the changes made in Bug 1652945 to how WMF creates its decoders.

Tomasz, could you try to obtain logs according to the instructions in this comment? It's also possible that you will need to change "security.sandbox.content.level" to 0 and restart to get the logs to write (see https://firefox-source-docs.mozilla.org/xpcom/logging.html#enabling-logging), although if you run from cmd it shouldn't be necessary in my experience. This will hopefully give us an idea of where the decoder initialization is failing.

Also, your full about:support may help narrow down the causes, although the specific sections I'm interested in would be "Application Basics" and "Important Modified Preferences" if you'd rather not share the whole document.

Comment 14

[Tomasz Sobczyk]

Attachment: more from about:support

Comment 15

[Tomasz Sobczyk]

(In reply to Zaggy1024 from comment #13)

(In reply to Jeff Muizelaar [:jrmuizel] from comment #12)

Zaggy1024, what are the implications of backing out bug 1758605 from beta?

There should be no issue with backing that out, but I doubt that's the cause of the issue. I would have to guess it was due to the changes made in Bug 1652945 to how WMF creates its decoders.

Tomasz, could you try to obtain logs according to the instructions in this comment? It's also possible that you will need to change "security.sandbox.content.level" to 0 and restart to get the logs to write (see https://firefox-source-docs.mozilla.org/xpcom/logging.html#enabling-logging), although if you run from cmd it shouldn't be necessary in my experience. This will hopefully give us an idea of where the decoder initialization is failing.

I'm unable to create a non-empty log file. I tried setting "security.sandbox.content.level" to 0, running cmd as admin. I will only have more time to spend on this tomorrow.

Comment 16

[Tomasz Sobczyk]

Attachment: firefox_log.zip (launch nightly 101.0a1, navigate some twitch.tv stream directly, close firefox after ~5 seconds)

I was able to generate the log. The issue I had is outlined here https://bugzilla.mozilla.org/show_bug.cgi?id=1364842#c4. Had to use nightly.

Comment 17

[Zaggy1024]

(In reply to Tomasz Sobczyk from comment #16)

Created attachment 9271596 [details]
firefox_log.zip (launch nightly 101.0a1, navigate some twitch.tv stream directly, close firefox after ~5 seconds)

I was able to generate the log. The issue I had is outlined here https://bugzilla.mozilla.org/show_bug.cgi?id=1364842#c4. Had to use nightly.

I'm glad you were able to figure that out, thank you for linking the solution! For some reason I had thought I got logs to write in beta, but maybe I was wrong.

It looks like it's attempting to use DXVA, and if I'm correct that you're on Windows 7, it will be trying to use the D3D9 implementation. Unfortunately, it currently doesn't log where the DXVA implementation failed to initialize, so I'll submit a patch to add that so we can narrow down the cause further.

I'm not sure where the cause would likely be, since the H264 decoder MFT it uses should still be the exact same one as before AV1 decoding was introduced. I don't believe there are any other changes that should have affected it, but I may be wrong.

Comment 18

[Zaggy1024]

Attachment: Bug 1763680 - Part 1 - Add logging of DXVA decoding failure reasons to help narrow down causes of hardware acceleration failure. r=alwu

Depends on D141720

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/68722c984f71
Part 1 - Add logging of DXVA decoding failure reasons to help narrow down causes of hardware acceleration failure. r=media-playback-reviewers,alwu

Comment 20

[Zaggy1024]

Looks like the optimized Windows builds have finished for that logging patch, if you want to grab "Windows 2012 x64 opt B" -> "Artifacts and Debugging Tools" -> target.zip (which is just a portable build) from here to get a Media profile which will contain information about where the hardware decoder is failing:
https://treeherder.mozilla.org/jobs?repo=autoland&revision=68722c984f71c9adb23fba5405a47c5f05a9266b&selectedTaskRun=IBoOCktrTsahIcYvVUtKZA.0

Alternatively, you could wait for it to land in Nightly and do the same there.

Comment 21

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/68722c984f71

Comment 22

[Zaggy1024]

The bug is still present, the patch has been landed to get more information about what's causing the issue.

Comment 23

[Tomasz Sobczyk]

Attachment: firefox_log_patch1.zip

(In reply to Zaggy1024 from comment #20)

Looks like the optimized Windows builds have finished for that logging patch, if you want to grab "Windows 2012 x64 opt B" -> "Artifacts and Debugging Tools" -> target.zip (which is just a portable build) from here to get a Media profile which will contain information about where the hardware decoder is failing:
https://treeherder.mozilla.org/jobs?repo=autoland&revision=68722c984f71c9adb23fba5405a47c5f05a9266b&selectedTaskRun=IBoOCktrTsahIcYvVUtKZA.0

Alternatively, you could wait for it to land in Nightly and do the same there.

attaching new logs. The relevant bit probably being:

2022-04-12 08:48:17.316000 UTC - [GPU 676: Main Thread]: D/PlatformDecoderModule PDMInitializer, Init PDMs in GPU process
2022-04-12 08:48:17.316000 UTC - [GPU 676: Main Thread]: D/PlatformDecoderModule sDXVAEnabled: true, CanUseHardwareVideoDecoding: false
2022-04-12 08:48:17.321000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule H264 is enabled
2022-04-12 08:48:17.321000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule VP8 MFT requires Windows build 16287 or later
2022-04-12 08:48:17.321000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule VP9 MFT requires DXVA
2022-04-12 08:48:17.321000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule AV1 MFT requires DXVA
2022-04-12 08:48:17.322000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule MP3 is enabled
2022-04-12 08:48:17.326000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule AAC is enabled
2022-04-12 08:48:17.326000 UTC - [GPU 676: Main Thread]: D/PlatformDecoderModule WMFDecoderModule::Init finishing
2022-04-12 08:48:17.379000 UTC - [GPU 676: Main Thread]: D/PlatformDecoderModule sDXVAEnabled: true, CanUseHardwareVideoDecoding: true
2022-04-12 08:48:17.380000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule H264 is enabled
2022-04-12 08:48:17.380000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule VP8 MFT requires Windows build 16287 or later
2022-04-12 08:48:17.381000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule VP9 failed with code 0x80040154
2022-04-12 08:48:17.382000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule AV1 failed with code 0x88982f50
2022-04-12 08:48:17.382000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule MP3 is enabled
2022-04-12 08:48:17.383000 UTC - [GPU 676: COM MTA]: D/PlatformDecoderModule AAC is enabled
2022-04-12 08:48:17.383000 UTC - [GPU 676: Main Thread]: D/PlatformDecoderModule WMFDecoderModule::Init finishing
2022-04-12 08:48:32.017000 UTC - [GPU 676: MediaSupervisor #1]: D/PlatformDecoderModule Send message 'MFT_MESSAGE_SET_D3D_MANAGER'
2022-04-12 08:48:32.018000 UTC - [GPU 676: MediaSupervisor #1]: D/PlatformDecoderModule Send message 'MFT_MESSAGE_NOTIFY_BEGIN_STREAMING'
2022-04-12 08:48:32.041000 UTC - [GPU 676: MediaSupervisor #1]: D/PlatformDecoderModule Send message 'MFT_MESSAGE_NOTIFY_START_OF_STREAM'
2022-04-12 08:48:32.043000 UTC - [GPU 676: MediaSupervisor #1]: D/PlatformDecoderModule DXVA failure: Hardware video decoding disabled or blacklisted
2022-04-12 08:48:32.044000 UTC - [GPU 676: MediaSupervisor #1]: D/PlatformDecoderModule Send message 'MFT_MESSAGE_NOTIFY_BEGIN_STREAMING'
2022-04-12 08:48:32.047000 UTC - [GPU 676: MediaSupervisor #1]: D/PlatformDecoderModule Send message 'MFT_MESSAGE_NOTIFY_START_OF_STREAM'
2022-04-12 08:48:32.047000 UTC - [GPU 676: MediaSupervisor #1]: D/PlatformDecoderModule Video Decoder initialized, Using DXVA: No
2022-04-12 08:48:32.047000 UTC - [GPU 676: MediaSupervisor #1]: D/PlatformDecoderModule WMFVideoMFTManager frame geometry stride=640 picture=(0, 0, 640, 360) display=(640,360)
2022-04-12 08:48:32.058000 UTC - [GPU 676: MediaSupervisor #1]: D/PlatformDecoderModule WMFDecoderModule::CreateVideoDecoder success for manager with description wmf H264 codec software video decoder - yuv420 - DXVA failure: Hardware video decoding disabled or blacklisted
2022-04-12 08:48:32.059000 UTC - [GPU 676: MediaPDecoder #1]: D/PlatformDecoderModule ProcessDecode, type=Video, sample=0

Comment 24

[Tomasz Sobczyk]

media profile: https://share.firefox.dev/3juHmQc

Comment 25

[Zaggy1024]

(In reply to Tomasz Sobczyk from comment #23)

attaching new logs. The relevant bit probably being:

Right, thank you for this! The profile also includes the DXVA failure reason, but these logs help a lot too.

2022-04-12 08:48:32.058000 UTC - [GPU 676: MediaSupervisor #1]: D/PlatformDecoderModule WMFDecoderModule::CreateVideoDecoder success for manager with description wmf H264 codec software video decoder - yuv420 - DXVA failure: Hardware video decoding disabled or blacklisted

Looks like another blocklist issue, sDXVAEnabled is set to true in WMFDecoderModule before the decoders are instantiated, so this indicates that !aOptions.contains(CreateDecoderParams::Option::HardwareDecoderNotAllowed) is failing again for the GTX 750. , Relevant about:support info is:

Device ID: 0x1381
Driver Version: 26.21.14.4122```

We also have another user with a GTX 750 Ti having issues with H264 on Bug 1762660, unconfirmed yet whether they are also getting blocklisted, but it seems likely. Their GPU information is:
Vendor ID: 0x10de
Device ID: 0x1380
Driver Version: 30.0.14.7304

:jrmuizel, I'm assuming you should take a look to find the culprit?

Comment 26

[Jeff Muizelaar [:jrmuizel]]

I don't think there's been any changes to the blocklist in the time period associated with the regression and the regression window suggests that this is a recent problem. Perhaps CanUseDXVA is failing now? https://searchfox.org/mozilla-central/rev/22a9cf9388cefa244ea2558efa748ce17a38d607/dom/media/platforms/wmf/WMFVideoMFTManager.cpp#586

Comment 27

[Zaggy1024]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #26)

Perhaps CanUseDXVA is failing now? https://searchfox.org/mozilla-central/rev/22a9cf9388cefa244ea2558efa748ce17a38d607/dom/media/platforms/wmf/WMFVideoMFTManager.cpp#586
Ahh, I think you're right, I missed that possible failure point. I'm not sure yet what changed, but I have some theories about where this may be failing.

Throwing together a patch to build to hopefully enable the hardware acceleration to work and otherwise log the cause of the issue.

Comment 28

[Jeff Muizelaar [:jrmuizel]]

Tomasz, can you try this build with bug 1758605 reverted to see if it resolves your problem?
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ODZT0QKhQtmeBAGlW19Isg/runs/0/artifacts/public/build/target.zip

Comment 29

[Zaggy1024]

Sorry to pile on another thing to try, but here is a build that should change the behavior of DXVA to match what it was previously:
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/HVaX1b7CQBuYuwbGjff3SA/runs/0/artifacts/public/build/target.zip
Diff here

Please also grab logs again with this build, since it also adds some logging to determine what's causing the failure. I'm hoping it'll be the last logs we need to grab to figure out what needs fixing.

Comment 30

[Zaggy1024]

Actually, come to think of it, I think I was wrong to think it was attempting to use D3D11. No need to try the build above, as it likely won't fix the issue.

It seems as though changes to how SupportsConfig is called for D3D9 broke the check, I'll submit a patch to fix that in a moment.

Comment 31

[Zaggy1024]

Attachment: Bug 1763680 - Allow D3D9DXVA2Manager to correctly determine hardware support for the H264 decoder. r=alwu

D3D9 hardware decoding detection was failing due to a change to provide DXVA2Manager::SupportsConfig with information about the compressed video format. Provide both input and output media types to allow it to correctly determine support.

Changed WMFVideoMFTManager logging to log when DXVA initialization fails for D3D11 but succeeds for D3D9.

Comment 32

[Tomasz Sobczyk]

The two linked builds above don't fix the issue. I could also test the proposed patch but I don't know how to get built binaries for it or if it's even possible.

Comment 33

[Zaggy1024]

You can! I would appreciate it. The build for that patch is here:
https://treeherder.mozilla.org/jobs?repo=try&revision=0d949d650fe18d70e399ddf62936c521bf7bf1c7
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/F7DE0bW5QFqDtuLYYxQ9Yg/runs/0/artifacts/public/build/target.zip

I forgot to mention here when that build finished, but it's good to go now, let me know if it solves the issue.

Comment 34

[Tomasz Sobczyk]

(In reply to Zaggy1024 from comment #33)

You can! I would appreciate it. The build for that patch is here:
https://treeherder.mozilla.org/jobs?repo=try&revision=0d949d650fe18d70e399ddf62936c521bf7bf1c7
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/F7DE0bW5QFqDtuLYYxQ9Yg/runs/0/artifacts/public/build/target.zip

I forgot to mention here when that build finished, but it's good to go now, let me know if it solves the issue.

solves the issue

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7bfc1aacfe2c
Allow D3D9DXVA2Manager to correctly determine hardware support for the H264 decoder. r=alwu

Comment 37

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/7bfc1aacfe2c

Comment 38

[Zaggy1024]

Comment on attachment 9271953 [details]
Bug 1763680 - Allow D3D9DXVA2Manager to correctly determine hardware support for the H264 decoder. r=alwu

Beta/Release Uplift Approval Request

• User impact if declined: Users on Windows 7 machines will not get hardware H264 decoding.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This change reverts behavior to match when H264 hardware decoding was working on the affected systems.
• String changes made/needed:

Comment 39

[Zaggy1024]

(In reply to Tomasz Sobczyk from comment #34)

solves the issue

Patch should be going out to Nightly, thank you for your help!

Comment 40

[Dianna Smith [:diannaS]]

Comment on attachment 9271626 [details]
Bug 1763680 - Part 1 - Add logging of DXVA decoding failure reasons to help narrow down causes of hardware acceleration failure. r=alwu

Approved for 100.0b7

Comment 41

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/fbf9af7ca2f7
https://hg.mozilla.org/releases/mozilla-beta/rev/a6a6c6ddfefa

Comment 42

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Dianna Smith [:diannaS] from comment #41)

https://hg.mozilla.org/releases/mozilla-beta/rev/fbf9af7ca2f7
https://hg.mozilla.org/releases/mozilla-beta/rev/a6a6c6ddfefa

:zaggy1024, makes sense the below improvement from beta?
== Change summary for alert #33877 (as of Wed, 20 Apr 2022 14:54:20 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
6%	imdb fcp	linux1804-64-shippable-qr	fission warm webrender	184.70 -> 172.92
5%	ebay fnbpaint	windows10-64-shippable-qr	fission warm webrender	91.93 -> 87.38
5%	youtube dcf	linux1804-64-shippable-qr	cold fission webrender	1,352.04 -> 1,289.67
5%	imdb LastVisualChange	linux1804-64-shippable-qr	fission warm webrender	4,871.67 -> 4,650.00
4%	youtube dcf	linux1804-64-shippable-qr	fission warm webrender	1,288.41 -> 1,242.38
...	...	...	...	...
3%	yahoo-mail LastVisualChange	linux1804-64-shippable-qr	fission warm webrender	1,071.30 -> 1,040.00

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=33877

Comment 43

[Zaggy1024]

I can't think of any reason these changes would affect tests on Linux, could these improvements be attributed to any other revisions?

For Windows, if the improvement is consistent, it's possible that enough time was being wasted trying to test for H264 decoder support in D3D9 to affect first paint, but it does seem strange even there. Assuming that the D3D9 APIs aren't particularly slow, I wouldn't expect that to be a significant difference.