Closed Bug 1865928 Opened 1 year ago Closed 1 year ago

YouTube videos all green with hardware accel enabled in FF 120

Categories

(Core :: Graphics: WebRender, defect)

Firefox 120
Desktop
Windows 10
defect

Tracking

()

RESOLVED FIXED
122 Branch
Tracking Status
relnote-firefox --- 120+
firefox-esr115 --- unaffected
firefox120 + fixed
firefox121 + fixed
firefox122 + fixed

People

(Reporter: rhpuckett3, Assigned: sotaro)

References

(Regression, )

Details

(Keywords: regression)

Attachments

(12 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0

Steps to reproduce:

Updated to FF 120.
Created new profile, no add-ons, HWA enabled (default)
Navigated to YouTube page: https://www.youtube.com/watch?v=HeQX2HjkcNo
Tested with HWA enabled and disabled.

Actual results:

Video displayed as all green, sound normal. (see attached)
Note: HWA enabled.
Issue did NOT occur on Twitter/X or Instagram.

Expected results:

Video should have display correctly. (see attached)
Note: HWA disabled.

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Audio/Video: Playback
Product: Firefox → Core
Attached file about-support.txt

Information from about:support

Profiling information using Graphics setting: https://share.firefox.dev/47oydQm

Summary: YouTube videos all with hardware accel enabled in FF 120 → YouTube videos all green with hardware accel enabled in FF 120

We have received multiple reports on webcompat.com for this issue as well, however I was not able to reproduce this issue.

I don't know if this helps with anything, but I noticed that even when displaying all green, the (white) Veritasium logo is still visible in the lower right corner of the video window in the screen shot(s) I provided.

Blocks: gfx-triage
Component: Audio/Video: Playback → Graphics: WebRender
See Also: → 1866084

The component has been changed since the backlog priority was decided, so we're resetting it.
For more information, please visit BugBot documentation.

Priority: P3 → --

Reported as being fixed in 122 in bug 1864618

Duplicate of this bug: 1864618
See Also: → 1864307

Hello, Rick,
Would you mind to help us capture a media playback profiled result?

Flags: needinfo?(rhpuckett3)

I checked the bug landed between Fx120 to Fx122 [1], I only see one bug (1853024) related with HW decoding, but that one shouldn't affect Windows. Sotaro, I wonder if you're aware of any possible gfx changes which can cause this?

Thank you.

[1]
https://shorturl.at/nwEIU
https://shorturl.at/cjmoP
https://shorturl.at/ghNS8

Flags: needinfo?(sotaro.ikeda.g)

(In reply to Alastor Wu [:alwu] from comment #10)

Hello, Rick,
Would you mind to help us capture a media playback profiled result?

I uploaded profiling data using the Graphics setting with my initial batch info (above) through, I think, 10s of green playback (link also below). Did you need something else or a different setting...? If so, let me know.

https://share.firefox.dev/47oydQm

Flags: needinfo?(rhpuckett3)

(In reply to Rick Puckett from comment #12)

I uploaded profiling data using the Graphics setting with my initial batch info (above) through, I think, 10s of green playback (link also below). Did you need something else or a different setting...? If so, let me know.

https://share.firefox.dev/47oydQm

Yes, sorry I wasn't clear enough. Could you help me use media playback preset to capture a result? See instructions here.

Thank you.

Flags: needinfo?(rhpuckett3)

(In reply to Alastor Wu [:alwu] from comment #13)

(In reply to Rick Puckett from comment #12)

Yes, sorry I wasn't clear enough. Could you help me use media playback preset to capture a result? See instructions here.

Thank you.

Done. Logging preset: "Media playback", 15s of green video playback, all upload boxes checked.
Thanks for walking me through that. Let me know if I can help out further.

https://share.firefox.dev/47Rerg3

Flags: needinfo?(rhpuckett3)

Rick, can you try getting a regression window using mozregresion?

Flags: needinfo?(rhpuckett3)

Okay, thank you for the profiled result.

I see so many decode errors happened inside dav1 decoder, but that happened inside the RDD process, which means actually no hw decoding in use. In GPU process, I also see

LogMessages — (PlatformDecoderModule) DXVA failure: Failed to create D3D11 device for decoder; D3D9: D3D9DXVA2Manager is disabled on AMDPreUVD4

In the about:support provided in the comment3, I also noticed that HW_DECODED_VIDEO_ZERO_COPY and VIDEO_HARDWARE_OVERLAY are blocked by the gfx. So when you said the green frame issues was gone for disabling HWA, do you mean turning off the pref media.hardware-video-decoding.enabled?


In addition, could you help me verify following things?

  1. First, delete the pref media.hardware-video-decoding.failed
  2. Then ensure the media.hardware-video-decoding.enabled is true
  3. Go to any video on Youtube and right click on the video, select Stats for nerds
  4. On Stats for nerds, there would be a Codes column
  5. Go to more video on Youtube, I would like to know whether all those videos showing green frames are codec av1? or you would see green frames on h264 (like this video) or vp9 (like this video)

Thank you.

In addition, I noticed we have one dav1d update on Fx120 (bug 1846318) and one dav1d update on Fx121 (bug1860626). If the problem is caused by dav1d, then they could be the regression and solution. (if the issue is gone on Fx122)

(In reply to Jeff Muizelaar [:jrmuizel] from comment #15)

Rick, can you try getting a regression window using mozregresion?

Done, but I know it worked correctly with FF 119.0.1 as I used Youtube with that version the day before I upgraded to FF 120.

I ran a regression using a Good date of 2023-11-07 (when I downloaded 119.0.1) and a Bad date of 2023-11-21 (when I downloaded 120) -- I always keep a local copy of recent FF/TB install files. I'm not sure what/how to report the results here, but here goes and I'll leave up my regression instance for a while...

I used the same Youtube video (also initially reported): https://www.youtube.com/watch?v=HeQX2HjkcNo
I got 6 bad cases, the build dates and pushlog urls are below and I'll upload the contents of the Log window...
Some of the instances reported a crash while trying to display the video.

build_date: 2023-10-05:
pushlog_url: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=00ae001484c9ff1069878dc66b64e3cd07b706f4&tochange=11b3d438788a84aecc7e53efdda19bb0814e13be

build_date: 2023-10-05 11:24:22.117000
pushlog_url: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=c559095402a2380ba171d9c60662e36c8d5e948d&tochange=97d412cf7a31602d9485ebc549b56aed797523e9

build_date: 2023-10-05 10:04:48.435000
pushlog_url: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=c559095402a2380ba171d9c60662e36c8d5e948d&tochange=c5b63a02d71951f66853cbeb8a677986fc2ddf47

build_date: 2023-10-04 21:33:46.928000
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=8448ddef9c5e7e7bb18384ace2096839b67a4b49&tochange=7cc8baac415ebcf55e5bfe27795610c8f53bbbad

build_date: 2023-10-06 12:58:34.598000
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=8448ddef9c5e7e7bb18384ace2096839b67a4b49&tochange=7683643c809e606a8b8444a8ec6b39987d11fe24

build_date: 2023-10-06 13:02:42.131000
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=89c56ef7fc57cadc37df4482cffc21aefe726bf8&tochange=524595183106de9906df2a1b185b14ec69775549

Flags: needinfo?(rhpuckett3)
Attached file ff-regression-log.txt

mozregression log for FF 2023-11-07 (good) to FF 2023-11-21 (bad)

mozregression progress for FF 2023-11-07 (good) to FF 2023-11-21 (bad).

(In reply to Alastor Wu [:alwu] from comment #16)

Okay, thank you for the profiled result.

I see so many decode errors happened inside dav1 decoder, but that happened inside the RDD process, which means actually no hw decoding in use. In GPU process, I also see

LogMessages — (PlatformDecoderModule) DXVA failure: Failed to create D3D11 device for decoder; D3D9: D3D9DXVA2Manager is disabled on AMDPreUVD4

In the about:support provided in the comment3, I also noticed that HW_DECODED_VIDEO_ZERO_COPY and VIDEO_HARDWARE_OVERLAY are blocked by the gfx. So when you said the green frame issues was gone for disabling HWA, do you mean turning off the pref media.hardware-video-decoding.enabled?


In addition, could you help me verify following things?

  1. First, delete the pref media.hardware-video-decoding.failed
  2. Then ensure the media.hardware-video-decoding.enabled is true
  3. Go to any video on Youtube and right click on the video, select Stats for nerds
  4. On Stats for nerds, there would be a Codes column
  5. Go to more video on Youtube, I would like to know whether all those videos showing green frames are codec av1? or you would see green frames on h264 (like this video) or vp9 (like this video)

Thank you.

Okay...
0: When I mentioned HWA I meant un/checking "Settings->Use hardware acceleration when available."
1: Deleted 'media.hardware-video-decoding.failed'
2: Ensured: 'media.hardware-video-decoding.enabled' is 'true' (then restarted FF, rechecked values)
3: Went to same video, "https://www.youtube.com/watch?v=HeQX2HjkcNo", selected "Stats for nerds" ...
4: Stats for nerds shows "Codecs av01.0.04M.08 (397) / opus (251)" (will upload screenshot of nerds values).
5a: Video "bunny.mp4" displayed correctly, audio okay.
5b: Video "https://www.youtube.com/watch?v=cxeQmF4sUJY" displayed all green, audio okay.

Attached image youtube-nerd-stats.jpg
Severity: -- → S2

(In reply to Alastor Wu [:alwu] from comment #16)

I see so many decode errors happened inside dav1 decoder, but that happened inside the RDD process, which means actually no hw decoding in use. In GPU process, I also see

LogMessages — (PlatformDecoderModule) DXVA failure: Failed to create D3D11 device for decoder; D3D9: D3D9DXVA2Manager is disabled on AMDPreUVD4

In the about:support provided in the comment3, I also noticed that HW_DECODED_VIDEO_ZERO_COPY and VIDEO_HARDWARE_OVERLAY are blocked by the gfx.


Don't know if this helps, but I'll note that I submitted a bug previously, that was fixed, concerning bad video playback and FF/Windows crashing when playing a video on Twitter with HWA enabled for the video card on this system and "AMDPreUVD4" sounds familiar. I can't find the Bug ID, but I believe the resolution was to disable something when this video card is present-- "ATI Radeon HD 2600 XT" (Windows 10)

From about:support:
GPU #1
Active: Yes
Description: ATI Radeon HD 2600 XT
Vendor ID: 0x1002
Device ID: 0x9588
Driver Version: 8.970.100.9001
Driver Date: 1-13-2015
Drivers: aticfx64 aticfx64 aticfx32 aticfx32 atiumd64 atidxx64 atiumdag atidxx32 atiumdva atiumd6a atitmm64
Subsys ID: 25421028
RAM: 256

The bug has a release status flag that shows some version of Firefox is affected, thus it will be considered confirmed.

Status: UNCONFIRMED → NEW
Ever confirmed: true

Set release status flags based on info from the regressing bug 1856516

As already reported for bug 1864618 (which is probably a duplicate of 1865928), all is working again beginning with FF Nightly 122.0a1 (2023-11-23). Each Youtube video can be played again without this green screen. However, for bug 1864618, turning the hardware accelaration off had no effect. Anyway, whatever you have changed from FF Nightly 122.0a1 2023-11-20 to 2023-11-23, please add this fix asap to the normal FF builds. Thanks!

Rick, would you be able to see if things are working better for you as well with current Nightly builds? If so, a mozregression "find fix" run (basically running a bisection in reverse to find what fixed the problem instead of what caused it) would be immensely helpful in expediting a fix into a future release. Thanks!

Flags: needinfo?(rhpuckett3)

Sotaro, would reverting bug 1857385 on Release be another option for Fx120?

And for people testing on Fx120, can you confirm that setting the gfx.video.convert-yuv-to-nv12.image-host-win pref to false via about:config makes the problem go away?

(In reply to Ryan VanderMeulen [:RyanVM] from comment #30)

And for people testing on Fx120, can you confirm that setting the gfx.video.convert-yuv-to-nv12.image-host-win pref to false via about:config makes the problem go away?

Yes. Tried this in my test profile with FF 120 and video displayed green when 'true' and displayed correctly when 'false'.

Flags: needinfo?(rhpuckett3)

Hi, I have run mozregression between FF Nightly 2023-11-18 and 2023-11-24. Results: 2023-11-20 was bad, 2023-11-21 was good again. How can I forward you the full results?

Attached is the log text of my mozregression.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #28)

Rick, would you be able to see if things are working better for you as well with current Nightly builds? If so, a mozregression "find fix" run (basically running a bisection in reverse to find what fixed the problem instead of what caused it) would be immensely helpful in expediting a fix into a future release. Thanks!

Done. Ran mozregression find-fix from build 120 (last bad) to date 2023-11-28 (last good).
Still not sure which pushlog to report (will post screenshot) but last three are below:

status: bad
build_date: 2023-11-21 08:56:00.696000
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=f2558d4ebe41520695c83792fa6725511d388d7e&tochange=28d400b612e541c67c7c303113ab261f6463e848

status: good
build_date: 2023-11-21 08:25:21.514000
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=f2558d4ebe41520695c83792fa6725511d388d7e&tochange=4a6ed43174388497963a5e476b2b22e495ba088e

status: bad
build_date: 2023-11-21 09:07:00.188000
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=dde990208a4fa2791e4fc928b9155e3499e2a10f&tochange=4a6ed43174388497963a5e476b2b22e495ba088e

Mozregression find-fix progress from release 120 (last bad) to date 2023-11-28 (last good).

Interesting, both of your runs are pointing at bug 1860062 as what "fixed" this (or at least worked around the underlying problem). But that was reverted a day later for causing other problems. But things are still working on current Nightly builds?!?!

(In reply to Ryan VanderMeulen [:RyanVM] from comment #36)

Interesting, both of your runs are pointing at bug 1860062 as what "fixed" this (or at least worked around the underlying problem). But that was reverted a day later for causing other problems. But things are still working on current Nightly builds?!?!

Just curious.
Is this related to why setting 'gfx.video.convert-yuv-to-nv12.image-host-win' to 'false' does seem to "fix" the issue.
See my comment #31: https://bugzilla.mozilla.org/show_bug.cgi?id=1865928#c31

Yes, all newer Nightly builds still work fine. Just tested it with 122.0a1 (2023-11-28) (64-Bit).

(In reply to Rick Puckett from comment #37)

Just curious.
Is this related to why setting 'gfx.video.convert-yuv-to-nv12.image-host-win' to 'false' does seem to "fix" the issue.
See my comment #31: https://bugzilla.mozilla.org/show_bug.cgi?id=1865928#c31

That's the pref the initial regressing bug changed in Fx120 which first introduced the problem. So setting it back to false effectively reverted it back to the state it was in for previous releases. It's unclear at this point why the other change to the GPU sandboxing code caused the problem to disappear again. I'd also be curious to know if a Nightly from 2023-11-22 was green again (and then fixed again by another change).

(In reply to Ryan VanderMeulen [:RyanVM] from comment #39)

(In reply to Rick Puckett from comment #37)

Just curious.
Is this related to why setting 'gfx.video.convert-yuv-to-nv12.image-host-win' to 'false' does seem to "fix" the issue.
See my comment #31: https://bugzilla.mozilla.org/show_bug.cgi?id=1865928#c31

That's the pref the initial regressing bug changed in Fx120 which first introduced the problem. So setting it back to false effectively reverted it back to the state it was in for previous releases. It's unclear at this point why the other change to the GPU sandboxing code caused the problem to disappear again. I'd also be curious to know if a Nightly from 2023-11-22 was green again (and then fixed again by another change).

Thanks for that info.
I'd be happy to run a mozregression on that last sentence if you can specify the settings... (still new to using mozregression)
(Comment #38 notes that it works in 122.0a1.)

From the command line, it would be something along the lines of mozregression --find-fix --bad 42742c4655f65e195789f7bbef595f5b7f3ed727 (assuming that you can reproduce the problem again in the first place with mozregression --launch 42742c4655f65e195789f7bbef595f5b7f3ed727). Sorry, I'm not as familiar with the GUI version.

Attached are now also the results of another mozregession, this time between FF 119 (good) and FF 120 (bad). I think it confirms what you already suspected.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #41)

From the command line, it would be something along the lines of mozregression --find-fix --bad 42742c4655f65e195789f7bbef595f5b7f3ed727 (assuming that you can reproduce the problem again in the first place with mozregression --launch 42742c4655f65e195789f7bbef595f5b7f3ed727). Sorry, I'm not as familiar with the GUI version.

I tried two things using the GUI version and that change set.
First a regular run using the change set as the last good through today's date as first bad. They were all Good.
Second a find-fix using build 120 as last bad and the change set as first good. The last two results are below:
I'll also post a screenshot of the intermediate progress as there were good and bad in between.
This looks like the same results as before...

status: good
build_date: 2023-11-21 08:25:21.514000
changeset: 4a6ed43174388497963a5e476b2b22e495ba088e
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=f2558d4ebe41520695c83792fa6725511d388d7e&tochange=4a6ed43174388497963a5e476b2b22e495ba088e

status: bad
build_date: 2023-11-21 09:07:00.188000
changeset: dde990208a4fa2791e4fc928b9155e3499e2a10f
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=dde990208a4fa2791e4fc928b9155e3499e2a10f&tochange=4a6ed43174388497963a5e476b2b22e495ba088e

mozregression find-fix (Related to Comment#45)
from last bad build 120
to first good change set 42742c4655f65e195789f7bbef595f5b7f3ed727

fixed for the 120.0.1 dot release in 120 by backing out regressor ( Backout of this in 120 simply sets the flag to IS_EARLY_BETA_OR_EARLIER essentially disabling it in 120 release but leaving this enabled in 121)

(In reply to Ryan VanderMeulen [:RyanVM] from comment #29)

Sotaro, would reverting bug 1857385 on Release be another option for Fx120?

It could be one option. Another option could be D194171 of bug 1865426.

From Comment 37 and the following log of Attachment 9364838 [details], the GPU might not support NV12.

"GP+[GFX1-]: GpuProcessTextureId is not valid",

NV12 support check was added by Bug 1865426. Then latest nightly might not cause the problem.

Flags: needinfo?(sotaro.ikeda.g)

Hi Rick Puckett, can you check if the problem is addressed with latest nightly? Thank you.

Flags: needinfo?(rhpuckett3)

Below is a test build of Beta with bug 1865426 included. Can you please try it out and see if the problem is gone from it? Thanks in advance!
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/MkBIqRRbSfGQtmWnafcCjA/runs/0/artifacts/public/build/target.zip

Flags: needinfo?(urbi1464)

(In reply to Sotaro Ikeda [:sotaro] from comment #49)

Hi Rick Puckett, can you check if the problem is addressed with latest nightly? Thank you.

I'm not sure how to specify the latest nightly in mozregression (or determine that info on the web), but I previously ran two sets that may cover that, noted at the beginning of Comment #45 and I just re-ran a single build using Change Set "42742c4655f65e195789f7bbef595f5b7f3ed727", which logged an app version of 122.0a1 and that display the video correctly. If there's something more specific I can do, let me know ...

Flags: needinfo?(rhpuckett3)

(In reply to Ryan VanderMeulen [:RyanVM] from comment #50)

Below is a test build of Beta with bug 1865426 included. Can you please try it out and see if the problem is gone from it? Thanks in advance!
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/MkBIqRRbSfGQtmWnafcCjA/runs/0/artifacts/public/build/target.zip

I'm still relatively new to this process... is there a way to install/test this on Windows w/o hosing my normal FF install and profiles, like when suing mozregression?

(In reply to Ryan VanderMeulen [:RyanVM] from comment #50)

Below is a test build of Beta with bug 1865426 included. Can you please try it out and see if the problem is gone from it? Thanks in advance!
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/MkBIqRRbSfGQtmWnafcCjA/runs/0/artifacts/public/build/target.zip

I unzipped the file, [ virus checked the files :-) ] , changed to that directory and ran:
.\firefox.exe -new-instance -profile "%CD%\profiles\zzz"

Checked the version: 121.0b5
Navigated to: https://www.youtube.com/watch?v=HeQX2HjkcNo
The video displayed correctly.

(In reply to Sotaro Ikeda [:sotaro] from comment #49)

Hi Rick Puckett, can you check if the problem is addressed with latest nightly? Thank you.

I ran an archive from Ryan VanderMeulen with FF 121.0b5 and the video displayed correctly. See Comment #53.

Great! Thank you for checking.

FYI: FF 120.0 braught me a new option regarding NV12: gfx.video.convert-yuv-to-nv12.image-host-win. When I set this to false, then the videos are played without the green screen.

So again: The following setting heals it on Windows with FF 120.0 for me: gfx.video.convert-yuv-to-nv12.image-host-win=false

I also tried FF 120.0 with the following settings, but they have no effect on getting the green screen for any YouTube video:
layers.iosurfaceimage.use-nv12=false
media.wmf.use-nv12-format=false
media.wmf.zero-copy-nv12-textures=false

Perhaps interesting: On Linux Mint 20.3 (same computer), FF 120 does not show me the green screen. YouTube videos are played normally there. Perhaps NV12 is not supported on Linux? Or at least not with my graphic card driver?

Flags: needinfo?(urbi1464)

The bug is marked as tracked for firefox121 (beta). However, the bug still isn't assigned.

:bhood, could you please find an assignee for this tracked bug? Given that it is a regression and we know the cause, we could also simply backout the regressor. If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Flags: needinfo?(bhood)
Assignee: nobody → sotaro.ikeda.g
Flags: needinfo?(bhood)

Alright, to summarize:

  • This was originally caused by bug 1857385, which set the gfx.video.convert-yuv-to-nv12.image-host-win preference to true for Release builds.
  • We reverted bug 1857385 from the Release branch and are building a 120.0.1 dot release later today with that revert included and are aiming to ship it tomorrow if all goes well.
  • Bug 1865426 landed on Nightly during the Fx122 cycle to resolve a different issue but also fixed this one as a side effect.
  • Bug 1865426 has been backported to the Beta branch for 121.0b6 building Friday. This also includes DevEdition.

Based on all of the above, I'm resolving this bug as fixed for 121 and later by bug 1865426 and fixed by backout for Fx120. The bullet points above cover when users should see the fix arrive on their respective channels.

Status: NEW → RESOLVED
Closed: 1 year ago
Depends on: 1865426
Resolution: --- → FIXED
Target Milestone: --- → 122 Branch
No longer blocks: gfx-triage
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: