Bugzilla

Updated

•

2 years ago

Severity: -- → S3

Updated

•

2 years ago

Keywords: parity-edge, perf

Updated

•

2 years ago

Blocks: 4k-video

Reporter

Comment 2

•

2 years ago

Attached file Mozregression Bisection Log.txt — Details

Here is the log from mozregression that you requested. I believe all of the versions that had smooth playback we're using CPU decoding for AV1, while all of the bad builds were using GPU decoding. I also noticed some small color artifacts on the bad builds. Let me know if you need any more info.

Flags: needinfo?(micromine2004)

BugBot [:suhaib / :marco/ :calixte]

Updated

•

2 years ago

Regressed by: 1758605

Comment 3

•

2 years ago

:Zaggy1024, since you are the author of the regressor, bug 1758605, could you take a look?

For more information, please visit auto_nag documentation.

Flags: needinfo?(Zaggy1024)

Keywords: regression

Comment 4

•

2 years ago

Looks like mozregression attributed this to the wrong bug, I'm changing this over to reference the bug# for implementation of AV1 decoding.

Unfortunately I don't have an AMD GPU to test this on, it would require an RDNA 2 GPU (Ryzen 6000+ or RX 6000+). My Nvidia GPU has no issues with 8K AV1 decoding.

Firefox is using the Windows Media Foundation decoder implementation rather than DirectX 11 decoding. There shouldn't need to be a significant performance difference with WMF, so this may be an issue on WMF's side. Chrome does have an implementation of the WMF decoder as well, but I believe that would require code changes to force its usage over DX11, but that would allow someone to determine if it's WMF's fault.

Flags: needinfo?(Zaggy1024)

No longer regressed by: 1758605

Comment 5

•

2 years ago

I am running Firefox on my desktop as well which has a 3060 ti. That card is able to decode AV1 in Firefox just fine. The issue is isolated to the AMD GPU in my laptop. One other thing I should note is that while the laptop does have an RX 6700S in it, I have it disabled while on battery, so all of the tests I did earlier with mozregression were on the 680M iGPU on the Ryzen 9 6900HS. I don't think it should make a difference since the 680M is still an RDNA2 GPU, but I thought I would make note of it. I don't have chrome installed on my laptop, but I do have the latest version of microsoft edge. Do you know how I can force WMF in edge, so I can see if the issue is with Firefox or WMF?

Flags: needinfo?(Zaggy1024)

Comment 6

•

2 years ago

•

Edited

(In reply to Brandon Karaca from comment #5)

Do you know how I can force WMF in edge, so I can see if the issue is with Firefox or WMF?

The only easy possibility I see is going to about:flags and setting MediaFoundation for Clear to Enabled. That switches to a Windows-provided video decoding and display pipeline that (ideally) should be equivalent to the WMF decoder.

As a note, I believe that same system is also being implemented into Firefox by :alwu in Bug 1752052, so if you set media.wmf.media-engine.*.enabled to true in about:config, that'll allow a direct comparison of that mode of playback.

Flags: needinfo?(Zaggy1024)

Comment 7

•

2 years ago

•

Edited

If that doesn't lay the blame on Microsoft, unfortunately there's not much I can do to help, since I don't have the hardware necessary. :(

Reporter

Comment 8

•

2 years ago

(In reply to Zaggy1024 from comment #6)

(In reply to Brandon Karaca from comment #5)

Do you know how I can force WMF in edge, so I can see if the issue is with Firefox or WMF?

The only easy possibility I see is going to about:flags and setting MediaFoundation for Clear to Enabled. That switches to a Windows-provided video decoding and display pipeline that (ideally) should be equivalent to the WMF decoder.

As a note, I believe that same system is also being implemented into Firefox by :alwu in Bug 1752052, so if you set media.wmf.media-engine.*.enabled to true in about:config, that'll allow a direct comparison of that mode of playback.

I tested both edge and firefox with the WMF flags enabled, and got some interesting results. Edge still worked just fine with MediaFoundation for Clear enabled, and the performance in firefox actually greatly improved when enabling the wmf media engine flags. There is a bit of stutter/frame drop for a second after the video loads which doesn't really happen in edge. It also shows a blank green frame as the video loads in firefox, but after the first couple seconds 8K AV1 playback works as it should. I also checked task manager, and can confirm that hardware acceleration is working now as well. I think I am going to leave these flags enabled, and use it in this configuration from now on. It works well enough for me.

Comment 9

•

2 years ago

(In reply to Brandon Karaca from comment #8)

I think I am going to leave these flags enabled, and use it in this configuration from now on. It works well enough for me.

This was mainly as a test, so be aware that those changes are currently extremely untested, so you're kind of on the bleeding edge of something that may be crashy or buggy at the moment. I haven't even updated to be able to try daily driving it yet :^)

If you notice anything weird happening, you'll probably want to switch those flags back.

Comment 10

•

2 years ago

You should also grab a media profile with those flags off and the frames dropping so that a developer can take a look. I would imagine that the slow code will be the WMF decoder, but it's possible it'll turn out to be graphics-related or something.

Flags: needinfo?(micromine2004)

Reporter

Comment 11

•

2 years ago

Attached file Firefox 2022-10-20 10.16 profile.json.gz — Details

Here is a media profile I captured while playing an 8K AV1 video. The wmf media engine flags were off, so there was lots of stuttering and frame drops. I attached the file, but I will also paste the link here: https://share.firefox.dev/3CNd8Ba.

Flags: needinfo?(micromine2004)

Comment 12

•

2 years ago

•

Edited

Looks like some decoding calls are running for much longer than normal, often between 80ms~100ms. I can see a similar behavior when profiling with my Nvidia card, but it doesn't manage to drop nearly as many frames as you're probably seeing.

This may be related to some non-fatal COM errors that periodically show up when running the WMF AV1 decoder with 8K video, which on my machine coincided with some frames being slightly delayed. On my hardware, the frames were never dropping, though, since it looks like I only get up to ~30ms of delay on a frame. Anyone with hardware to test this could run with Visual Studio attached as a debugger and the COM errors should show up in the output like this:

Exception thrown at 0x00007FFD04CA428C in firefox.exe: Microsoft C++ exception: _com_error at memory location 0x000000BD6BEFE820.

The stack trace when the error occurs is as follows:

 	KernelBase.dll!00007ffd04ca428c()	Unknown
 	msvcrt.dll!00007ffd05b4b3b7()	Unknown
 	d3d11.dll!00007ffcfdd3b29e()	Unknown
 	d3d11.dll!00007ffcfdca603f()	Unknown
 	av1decodermft_store.dll!00007ffc92350ca1()	Unknown
 	av1decodermft_store.dll!00007ffc92340c72()	Unknown
 	av1decodermft_store.dll!00007ffc923d1ac2()	Unknown
 	av1decodermft_store.dll!00007ffc92389bed()	Unknown
 	av1decodermft_store.dll!00007ffc92380db1()	Unknown
 	av1decodermft_store.dll!00007ffc9237794e()	Unknown
 	av1decodermft_store.dll!00007ffc923776e4()	Unknown
 	av1decodermft_store.dll!00007ffc9234134a()	Unknown
 	av1decodermft_store.dll!00007ffc9233602f()	Unknown
 	av1decodermft_store.dll!00007ffc92334308()	Unknown
 	av1decodermft_store.dll!00007ffc923322b9()	Unknown
>	xul.dll!mozilla::MFTDecoder::Output(RefPtr<IMFSample> * aOutput) Line 334	C++
 	xul.dll!mozilla::WMFVideoMFTManager::Output(__int64 aStreamOffset, RefPtr<mozilla::MediaData> & aOutData) Line 808	C++
 	xul.dll!mozilla::WMFMediaDataDecoder::ProcessOutput(nsTArray<RefPtr<mozilla::MediaData>> & aResults) Line 151	C++
 	xul.dll!mozilla::WMFMediaDataDecoder::ProcessDecode(mozilla::MediaRawData * aSample) Line 111	C++
 	[Inline Frame] xul.dll!mozilla::detail::RunnableMethodArguments<mozilla::MediaRawData *>::applyImpl(mozilla::WMFMediaDataDecoder * o, RefPtr<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData>>,mozilla::MediaResult,1>>(mozilla::WMFMediaDataDecoder::*)(mozilla::MediaRawData *) m, mozilla::Tuple<StoreRefPtrPassByPtr<mozilla::MediaRawData>> & args, std::integer_sequence<unsigned long long,0>) Line 1147	C++
 	[Inline Frame] xul.dll!mozilla::detail::RunnableMethodArguments<mozilla::MediaRawData *>::apply(mozilla::WMFMediaDataDecoder * o, RefPtr<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData>>,mozilla::MediaResult,1>>(mozilla::WMFMediaDataDecoder::*)(mozilla::MediaRawData *) m) Line 1153	C++
 	[Inline Frame] xul.dll!mozilla::detail::MethodCall<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData>>,mozilla::MediaResult,1>,RefPtr<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData>>,mozilla::MediaResult,1>> (mozilla::WMFMediaDataDecoder::*)(mozilla::MediaRawData *),mozilla::WMFMediaDataDecoder,mozilla::MediaRawData *>::Invoke() Line 1518	C++
 	xul.dll!mozilla::detail::ProxyRunnable<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData>>,mozilla::MediaResult,1>,RefPtr<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData>>,mozilla::MediaResult,1>> (mozilla::WMFMediaDataDecoder::*)(mozilla::MediaRawData *),mozilla::WMFMediaDataDecoder,mozilla::MediaRawData *>::Run() Line 1539	C++
 	xul.dll!mozilla::TaskQueue::Runner::Run() Line 266	C++
 	xul.dll!nsThreadPool::Run() Line 311	C++
 	xul.dll!nsThread::ProcessNextEvent(bool aMayWait, bool * aResult) Line 1199	C++
 	xul.dll!NS_ProcessNextEvent(nsIThread * aThread, bool aMayWait) Line 465	C++
 	xul.dll!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate * aDelegate) Line 301	C++
 	xul.dll!MessageLoop::RunHandler() Line 375	C++
 	xul.dll!MessageLoop::Run() Line 357	C++
 	xul.dll!nsThread::ThreadFunc(void * aArg) Line 386	C++
 	nss3.dll!_PR_NativeRunThread(void * arg) Line 408	C
 	nss3.dll!pr_root(void * arg) Line 140	C
 	[External Code]

I'm unsure of how to load symbols to find out where this is failing, if any are available at all. Since the error is occurring in d3d11.dll, this could be related to how we copy the frame for display, since I don't see the same issue with my card when zero-copy is enabled.

We also need to determine whether this COM error is also occurring on an RX 6000 or newer card, but it seems pretty likely that it is.

Hopefully it's all right if I needinfo you :alwu, and also :sotaro, since your domain has been the graphics side of Windows video lately, any ideas on why this might be happening? I may be a bit out of my depth without more tools to investigate this error. :(

Flags: needinfo?(sotaro.ikeda.g)

Flags: needinfo?(alwu)

Sotaro Ikeda [:sotaro]

Comment 13

•

2 years ago

Hi Brandon Karaca, can you attach about:support to this bug?
The problem also happen with Firefox nightly? And can you take a profile with Firefox nightly? Thank you.

Flags: needinfo?(sotaro.ikeda.g)

Reporter

Comment 14

•

2 years ago

Attached file Firefox about_support info.txt — Details

I have attatched the about:support info to this bug in the form of a txt file. I also installed the latest nightly build and ran the media profiler. The bug did still occur. Although YouTube didn't report many dropped frames, I could still see significant stuttering. Here is the link to the profile https://share.firefox.dev/3TRUjUI. I would also like to note that I was looking at the release notes for the latest AMD gpu driver, and they did list "Stuttering may occur during video playback using hardware acceleration with Firefox on some AMD Graphics Products such as the Radeon™ RX 6900 XT Graphics" as a known issue.

Assignee

Comment 15

•

2 years ago

Sorry for late reply. It's interesting to check the profiled result you provided in the comment 14. That result actually shows no dropped frame at all, Firefox kept decoding video in time. I can see around ~85% of frames were decoded within 30 ms.

Is this issue only happening on hardware decoding? Would you mind to help me do a screen-recording? And also provide me an example video to allow me to test?

Thank you so much.

Flags: needinfo?(alwu) → needinfo?(micromine2004)

Assignee

Comment 16

•

2 years ago

Zaggy found an interesting point where the video sink seems blocked by requesting video data. However, the process of requesting data is an async operation which should not keep blocking the MDSM thread, so it doesn't sound possible at my first thought.

So I started to look at when we would schedule the video sink to render video frames instead,

when the video sink just starts
when the video sink starts playing
schedule on timer by calculating the difference between current owning frame and clock time
when video queue has new data coming in
when video queue finishes

In our scenario, we would expect either (3) or (4) to kick in time. Unfortunately, we don't have the clock time information on the profiled result, so we don't know what scheduled time for the timer. But what I assume is that. By looking at PlayVideo at 17.934s and 18.028s. The second playVideo is apparently triggered by (4) which matches pushVideo at 18.028s as well.

So I think the issue is caused by following factors combined together,
Video decoding too slow (due to specific graphic card)
↓
MDSM will keep decoding more video without resting (it usually has a threshold, once the duration of decoded data reaches that threshold, MDSM would stop decoding until decoded duration becomes lower again)
↓
The timer used to schedule rendering uses too large waiting time (because the new video frame is larger than the clock time too much)
↓
The rendering would mostly rely on when video is decoded (push to the queue)
↓
As video doesn't video fast enough, we doesn't render video frames in a regular and faster enough interval (like what I mention above, worst case would only render ~1 per second)

Assignee: nobody → alwu

Priority: -- → P2

Assignee

Comment 17

•

2 years ago

Based on the analysis above, I think I've known what happened so don't really need a screen recording anymore.

Flags: needinfo?(micromine2004)

Comment 18

•

2 years ago

(In reply to Zaggy1024 from comment #12)

Looks like some decoding calls are running for much longer than normal, often between 80ms~100ms. I can see a similar behavior when profiling with my Nvidia card, but it doesn't manage to drop nearly as many frames as you're probably seeing.

This may be related to some non-fatal COM errors that periodically show up when running the WMF AV1 decoder with 8K video, which on my machine coincided with some frames being slightly delayed. On my hardware, the frames were never dropping, though, since it looks like I only get up to ~30ms of delay on a frame. Anyone with hardware to test this could run with Visual Studio attached as a debugger and the COM errors should show up in the output like this:

Exception thrown at 0x00007FFD04CA428C in firefox.exe: Microsoft C++ exception: _com_error at memory location 0x000000BD6BEFE820.

The stack trace when the error occurs is as follows:

For future reference, the actual profiler stack for the long-running decoder call is here:

ZwWaitForAlertByThreadId
RtlpWaitOnCriticalSection
RtlpEnterCriticalSectionContended
RtlEnterCriticalSection
CDDISync::CObjectAndSingleThreadedLowFreqLock::CObjectAndSingleThreadedLowFreqLock
CContext::DecoderBeginFrame(ID3D11VideoDecoder*, ID3D11VideoDecoderOutputView*, unsigned int, void const*)
0x7fff6f0f0ca1
0x7fff6f0e0c72
0x7fff6f171ac2
0x7fff6f129bed
0x7fff6f120db1
0x7fff6f11794e
0x7fff6f1176e4
0x7fff6f0e134a
0x7fff6f0d602f
0x7fff6f0d4308
0x7fff6f0d22b9
mozilla::MFTDecoder::Output(RefPtr<IMFSample>*)
mozilla::WMFVideoMFTManager::Output(long long, RefPtr<mozilla::MediaData>&)
mozilla::WMFMediaDataDecoder::ProcessOutput(nsTArray<RefPtr<mozilla::MediaData> >&)
mozilla::WMFMediaDataDecoder::ProcessDecode(mozilla::MediaRawData*)
mozilla::detail::RunnableMethodArguments<mozilla::MediaRawData *>::applyImpl(mozilla::AOMDecoder*, RefPtr<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData> >,mozilla::MediaResult,1> > (mozilla::AOMDecoder::*)(mozilla::MediaRawData*), mozilla::Tuple<StoreRefPtrPassByPtr<mozilla::MediaRawData> >&, std::integer_sequence<unsigned long long,0>)
mozilla::detail::RunnableMethodArguments<mozilla::MediaRawData *>::apply(mozilla::AOMDecoder*, RefPtr<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData> >,mozilla::MediaResult,1> > (mozilla::AOMDecoder::*)(mozilla::MediaRawData*))
mozilla::detail::MethodCall<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData> >,mozilla::MediaResult,1>,RefPtr<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData> >,mozilla::MediaResult,1> > (mozilla::AOMDecoder::*)(mozilla::MediaRawData *),mozilla::AOMDecoder,mozilla::MediaRawData *>::Invoke()
mozilla::detail::ProxyRunnable<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData> >,mozilla::MediaResult,1>,RefPtr<mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData> >,mozilla::MediaResult,1> > (mozilla::AOMDecoder::*)(mozilla::MediaRawData *),mozilla::AOMDecoder,mozilla::MediaRawData *>::Run()
mozilla::TaskQueue::Runner::Run()
nsThreadPool::Run()
nsThread::ProcessNextEvent(bool, bool*)
NS_ProcessNextEvent(nsIThread*, bool)
mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*)
MessageLoop::RunInternal()
MessageLoop::RunHandler()
MessageLoop::Run()
nsThread::ThreadFunc(void*)
_PR_NativeRunThread(void*)
pr_root(void*)
thread_start<unsigned int (__cdecl*)(void *),1>
BaseThreadInitThunk
mozilla::interceptor::FuncHook<mozilla::interceptor::WindowsDllInterceptor<mozilla::interceptor::VMSharingPolicyShared>,void (*)(int, void *, void *)>::operator()(int&, void*&, void*&) const
patched_BaseThreadInitThunk(int, void*, void*)
RtlUserThreadStart
(root)