2017189 - AV1 webrtc decoding hangs almost instantly when streaming

Status: NEW

(Core :: WebRTC: Audio/Video, defect, P3)

Description

[sphet128]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:147.0) Gecko/20100101 Firefox/147.0

Steps to reproduce:

I have tested this on Linux, but have been told that the same happens when watching the stream on windows.

1. Use OBS to start a VAAPI AV1 WHIP stream to a broadcast-box server with an RDNA 3 or RDNA 4 GPU. In my case I have a 9070xt but I was able to confirm with others that have a 7900xtx and a 7900xt that the same behavior happens on RDNA 3 as well.

2. On firefox open the stream and only the first half a second will render before the image freezes. Refreshing the page causes the stream to play another half a second or so before freezing again.

Actual results:

The stream remains frozen after playing for less than one second. Refreshing the page will play the stream for another sub second duration before freezing again.

Disabling HW decoding in firefox resolves the issue. Using SVT-AV1 encoding instead of VAAPI AV1 encoding also works fine even with HW decoding enabled.

Chromium works fine with HW AV1 decoding enabled even when streaming VAAPI AV1 from an RDNA3/4 GPU.

Expected results:

The stream should play as expected and not freeze within the first second on playback when using HW AV1 decoding.

Comment 1

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Core::Widget: Gtk' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Martin Stránský [:stransky] (ni? me)]

Can you please run on terminal with MOZ_LOG="FFmpegVideo:5" env variable, reproduce the issue and attach the log here?
Thanks.

Comment 3

[sphet128]

Attachment: firefox FFmpegVideo:5 logs

Comment 4

[sphet128]

(In reply to Martin Stránský [:stransky] (ni? me) from comment #2)

Can you please run on terminal with MOZ_LOG="FFmpegVideo:5" env variable, reproduce the issue and attach the log here?
Thanks.

Logs uploaded.

Comment 5

[sphet128]

Attachment: firefox LIBVA logs

Comment 6

[sphet128]

Attachment: AMD_DEBUG logs firefox

Comment 7

[sphet128]

I have also attached the LIBVA logs and the AMD_DEBUG logs while reproducing the issue.
To me it seems like it might be a decoder hang, but seeing as it works fine on chromium, I'm more inclined to believe it is the firefox decoder thread hanging while reading/sending requests to the VAAPI decoder.

Comment 8

[Martin Stránský [:stransky] (ni? me)]

Thanks, but the FFmpegVideo:5 log is not complete - it looks like the VA-API decoding is not performed at all. Does it really hangs after the

D/FFmpegVideo FFVPX: Using preferred software codec pcm_s16le

line? If so it's not a VA-API decoding issue but rather a general media reading issue.

Please try again and open any clip locally from disk.
Thanks.

Comment 9

[Martin Stránský [:stransky] (ni? me)]

Also please attach about:support page. btw. do I understand correctly you have an issue with particular stream and not with video playback generally? (like YT or so).

Comment 10

[sphet128]

(In reply to Martin Stránský [:stransky] (ni? me) from comment #8)

Thanks, but the FFmpegVideo:5 log is not complete - it looks like the VA-API decoding is not performed at all. Does it really hangs after the

D/FFmpegVideo FFVPX: Using preferred software codec pcm_s16le

line? If so it's not a VA-API decoding issue but rather a general media reading issue.

Please try again and open any clip locally from disk.
Thanks.

yes these appear when I launch firefox and opening the stream does not add anything to the logs.
Opening the same stream in chromium works fine (I have firefox open with a hanged video and chromium open next to it and playing the stream fine).
Recording a clip to disk using obs with vaapi av1 and playing it in firefox works fine. I was only able to reproduce this behavior with webrtc. I can also see the additional logs from MOZ_LOG="FFmpegVideo:5" when playing the clip locally.

Comment 11

[sphet128]

Attachment: FFmpegVideo:5 logs when playing a locall vappi av1 file from disk

Comment 12

[sphet128]

Attachment: about:support

Comment 13

[sphet128]

(In reply to Martin Stránský [:stransky] (ni? me) from comment #9)

Also please attach about:support page. btw. do I understand correctly you have an issue with particular stream and not with video playback generally? (like YT or so).

Youtube and other video platforms work fine. I have only observed the issue when using VAAPI AV1 on RDNA3/4 in a webrtc stream. The same stream works fine on chromium.
Using SVT-AV1 works fine.
An easy way to reproduce the issue if you have an RDNA3/4 card is to:

1. Open OBS and set the encoder to vaapi av1. Make sure you have libdatachannel 0.24.1 or newer installed as older versions had issue with packing vaapi av1 packets.
2. In obs in the stream tab, set the service to WHIP, the server to "https://b.siobud.com/api/whip" and the key to any string you want.
3. Start streaming and head to https://b.siobud.com/ and input your stream key there to watch the stream.
4. The video will hang on the first frame
5. Watch the same stream with chromium and it works fine

If you would like to see the behavior on your end but do not have access to an AMD GPU with AV1 encoding, we can arrange a time and I can stream from my PC so you can see the hang on your firefox install.

Thank you for looking into this!

Comment 14

[sphet128]

Not sure how helpful this can be but libdatachannel also had issues with VAAPI AV1 due to missing temporal delimiters and it was fix in this PR: https://github.com/paullouisageneau/libdatachannel/pull/1495

Comment 15

[Martin Stránský [:stransky] (ni? me)]

Can you try to set media.hardware-video-decoding.enabled to false at about:config, restart browser and try again? I think you'll get the same result (i.e. without AV1 HW video decoding).

Can you run on terminal with MOZ_LOG="MediaFormatReader:5,MediaStreamTrack:5,HTMLMediaElement:5" make it hang and attach the log here?
Thanks.

Comment 16

[sphet128]

Attachment: MOZ_LOG="MediaFormatReader:5,MediaStreamTrack:5,HTMLMediaElement:5"

Comment 17

[sphet128]

Hello,
Sorry for the delay on this. I have sent the logs of the hang and was able to confirm that even with HW video decoding disabled I still get the issue.

However, I have discovered something very interesting and it is probably the reason why I originally though that turning off HW video decoding worked before. As long as the image on the stream is changing in a substantial way (dragging a window around the desktop rapidly for example), the stream will NOT freeze! But if I stop this movement for a second or two, the stream will completely freeze for good and never recover until I refresh the page. This is true with and without HW decoding. Moving something small on screen like the mouse cursor is not enough not to cause the freeze, it has to be a substantial part of the stream that is changing.

Comment 18

[sphet128]

I recorded a clip showing exactly this behavior: https://drive.google.com/file/d/1NsOHEs2XjiD-G66QsCPx_08z0cW2yf1C/view?usp=sharing

You can see how at the beginning the image instantly hangs when the stream loads and nothing is moving on screen, after I refresh the page and start shaking the obs window around it does not freeze until I stop moving it for a second, and then it stays frozen.

Comment 19

[Martin Stránský [:stransky] (ni? me)]

So do I understand correctly that's more rendering bug that video decoding one? Looks like the Firefox window itself is not updated, right?

Comment 20

[sphet128]

Yes I guess since the issue happens with both software and hardware decoding it must be a rendering bug.
The good news I guess is that its easily reproducible so I can easily test any future potential patch.
For extra clarity the issue does not happen with svt-av1 or with h264 even vaapi h264 encoding does not cause the window to freeze.

I havent tested it but I was told that vaapi av1 webrtc streams also hang on windows firefox.

Comment 21

[Jim Mathies [:jimm]]

Likely Bug 2016484 - High-bitrate AV1 freezes: DAV1DDecoder::InvokeDecode busy-waits on dav1d EAGAIN instead of draining frames

Comment 22

[sphet128]

This is not related to the same bug as 2016484.
The issue for me occurs at low bitrates (2500kbps) and higher bitrates, and it also occurs when using HW decoding.
About the mention of it being a renderer bug in my previous comment, I just wanted to clarify that the firefox windows itself does not freeze. I can still interact fine with firefox and with the buttons on the stream’s webpage fine. The video feed itself is the only thing that is frozen.

The issue seems to stem from how VAAPI AV1 encodes the stream.

Comment 23

[Jo Turk]

This issue happens because of trailing padding OBUs, from my testing they are always emitted in CBR mode (OBS+NVENC+whip), dav1d returns a picture before consuming the remaining padding: [frame OBUs] => picture returned => [padding OBU still buffered] The next dav1d_send_data() call returns EAGAIN because the previous input is still buffered. There is a libwebrtc patch to fix it: https://webrtc-review.googlesource.com/c/src/+/503960
When obu-paddings are filtered in broadcast-box, this issue doesn't happen: https://github.com/Glimesh/broadcast-box/issues/633#issuecomment-5850754198