1927280 - VideoEncoder creates h264 stream with invalid NALU ordering (Chrome identifies data as invalid)

Status: RESOLVED FIXED

(Core :: Audio/Video: Web Codecs, defect)

Description

[marten.richter]

Attachment: demo.html

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0

Steps to reproduce:

I have executed the attached html file. The json files, which are automated downloaded include the contents of an EncodedVideoChunk. The file can be uploaded to the same demo.html file with the button and show the decoded pictures.

Actual results:

Decoding the data generated with Firefox on Windows (Nvidia graphics card) works with Firefox but fails with Chrome (it must be a Chrome build with USE_PROPRIETARY_CODECS enabled, so a Chrome or Edge release build will do, but a debug build needs this build flag). The reason that it fails is that the NALU sequence in the generated h264 data is:
7 SPS, 8 PPS, 9 AUD, 7 SPS, 8 PPS, 6 SEI, 6 SEI, 14 Prefix, 5 IDR (you can see this in the json).
Having AUD nalu after SPS and PPS can be seen as violation as h264 specs says:
"When an access unit delimiter NAL unit is present, it shall be the first NAL unit. There shall be at most one access unit delimiter NAL unit in any access unit. "
The code in chromium checking this behavior is around:
https://source.chromium.org/chromium/chromium/src/+/main:media/formats/mp4/avc.cc;drc=343c42daf7519f572ed7d77c325143eb24c5bcc6;l=227
and I think Chrome is right that this is a violation. (But one may ask them to lift this rigid behavior).

Expected results:

AUD should be the first NALU, so the Firefox result is interoperable.
In this way, it could not be used without fixing the NALU stream, for example, in VideoConferencing software using mixed UA as clients.

(Note: I have reported that something is broken as part of Bug 1921623 but I have decided to split it up, after having a proper diagnosis.

Comment 1

[marten.richter]

Attachment: frame_0__firefox(1).json

I have attached the file generated by demo.html, which I have analyzed.

Comment 2

[marten.richter]

I would say the problem is probably here:
https://searchfox.org/mozilla-central/rev/9c0a612291420944d2849323532c72b13ffc44b5/dom/media/platforms/agnostic/bytestreams/AnnexB.cpp#71
Where SPS and PPS are prepended, they should not be prepended before the AUD. Alternatively, it would also be an option to file a bug with Chromium so that they are less strict.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Graphics' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 4

[marten.richter]

Attachment: Bug 1927280 - Prevents for h264 attaching SPS and PPS Nalu if already present on Windows r?#media-playback-reviewers

This patch prevents adding the SPS and PPS present in extradata WMFMediaDataEncoder,
if the data already contains SPS and PPS. Otherwise the resulting Byte stream would
not be spec complained and be rejected by Chromium for example.
It also includes a WPT test.

Comment 5

[marten.richter]

The diagnosis in comment 2 was wrong (it was the decoder's pathway). The submitted patch should fix it on Windows and uncover the problem for other platforms and user agents.

Comment 6

[marten.richter]

Sorry, I do not understand why the system thought it should go to beta. I just wanted to separate it from another patch (without depending on it). Let's say my knowledge about hg is limited (and all how to are still for hg)...

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:padenot, could you have a look please?

For more information, please visit BugBot documentation.

Comment 8

[C.M.Chang[:chunmin]]

(In reply to marten.richter from comment #2)

I would say the problem is probably here:
https://searchfox.org/mozilla-central/rev/9c0a612291420944d2849323532c72b13ffc44b5/dom/media/platforms/agnostic/bytestreams/AnnexB.cpp#71
Where SPS and PPS are prepended, they should not be prepended before the AUD. Alternatively, it would also be an option to file a bug with Chromium so that they are less strict.

AnnexB::ConvertAVCCSampleToAnnexB is for only decoder, not for encoder, so the problem should be somewhere else.

Comment 9

[C.M.Chang[:chunmin]]

Attachment: demo_v2.html

Comment 10

[marten.richter]

@ #7 : Yes, comment #2 was premature, but you already saw the patch.
@#9 demo_v2.html : This is the rescaling demo, it does not help for this problem as you need a mechanism to upload it to another browser.

Comment 11

[C.M.Chang[:chunmin]]

(In reply to marten.richter from comment #10)

@ #7 : Yes, comment #2 was premature, but you already saw the patch.
@#9 demo_v2.html : This is the rescaling demo, it does not help for this problem as you need a mechanism to upload it to another browser.

The demo.html doesn't work on my end. demo_v2.html resues the page on bug 1921623 and downloads the encoded data automatically as demo.html tries to do.

Comment 12

[marten.richter]

Ok. Did you use the automatically downloaded json file and upload it into the demo page running in a chromium browser?
But it is very possible, that the problem occurs only with certain hardware encoders. (I have an nvidia card and a intel processor, I am not sure which hardware encoder is used by mft).

Comment 13

[Paul Adenot (:padenot)]

Attachment: Bug 1927280 - Adjust WPT expectations for various configurations. r?chunmin

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8588b5f0f391
Prevents for h264 attaching SPS and PPS Nalu if already present on Windows r=chunmin
https://hg.mozilla.org/integration/autoland/rev/c3e7854c487e
Adjust WPT expectations for various configurations. r=chunmin

Comment 15

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/49302 for changes under testing/web-platform/tests

Comment 16

[amarc]

https://hg.mozilla.org/mozilla-central/rev/8588b5f0f391
https://hg.mozilla.org/mozilla-central/rev/c3e7854c487e

Comment 17

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot