1803530 - WebRTC hw encoding on Android will skip frames with odd resolutions

Status: VERIFIED FIXED

(Core :: WebRTC: Audio/Video, defect, P1)

Description

[Andreas Pehrson [:pehrsons]]

On Android when h264 hw encoding is available (it mostly is), we negotiate h264 by default.

These hw encoders tend to be picky on the resolution they will accept. libwebrtc for instance sets the resolution alignment on Android MediaCodecs to 16.

Before bug 1780310 we had a particular way of calculating resolutions, mainly for simulcast, that would normally result in 16-aligned dimensions. This behavior was due to limitations in libwebrtc; it would only accept layers with exactly matching aspect ratios.

Bug 1780310 left the handling to libwebrtc since those limitations were gone. However, on Android where the encoder is picky, we 1) do not tell libwebrtc about this pickiness, and 2) do not handle said resolution alignment when signaled.

Comment 1

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1803530 - Request 16-alignment for MediaDataEncoder. r?jolin!, r?#webrtc-reviewers!

This is a simple blanket fix to avoid a regression in 108.
This assumes MediaDataEncoder is mainly used for hardware encoding, therefore
the main use cases on desktop should still use other alignment (libwebrtc
encoders).

Comment 2

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1803530 - Follow resolution alignment in VideoStreamFactory. r?#webrtc-reviewers!

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fae38792970a
Request 16-alignment for MediaDataEncoder. r=jolin,webrtc-reviewers,jib
https://hg.mozilla.org/integration/autoland/rev/a57a28264a06
Follow resolution alignment in VideoStreamFactory. r=webrtc-reviewers,jib

Comment 4

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/fae38792970a
https://hg.mozilla.org/mozilla-central/rev/a57a28264a06

Comment 5

[Dianna Smith [:diannaS] - on Leave, back on August 9th]

:apehrsons do you have any clear STR that can be provided to QA before nominating this for uplift into 108? It would be good to have some verification in nightly first.

Comment 6

[Andreas Pehrson [:pehrsons]]

STR:

• Load https://jsfiddle.net/pehrsons/cq3wjnba/show (thanks jib for the original fiddle)
• Make sure h264 is checked
• Click gUM! to start capturing video
• Some Android devices may fail to encode h264, then they are no good for verification. We have for instance seen a Pixel 6 that may have been blocked. If you see two live video elements you are good.
• Set the downscale value so that either width*downscale or height*downscale would be an odd number.

This bug is verified if at this point there are two live video elements.
This bug is NOT verified if one video element is frozen.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:pehrsons, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox108 to wontfix.

For more information, please visit auto_nag documentation.

Comment 8

[Lorand Janos]

Verified as fixed on the latest Nightly 109.0a1 (2022-12-06) build.
Devices used:

• Oppo Find X5 (Android 12).
• Google Pixel 6 (Android 13).

Using the steps given above, the two live video elements were displayed.

Comment 9

[Andreas Pehrson [:pehrsons]]

Comment on attachment 9306236 [details]
Bug 1803530 - Follow resolution alignment in VideoStreamFactory. r?#webrtc-reviewers!

Beta/Release Uplift Approval Request

• User impact if declined: Video calls on Android (webrtc) are likely to freeze for remote peers at some point because we do not send frames for certain resolutions for the default codec on Android. The affected resolutions could be set by the site or by libwebrtc under the hood.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: see comment 6
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This bug adds a bit of plumbing and some logic to adjust the send resolution. gtest coverage of the resolution adjustment exists. mochitest coverage of both pieces exist. No lifetime/timing changes. Multiple threads access the relevant data but it is const so no real concern here.
• String changes made/needed:
• Is Android affected?: Yes

Comment 10

[Dianna Smith [:diannaS] - on Leave, back on August 9th]

Comment on attachment 9306235 [details]
Bug 1803530 - Request 16-alignment for MediaDataEncoder. r?jolin!, r?#webrtc-reviewers!

Changing flag to release since we are in RC week for 108

Comment 11

[Dianna Smith [:diannaS] - on Leave, back on August 9th]

Comment on attachment 9306236 [details]
Bug 1803530 - Follow resolution alignment in VideoStreamFactory. r?#webrtc-reviewers!

Approved for 108.0rc2

Comment 12

[Dianna Smith [:diannaS] - on Leave, back on August 9th]

https://hg.mozilla.org/releases/mozilla-release/rev/cd591db0ebc2
https://hg.mozilla.org/releases/mozilla-release/rev/a5c169668575

Comment 13

[Lorand Janos]

Verified as fixed on the latest RC 108.1.0 build2.
Devices used:

Oppo Find X5 (Android 12).

Using the same steps as before, the two live video elements were displayed.