1953722 - Crash in [@ mozilla::WebrtcGmpVideoEncoder::Encode_g]

Status: VERIFIED FIXED

(Core :: WebRTC: Audio/Video, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

120 crashes/assertions from 120 installs of Firefox 136.0 and 136.0.1 on different operating systems.

Crash report: https://crash-stats.mozilla.org/report/index/d6b66a56-59ea-4979-a04b-d1c9b0250313

MOZ_CRASH Reason:

MOZ_RELEASE_ASSERT(mInputImageMap.IsEmpty() || mInputImageMap.LastElement().rtp_timestamp < frame->Timestamp())

Top 10 frames:

0  xul.dll  mozilla::WebrtcGmpVideoEncoder::Encode_g(webrtc::VideoFrame const&, std::vect...  dom/media/webrtc/libwebrtcglue/WebrtcGmpVideoCodec.cpp:380
1  xul.dll  mozilla::detail::RunnableMethodArguments<webrtc::VideoFrame, std::vector<webr...  xpcom/threads/nsThreadUtils.h:1085
1  xul.dll  std::invoke<`lambda at /builds/worker/workspace/obj-build/dist/include/nsThre...  /builds/worker/fetches/vs/VC/Tools/MSVC/14.39.33519/include/type_traits:1739
2  xul.dll  std::_Apply_impl(mozilla::detail::RunnableMethodArguments<webrtc::VideoFrame,...  /builds/worker/fetches/vs/VC/Tools/MSVC/14.39.33519/include/tuple:1077
2  xul.dll  std::apply(mozilla::detail::RunnableMethodArguments<webrtc::VideoFrame, std::...  /builds/worker/fetches/vs/VC/Tools/MSVC/14.39.33519/include/tuple:1088
2  xul.dll  mozilla::detail::RunnableMethodArguments<webrtc::VideoFrame, std::vector<webr...  xpcom/threads/nsThreadUtils.h:1083
2  xul.dll  mozilla::detail::RunnableMethodImpl<mozilla::WebrtcGmpVideoEncoder*, void (mo...  xpcom/threads/nsThreadUtils.h:1134
3  xul.dll  nsThread::ProcessNextEvent(bool, bool*)  xpcom/threads/nsThread.cpp:1153
3  xul.dll  NS_ProcessNextEvent(nsIThread*, bool)  xpcom/threads/nsThreadUtils.cpp:480
4  xul.dll  mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*)  ipc/glue/MessagePump.cpp:329

Comment 1

[Andreas Pehrson [:pehrsons]]

Weird that time can go backwards given https://searchfox.org/mozilla-central/rev/27583d5afd9beea3a7551f35470cf188de1d368c/third_party/libwebrtc/video/video_stream_encoder.cc#1550-1559.

But this is RTP time, which is designed to wrap around. That's probably what happens here. Possibly exacerbated by https://searchfox.org/mozilla-central/rev/27583d5afd9beea3a7551f35470cf188de1d368c/dom/media/webrtc/libwebrtcglue/WebrtcGmpVideoCodec.cpp#390-391.

Comment 2

[Donal Meehan [:dmeehan]]

We have a Fx136 desktop dot release that builds on 2025-03-17, not sure if we'll have something safe to uplift for then?
Otherwise, next week is the final week of beta for Fx137. Hopefully it can be fixed there at least.

Comment 3

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1953722 - Key input frames to the webrtc gmp encoder off ntp timestamps instead of rtp. r?aosmond

This to avoid time going backwards due to uint32_t wraparound.

Comment 4

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1953722 - Key input frames to the webrtc gmp encoder off ntp timestamps instead of rtp. r?aosmond

This to avoid time going backwards due to uint32_t wraparound.

Original Revision: https://phabricator.services.mozilla.com/D241394

Comment 5

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1953722 - Key input frames to the webrtc gmp encoder off ntp timestamps instead of rtp. r?aosmond

This to avoid time going backwards due to uint32_t wraparound.

Original Revision: https://phabricator.services.mozilla.com/D241394

Comment 6

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: Content process crashes when in a video call encoding H264 video.
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: Use some video conferencing services that use H264 like Microsoft Teams and Facebook Messenger. I have also been using https://fippo.github.io/simulcast-playground/rid-as-mid?codec=h264 to test H264 simulcast.
• Risk associated with taking this patch: Low
• Explanation of risk level: This patch changes what type of timestamp we use to track video frames for encoding with OpenH264. The type used previously is now copied on the side instead, so encoder output should not change due to this patch.
• String changes made/needed: None
• Is Android affected?: no

Comment 7

[Phabricator Automation]

release Uplift Approval Request

• User impact if declined: Content process crashes when in a video call encoding H264 video.
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: Use some video conferencing services that use H264 like Microsoft Teams and Facebook Messenger. I have also been using https://fippo.github.io/simulcast-playground/rid-as-mid?codec=h264 to test H264 simulcast.
• Risk associated with taking this patch: Low
• Explanation of risk level: This patch changes what type of timestamp we use to track video frames for encoding with OpenH264. The type used previously is now copied on the side instead, so encoder output should not change due to this patch.
• String changes made/needed: None
• Is Android affected?: no

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9c021d42060a
Key input frames to the webrtc gmp encoder off ntp timestamps instead of rtp. r=aosmond

Comment 9

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/9c021d42060a

Comment 10

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/946c27e683cb

Comment 11

[Ciprian Georgiu, Desktop QA]

Hi, Andreas Pehrson! I tried to reproduce this crash on Windows 11 with Firefox 136.0 by making a video call on Microsoft Teams and Facebook Messenger, but I couldn't reproduce it.

Could you let me know if there's anything else I should try to repro the issue?

Comment 12

[Andreas Pehrson [:pehrsons]]

(In reply to Ciprian Georgiu, Desktop QA from comment #11)

Hi, Andreas Pehrson! I tried to reproduce this crash on Windows 11 with Firefox 136.0 by making a video call on Microsoft Teams and Facebook Messenger, but I couldn't reproduce it.

Could you let me know if there's anything else I should try to repro the issue?

Well if it's no worse that's good! :-)
The hypothesis is a millisecond timestamp uint32_t wraparound. ~50 days worth of millis fit in a uint32_t, but there was a multiplication with 1000 involved as well. Then we're down to 0.05 days, i.e. ~70 minutes. Could you leave a call running for an hour or two to see if it repros? If not, then if a call of the same duration with the patch is also fine, that's good enough, thanks.

Comment 13

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/0eca105106d4

Comment 14

[Ciprian Georgiu, Desktop QA]

Thanks, Andreas! I still couldn’t reproduce the issue by leaving a call running on Microsoft Teams.

I made another call using all the fixed builds and left it running for an hour and a half, and I did not encounter any issues. I believe this should be enough to close the bug as verified fixed based on this check.

Tested with Firefox 136.0.2, 137.0b7, and 138.0a1.