1875201 - No vp8 or vp9 WebRTC video sent with --with-system-libvpx

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect)

Description

[Karl Tomlinson (:karlt)]

STR:

1. Load https://mozilla.github.io/webrtc-landing/pc_test.html
2. Check "One-way call" and "Video-only call".
3. Click "Start!"

Actual: preview is seen on right, but no video plays on left.
Expected: video plays on left.

about:webrtc RTCPeerConnection Statistics shows peer connection ends but no "Send: video/" or "Receive video/".

This previously behaved as expected but the unexpected behavior was first observed in 44c4e62a38e1 Dec 14 on whereby.com, where it does not reproduce in a build --without-system-libvpx. WebRTC Video can be received over the network by the --with-system-libvpx build.

When "Require H.264 video" is checked, then behavior is as expected.

System libvpx is Gentoo version 1.13.1-r1

Profile recorded at https://share.firefox.dev/4aZUu9c

Comment 1

[Karl Tomlinson (:karlt)]

https://pernos.co/debug/9qdo1x-RsA6RKmdCOAjZRA/index.html

Comment 2

[Karl Tomlinson (:karlt)]

d43978d3d835 behaves as expected. Suspect bug 1864008. Will confirm tomorrow.

Comment 3

[Karl Tomlinson (:karlt)]

dom/media/test/test_mediarecorder_webm_support.html passes.

test_mediarecorder_record_addtracked_stream stalls after "Audio content ok" and "Frame: 0 IsPixel ref=255,0,0,255 threshold=128 value=0,0,0,0".

Edit: this fails locally even --without-system-libvpx, so this seems to be a separate issue.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1864008

:chunmin, since you are the author of the regressor, bug 1864008, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 5

[Karl Tomlinson (:karlt)]

https://hg.mozilla.org/mozilla-central/rev/87d8deb5500f triggered the change in behavior.

Comment 6

[Donal Meehan [:dmeehan]]

:jib could this be triaged for severity?

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Not sending VP8 seems pretty severe.

Karl says we don't ship with that configuration, but linux distros may build with it. E.g. Gentoo builds --with-system-libvpx by default

Comment 8

[Paul Adenot (:padenot)]

Chun-Min, if you can have a look?

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1864008

Comment 10

[C.M.Chang[:chunmin]]

I am unable to test it on my Linux machine since I don't have a camera at hand. I'll try reproducing this on my MacBook pro. Otherwise, I'll order one camera to debug this. Keep the NI for now.

Comment 11

[Karl Tomlinson (:karlt)]

I'm getting VPX_CODEC_ABI_MISMATCH from vpx_codec_enc_init_ver().

Might the wrong "vpx/vpx_encoder.h" be used for libvpx_interface.cc?

Comment 12

[Paul Adenot (:padenot)]

Seems plausible yeah. https://searchfox.org/mozilla-central/source/third_party/libwebrtc/modules/video_coding/codecs/interface/libvpx_interface.h#18-21 should be <> not "".

Comment 13

[C.M.Chang[:chunmin]]

(In reply to Paul Adenot (:padenot) from comment #12)

Seems plausible yeah. https://searchfox.org/mozilla-central/source/third_party/libwebrtc/modules/video_coding/codecs/interface/libvpx_interface.h#18-21 should be <> not "".

The VPX headers in libwebrtc should be redirected to the headers under third_party/libvpx/vpx.

To follow the libwebrtc merge procedure, the vpx headers are silently replaced with "trampoline" headers, which do nothing but include real VPX headers. This makes the libwebrtc-merge process easier without worrying headers' paths.

Comment 14

[Karl Tomlinson (:karlt)]

Should the "/media/libvpx/libvpx/" added in https://hg.mozilla.org/mozilla-central/rev/87d8deb5500f#l2.12 instead be "/third_party/libwebrtc/third_party/libvpx/"?

I tried making that change, but it is as if the real headers are not included: libvpx_interface.h:33:11: error: unknown type name 'vpx_image_t'.

What would prevent third_party/libwebrtc/third_party/libvpx/vpx/vpx_encoder.h including itself instead of the real header?
Should it be using #include_next?

The bug is resolved with those changes.

Comment 15

[torokati44]

I think I hit this regression just today after updating Firefox to 122 on Fedora from the official repo.

I can reproduce using the steps in the original Description.
However, when also checking the "Require H.264 video" box, the video appears on the left too.

This caused Google Meet to be unable to send camera feed or share the screen.
See: https://github.com/webcompat/web-bugs/issues/132676

Comment 16

[torokati44]

Workarounds for those just stumbling upon this issue:

Switching some flags on about:config to this state fixed Google Meet for me:

media.encoder.webm.enabled	true	
media.ffmpeg.encoder.enabled	true	
media.webrtc.platformencoder	true	
media.webrtc.platformencoder.sw_only	false	
media.webrtc.software_encoder.fallback	true

The "Simple RTCPeerConnection Video Test" reproducer is also fixed:
When checking "Require H.264 video", it's still working as before.
When checking "Require VP9 video", there is an image on the left, but it's very laggy.
When checking "Require VP8 video", the image on the left is perfectly fine.

Comment 17

[Donal Meehan [:dmeehan]]

:cchange any update on this investigation?
We have a scheduled dot release next week for Fx122, if there is anything that can be done to help linux distros

Comment 18

[C.M.Chang[:chunmin]]

(In reply to torokati44 from comment #15)

I think I hit this regression just today after updating Firefox to 122 on Fedora from the official repo.

I can reproduce using the steps in the original Description.
However, when also checking the "Require H.264 video" box, the video appears on the left too.

This caused Google Meet to be unable to send camera feed or share the screen.
See: https://github.com/webcompat/web-bugs/issues/132676

I just got a camera to check this. However, I am not able to reproduce this in the test page: https://mozilla.github.io/webrtc-landing/pc_test.html on my ubuntu machine with the following mozconfig:

ac_add_options --enable-debug
ac_add_options --enable-optimize="-O1"
ac_add_options --with-system-libvpx
ac_add_options --without-sysroot

Not sure if it's due to the libvpx versions. I am tweaking some settings and see if I can reproduce it.

Comment 19

[Donal Meehan [:dmeehan]]

:padenot were there any follow-ups on your thoughts in Comment 12?
Could we investigate landing a fix based on this?

Comment 20

[Donal Meehan [:dmeehan]]

Adding a list of some reddit threads where this issue was raised:

• https://old.reddit.com/r/firefox/comments/19dp1or/firefox_1220_see_all_new_features_updates_and/kj7t1up/
• https://old.reddit.com/r/firefox/comments/1adyo9s/google_meet_webcam_isnt_viewable_by_participants/
• https://old.reddit.com/r/firefox/comments/1ads643/people_cant_see_me_in_google_meet_any_more/
• https://old.reddit.com/r/firefox/comments/1adb0mi/cant_access_microphone_or_camera_on_google_meet/
• https://old.reddit.com/r/firefox/comments/19ete78/google_meetings_now_broken_in_firefox_linux/

Comment 21

[Paul Adenot (:padenot)]

Attachment: WIP: Bug 1875201 - DO NOT LAND - Only include vendored libvpx headers in WebRTC if using vendored libvpx

Comment 22

[C.M.Chang[:chunmin]]

(In reply to C.M.Chang[:chunmin] from comment #18)

Not sure if it's due to the libvpx versions. I am tweaking some settings and see if I can reproduce it.

I downgraded the libvpx in my system to v1.13 (bb5edbd8417277b6fdb45867b4cbdb7ac6300d4f), but the build failed at compiling. (if it's built by shared library, maybe it's why we have this bug?)

4:18.86 ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol 'stderr'; recompile with -fPIC
4:18.86 >>> defined in /lib/x86_64-linux-gnu/libc.so.6
4:18.86 >>> referenced by vp9_ext_ratectrl.c.o:(vp9_extrc_get_encodeframe_decision) in archive /usr/local/lib/libvpx.a
4:18.86 ld.lld: error: relocation R_X86_64_PC32 cannot be used against symbol 'stderr'; recompile with -fPIC
4:18.86 >>> defined in /lib/x86_64-linux-gnu/libc.so.6
4:18.86 >>> referenced by vp9_ext_ratectrl.c.o:(vp9_extrc_get_frame_rdmult) in archive /usr/local/lib/libvpx.a
4:18.88 clang++: error: linker command failed with exit code 1 (use -v to see invocation)

I'll check other version to see if I can pass. Or, maybe libvpx should be loaded by a shared library instead of being built as a static library (need to run ./configure in libvpx repo with --enable-shared --disable-static).

Comment 23

[C.M.Chang[:chunmin]]

(In reply to C.M.Chang[:chunmin] from comment #22)

maybe libvpx should be loaded by a shared library instead of being built as a static library (need to run ./configure in libvpx repo with --enable-shared --disable-static).

Using shared library can pass the build, but it fails when running ./mach run

 0:00.40 /home/cm/Work/gecko-dev/obj-x86_64-pc-linux-gnu/dist/bin/firefox -no-remote -profile /home/cm/Work/gecko-dev/obj-x86_64-pc-linux-gnu/tmp/profile-default
XPCOMGlueLoad error for file /home/cm/Work/gecko-dev/obj-x86_64-pc-linux-gnu/dist/bin/libxul.so:
libvpx.so.8: cannot open shared object file: No such file or directory
Couldn't load XPCOM.

Not sure why libvpx.so.8 cannot be found. It's under /usr/local/lib/ with permission 777. Maybe Firefox try to find libvpx.so.8 in another place?

Comment 24

[Mike Hommey [:glandium]]

Sounds like you need to run ldconfig. I don't know why you didn't just install the libvpx package from Ubuntu, though...

Comment 25

[Michael Froman [:mjf]]

Attachment: Bug 1875201 - Add method for conditionally including "media/libvpx/libvpx". r?ng,glandium

Comment 26

[Michael Froman [:mjf]]

Attachment: Bug 1875201 - Add method for conditionally including media/libvpx/libvpx - moz.build file updates. r?ng

Comment 28

[C.M.Chang[:chunmin]]

(In reply to Mike Hommey [:glandium] from comment #24)

Sounds like you need to run ldconfig. I don't know why you didn't just install the libvpx package from Ubuntu, though...

I had manually installed libvpx to try updating in-tree ffmpeg a few months ago, so I'd like to have one single installation in my system.

Comment 29

[Donal Meehan [:dmeehan]]

:chunmin this wont make a Fx122 release, but is there anything that will be fixed in time for Fx123?
Fx123 is in the last week of beta, and goes to release candidate next week.

Comment 30

[Paul Adenot (:padenot)]

I've picked this up from Michael and I have patches, doing a final test and uploading them in a minute.

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4468a6dab74e
Add method for conditionally including "media/libvpx/libvpx". r=webrtc-reviewers,ng
https://hg.mozilla.org/integration/autoland/rev/721a670e7807
Add method for conditionally including media/libvpx/libvpx - moz.build file updates. r=ng,webrtc-reviewers

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7b42bd13f188
Fix for black linting in gn_processor.py CLOSED TREE

Comment 33

[Paul Adenot (:padenot)]

Breaks webrtc for downstream, we want to take this in beta. This doesn't affect our own builds.

Comment 34

[Paul Adenot (:padenot)]

Attachment: Bug 1875201 - Add method for conditionally including "media/libvpx/libvpx". r?ng,glandium

Original Revision: https://phabricator.services.mozilla.com/D200117

Comment 35

[Paul Adenot (:padenot)]

Attachment: Bug 1875201 - Add method for conditionally including "media/libvpx/libvpx". r?ng,glandium

Original Revision: https://phabricator.services.mozilla.com/D200117

Comment 36

[Paul Adenot (:padenot)]

Attachment: Bug 1875201 - Add method for conditionally including media/libvpx/libvpx - moz.build file updates. r?ng

This patch is the result of running:

./mach python python/mozbuild/mozbuild/gn_processor.py
dom/media/webrtc/third_party_build/gn-configs/webrtc.json

Original Revision: https://phabricator.services.mozilla.com/D200118

Comment 37

[Phabricator Automation]

Uplift Approval Request

• Risk associated with taking this patch: None, not part of our build
• Needs manual QE test: yes
• Explanation of risk level: Not part of our build
• String changes made/needed: None
• Code covered by automated testing: no
• Fix verified in Nightly: no
• Is Android affected?: no
• User impact if declined: No WebRTC in some popular downstream linux distro builds
• Steps to reproduce for manual QE testing: Downstream will test

Comment 38

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/4468a6dab74e
https://hg.mozilla.org/mozilla-central/rev/721a670e7807
https://hg.mozilla.org/mozilla-central/rev/7b42bd13f188

Comment 39

[Pascal Chevrel:pascalc]

Downgrading to S2 as this is not affecting directly our builds and a large part of our users.

Comment 40

[Hani Yacoub, Desktop QA]

I'm having difficulties trying to reproduce the bug in order to verify it. I followed the steps form the description and the configs from comment 11.

Are there any steps you manage to reach to in order to reproduce this bug?
Thanks.

Comment 41

[Paul Adenot (:padenot)]

You'd need a special build from a downstream linux distribution -- we've already had external verification, and I've double-checked myself, it's fine to skip this, thanks!

Comment 42

[Hani Yacoub, Desktop QA]

Adding the Qa-not-actionable tag since this issue can't be reproduced on our side.

Comment 43

[Dianna Smith [:diannaS]]

:padenot fyi these failed to land due to conflicts

Comment 44

[Paul Adenot (:padenot)]

I rebased those for pascal yesterday, they should land fine now.

Comment 45

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/3b0a14bcdbfa
https://hg.mozilla.org/releases/mozilla-beta/rev/1bb55a790d7a