Open Bug 1738845 Opened 7 months ago Updated 14 days ago

firefox-96.0 nightly fails to link webrtc: multiple definition of `WebRtcSpl_ComplexBitReverse';


(Core :: WebRTC, defect, P2)

Firefox 96



Tracking Status
firefox-esr91 --- unaffected
firefox94 --- unaffected
firefox95 --- unaffected
firefox96 --- wontfix
firefox97 --- wontfix
firefox98 --- wontfix
firefox99 --- wontfix
firefox100 --- wontfix
firefox101 --- fix-optional


(Reporter: herrtimson, Assigned: mjf, NeedInfo)


(Blocks 1 open bug, Regression)


(Keywords: regression)


(3 files)

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

I downloaded latest nightly sources and compiled them on armv7a-linux

Actual results:

I got this bunch of errors while linking:

37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_gn/Unified_c_common_audio_c_gn0.o (symbol from plugin): in function WebRtc_InitBuffer': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_ComplexBitReverse'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_arm_asm_gn/complex_bit_reverse_arm.o:(.text+0x0): first defined here
37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_gn/Unified_c_common_audio_c_gn0.o (symbol from plugin): in function WebRtc_InitBuffer': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_FilterARFastQ12'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_arm_asm_gn/filter_ar_fast_q12_armv7.o:(.text+0x0): first defined here
37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/third_party/spl_sqrt_floor/spl_sqrt_floor_gn/spl_sqrt_floor_arm.o: in function WebRtcSpl_SqrtFloor': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_SqrtFloor'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/third_party/spl_sqrt_floor/spl_sqrt_floor_gn/Unified_c_spl_sqrt_floor_gn0.o (symbol from plugin):(.text+0x0): first defined here
37:09.59 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: cannot find -landroid_support
37:09.59 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: cannot find -lunwind
37:10.62 collect2: error: ld returned 1 exit status

Expected results:

I expect the linking to work.

the compressed build log is attached, the error can be found right at the bottom.

The Bugbug bot thinks this bug should belong to the 'Core::WebRTC' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Component: Untriaged → WebRTC
Product: Firefox → Core
Regressed by: 1654112
Has Regression Range: --- → yes

tt_1, thanks for the report. This appears to be a build that is not in our normal Tier-1/2/3 builds. What machine are you building on?

Flags: needinfo?(herrtimson)

unless I'm misstaken, it is a tier-3 build: armv7a-linux

I'm using cross compiler, ain't nobody can follow nightly branch with a rpi2 native compile.

It's pretty obvious that the build script believes it is compiling for armv7-android, but I'm uncertain how to bend the scripts in the correct direction.

Current workaround is to go with --disable-webrtc

Flags: needinfo?(herrtimson)

If you can send me the .mozconfig file you're using to cross-compile on linux, I can probably get those build configs added without too much trouble.

Flags: needinfo?(herrtimson)

Thanks for the offer, I will upload both native and from cross-compile tomorrow.

I can't keep my promise of providing you with the native mozconfig as well, but I believe only MOZ_OBJDIR, x-includess and --host switch would differ. If you don't find the information you seek, please ping me again any time.

Flags: needinfo?(herrtimson)

Set release status flags based on info from the regressing bug 1654112

Assignee: nobody → mfroman
Severity: -- → S3
Ever confirmed: true
Priority: -- → P2
Duplicate of this bug: 1739003
Blocks: 1738923

I believe introduced the regression.

There are at least two incorrect assumptions in these build configs:

  • That CONFIG["OS_TARGET"] == "Linux" implies CONFIG["CPU_ARCH"] != "arm". This explains the multiple definitions, because e.g. both complex_bit_reverse.c and complex_bit_reverse_arm.S are being built.

  • That CONFIG["CPU_ARCH"] == "arm" implies CONFIG["OS_TARGET"] == "Android". This explains the cannot find -landroid_support errors.

This breaks building Ubuntu packages, as well as the snap package, for armhf (armv7).

I gave a go at the instructions in dom/media/webrtc/third_party_build/gn-configs/ on hardware running armhf, but step 6 fails when running gclient config because depot_tools wants to bootstrap its environment by downloading pre-built artifacts from Google's servers, but those don't appear to exist for armhf. I'm not familiar enough with these tools, this will require further investigation.

For reference, here is the output:

$ ~/moz-libwebrtc# gclient config
  failed to resolve infra/3pp/tools/git/linux-armv6l@version:2.28.0.chromium.6 (line 27): no such tag
  failed to resolve infra/3pp/tools/cpython/linux-armv6l@version:2.7.17.chromium.22 (line 21): no such tag
  failed to resolve infra/3pp/tools/cpython3/linux-armv6l@version:3.8.0.chromium.8 (line 24): no such tag
~/depot_tools/bootstrap_python3: line 32: bootstrap-3.8.0.chromium.8_bin/python3/bin/python3: No such file or directory
cat: ~/depot_tools/python3_bin_reldir.txt: No such file or directory
[E2021-12-21T17:56:44.609360Z 1059 0 annotate.go:241] original error: no such file or directory
[E2021-12-21T17:56:44.609561Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.609624Z 1059 0 annotate.go:241] goroutine 1:
[E2021-12-21T17:56:44.609710Z 1059 0 annotate.go:241] #0 - python.(*Interpreter).GetVersion()
[E2021-12-21T17:56:44.609796Z 1059 0 annotate.go:241] #1 - venv.(*Config).resolvePythonInterpreter()
[E2021-12-21T17:56:44.609854Z 1059 0 annotate.go:241]   reason: failed to determine Python version for: ~/depot_tools//python3
[E2021-12-21T17:56:44.609909Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610003Z 1059 0 annotate.go:241] #2 - venv.(*Config).makeEnv()
[E2021-12-21T17:56:44.610088Z 1059 0 annotate.go:241]   reason: failed to resolve system Python interpreter
[E2021-12-21T17:56:44.610156Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610209Z 1059 0 annotate.go:241] #3 - venv.With()
[E2021-12-21T17:56:44.610264Z 1059 0 annotate.go:241]   reason: failed to initialize empty probe environment
[E2021-12-21T17:56:44.610318Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610372Z 1059 0 annotate.go:241] #4 - vpython.Run()
[E2021-12-21T17:56:44.610429Z 1059 0 annotate.go:241] #5 - application.(*application).mainImpl()
[E2021-12-21T17:56:44.610513Z 1059 0 annotate.go:241] #6 - application.(*Config).Main.func1()
[E2021-12-21T17:56:44.610577Z 1059 0 annotate.go:241] #7 -
[E2021-12-21T17:56:44.610638Z 1059 0 annotate.go:241] #8 - application.(*Config).Main()
[E2021-12-21T17:56:44.610696Z 1059 0 annotate.go:241] #9 vpython/main.go:109 - main.mainImpl()
[E2021-12-21T17:56:44.610749Z 1059 0 annotate.go:241] #10 vpython/main.go:115 - main.main()
[E2021-12-21T17:56:44.610805Z 1059 0 annotate.go:241] #11 runtime/proc.go:204 - runtime.main()
[E2021-12-21T17:56:44.610862Z 1059 0 annotate.go:241] #12 runtime/asm_arm.s:857 - runtime.goexit()

There was an update to webrtc in the nightly branch, did it help with this bug? I can't test it on the spot, as I need to update my rust toolchain to a llvm-13 based one first.

Adding arm alone breaks ppc64el.

I updated to llvm-13 toolchain in the meantime, so will give the patch a try.

Can see some stray include dir, is this correct?

@@ -9069,6 +9451,7 @@
"include_dirs": [

  •            "/tmp/central/obj-arm-unknown-linux-gnueabihf/third_party/libwebrtc/gn-output/gen/",
           "libs": [],

Damn, forgot to re-run the fixer script on those jsons.

Ok, I saw that you push forced the patch. Its seems to work just fine with my cross-gcc setup, but have no idea how it behaves at runtime. Can you please kindly test that on an actuall armv7a device?

OS: Unspecified → Linux
Hardware: Unspecified → ARM

I successfully rebuilt the firefox snap (96.0-2) on armhf with the following patches applied, and it seems to behave correctly at runtime, including WebRTC features (tested on a RPi4 running Ubuntu 20.04 armhf, with

(the first 4 patches are pre-requisites for the last one to apply cleanly).

I have confirmed that D134738 fixes building (and running) Firefox 97.0 on armhf. What is missing to move forward with this and get the patch merged?

Flags: needinfo?(mh+mozilla)
Duplicate of this bug: 1754707

The patch that's up for review at D134738 doesn't apply cleanly on nightly any longer (it was applying until yesterday). Is there anything that we can do to help with moving this forward?

You need to log in before you can comment on or make changes to this bug.