Closed Bug 1738845 Opened 3 years ago Closed 2 years ago

firefox-96.0 nightly fails to link webrtc: multiple definition of `WebRtcSpl_ComplexBitReverse';


(Core :: WebRTC, defect, P2)

Firefox 96



103 Branch
Tracking Status
firefox-esr91 --- unaffected
firefox-esr102 --- fixed
firefox94 --- unaffected
firefox95 --- unaffected
firefox96 --- wontfix
firefox97 --- wontfix
firefox98 --- wontfix
firefox99 --- wontfix
firefox100 --- wontfix
firefox101 --- wontfix
firefox102 --- wontfix
firefox103 --- fixed


(Reporter: herrtimson, Assigned: glandium)




(Keywords: regression)


(4 files)

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

I downloaded latest nightly sources and compiled them on armv7a-linux

Actual results:

I got this bunch of errors while linking:

37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_gn/Unified_c_common_audio_c_gn0.o (symbol from plugin): in function WebRtc_InitBuffer': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_ComplexBitReverse'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_arm_asm_gn/complex_bit_reverse_arm.o:(.text+0x0): first defined here
37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_gn/Unified_c_common_audio_c_gn0.o (symbol from plugin): in function WebRtc_InitBuffer': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_FilterARFastQ12'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_arm_asm_gn/filter_ar_fast_q12_armv7.o:(.text+0x0): first defined here
37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/third_party/spl_sqrt_floor/spl_sqrt_floor_gn/spl_sqrt_floor_arm.o: in function WebRtcSpl_SqrtFloor': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_SqrtFloor'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/third_party/spl_sqrt_floor/spl_sqrt_floor_gn/Unified_c_spl_sqrt_floor_gn0.o (symbol from plugin):(.text+0x0): first defined here
37:09.59 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: cannot find -landroid_support
37:09.59 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: cannot find -lunwind
37:10.62 collect2: error: ld returned 1 exit status

Expected results:

I expect the linking to work.

the compressed build log is attached, the error can be found right at the bottom.

The Bugbug bot thinks this bug should belong to the 'Core::WebRTC' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Component: Untriaged → WebRTC
Product: Firefox → Core
Regressed by: 1654112
Has Regression Range: --- → yes

tt_1, thanks for the report. This appears to be a build that is not in our normal Tier-1/2/3 builds. What machine are you building on?

Flags: needinfo?(herrtimson)

unless I'm misstaken, it is a tier-3 build: armv7a-linux

I'm using cross compiler, ain't nobody can follow nightly branch with a rpi2 native compile.

It's pretty obvious that the build script believes it is compiling for armv7-android, but I'm uncertain how to bend the scripts in the correct direction.

Current workaround is to go with --disable-webrtc

Flags: needinfo?(herrtimson)

If you can send me the .mozconfig file you're using to cross-compile on linux, I can probably get those build configs added without too much trouble.

Flags: needinfo?(herrtimson)

Thanks for the offer, I will upload both native and from cross-compile tomorrow.

See Also: → 1739003

I can't keep my promise of providing you with the native mozconfig as well, but I believe only MOZ_OBJDIR, x-includess and --host switch would differ. If you don't find the information you seek, please ping me again any time.

Flags: needinfo?(herrtimson)

Set release status flags based on info from the regressing bug 1654112

Assignee: nobody → mfroman
Severity: -- → S3
Ever confirmed: true
Priority: -- → P2
Blocks: 1738923

I believe introduced the regression.

There are at least two incorrect assumptions in these build configs:

  • That CONFIG["OS_TARGET"] == "Linux" implies CONFIG["CPU_ARCH"] != "arm". This explains the multiple definitions, because e.g. both complex_bit_reverse.c and complex_bit_reverse_arm.S are being built.

  • That CONFIG["CPU_ARCH"] == "arm" implies CONFIG["OS_TARGET"] == "Android". This explains the cannot find -landroid_support errors.

This breaks building Ubuntu packages, as well as the snap package, for armhf (armv7).

I gave a go at the instructions in dom/media/webrtc/third_party_build/gn-configs/ on hardware running armhf, but step 6 fails when running gclient config because depot_tools wants to bootstrap its environment by downloading pre-built artifacts from Google's servers, but those don't appear to exist for armhf. I'm not familiar enough with these tools, this will require further investigation.

For reference, here is the output:

$ ~/moz-libwebrtc# gclient config
  failed to resolve infra/3pp/tools/git/linux-armv6l@version:2.28.0.chromium.6 (line 27): no such tag
  failed to resolve infra/3pp/tools/cpython/linux-armv6l@version:2.7.17.chromium.22 (line 21): no such tag
  failed to resolve infra/3pp/tools/cpython3/linux-armv6l@version:3.8.0.chromium.8 (line 24): no such tag
~/depot_tools/bootstrap_python3: line 32: bootstrap-3.8.0.chromium.8_bin/python3/bin/python3: No such file or directory
cat: ~/depot_tools/python3_bin_reldir.txt: No such file or directory
[E2021-12-21T17:56:44.609360Z 1059 0 annotate.go:241] original error: no such file or directory
[E2021-12-21T17:56:44.609561Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.609624Z 1059 0 annotate.go:241] goroutine 1:
[E2021-12-21T17:56:44.609710Z 1059 0 annotate.go:241] #0 - python.(*Interpreter).GetVersion()
[E2021-12-21T17:56:44.609796Z 1059 0 annotate.go:241] #1 - venv.(*Config).resolvePythonInterpreter()
[E2021-12-21T17:56:44.609854Z 1059 0 annotate.go:241]   reason: failed to determine Python version for: ~/depot_tools//python3
[E2021-12-21T17:56:44.609909Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610003Z 1059 0 annotate.go:241] #2 - venv.(*Config).makeEnv()
[E2021-12-21T17:56:44.610088Z 1059 0 annotate.go:241]   reason: failed to resolve system Python interpreter
[E2021-12-21T17:56:44.610156Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610209Z 1059 0 annotate.go:241] #3 - venv.With()
[E2021-12-21T17:56:44.610264Z 1059 0 annotate.go:241]   reason: failed to initialize empty probe environment
[E2021-12-21T17:56:44.610318Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610372Z 1059 0 annotate.go:241] #4 - vpython.Run()
[E2021-12-21T17:56:44.610429Z 1059 0 annotate.go:241] #5 - application.(*application).mainImpl()
[E2021-12-21T17:56:44.610513Z 1059 0 annotate.go:241] #6 - application.(*Config).Main.func1()
[E2021-12-21T17:56:44.610577Z 1059 0 annotate.go:241] #7 -
[E2021-12-21T17:56:44.610638Z 1059 0 annotate.go:241] #8 - application.(*Config).Main()
[E2021-12-21T17:56:44.610696Z 1059 0 annotate.go:241] #9 vpython/main.go:109 - main.mainImpl()
[E2021-12-21T17:56:44.610749Z 1059 0 annotate.go:241] #10 vpython/main.go:115 - main.main()
[E2021-12-21T17:56:44.610805Z 1059 0 annotate.go:241] #11 runtime/proc.go:204 - runtime.main()
[E2021-12-21T17:56:44.610862Z 1059 0 annotate.go:241] #12 runtime/asm_arm.s:857 - runtime.goexit()

There was an update to webrtc in the nightly branch, did it help with this bug? I can't test it on the spot, as I need to update my rust toolchain to a llvm-13 based one first.

Adding arm alone breaks ppc64el.

I updated to llvm-13 toolchain in the meantime, so will give the patch a try.

Can see some stray include dir, is this correct?

@@ -9069,6 +9451,7 @@
"include_dirs": [

  •            "/tmp/central/obj-arm-unknown-linux-gnueabihf/third_party/libwebrtc/gn-output/gen/",
           "libs": [],

Damn, forgot to re-run the fixer script on those jsons.

Ok, I saw that you push forced the patch. Its seems to work just fine with my cross-gcc setup, but have no idea how it behaves at runtime. Can you please kindly test that on an actuall armv7a device?

OS: Unspecified → Linux
Hardware: Unspecified → ARM

I successfully rebuilt the firefox snap (96.0-2) on armhf with the following patches applied, and it seems to behave correctly at runtime, including WebRTC features (tested on a RPi4 running Ubuntu 20.04 armhf, with

(the first 4 patches are pre-requisites for the last one to apply cleanly).

I have confirmed that D134738 fixes building (and running) Firefox 97.0 on armhf. What is missing to move forward with this and get the patch merged?

Flags: needinfo?(mh+mozilla)

The patch that's up for review at D134738 doesn't apply cleanly on nightly any longer (it was applying until yesterday). Is there anything that we can do to help with moving this forward?

Depends on: 1773223

Mike, do the patches linked to bug 1773223 render D134738 unnecessary? Or will it need refreshing once all the former have landed?
If so, is it reasonable to expect an updated patch before the soft code freeze for 103.0 ?

The patch not applying any longer on the 102 branch means we cannot build Ubuntu packages for armhf, which is (for Ubuntu) a supported architecture.

Assignee: mfroman → nobody
Pushed by
Add webrtc configs for arm and ppc64 linux. r=mjf
Assignee: nobody → mh+mozilla
Backout by
Backed out 11 changesets (bug 1738845, bug 1760484, bug 1773223) for causing build bustages and python failures. CLOSED TREE
Pushed by
Add webrtc configs for arm and ppc64 linux. r=mjf

Ideally, the code would handle things in a more general way that doesn't
require manually dealing with these lists, but this would require more
testing than there is time left before 102 releases.

While here, remove HOST_CPU_ARCH, which is always the same and thus
never appears in a condition.

This changes none of the generated moz.builds for the current
configuration (but changes the outcome when adding new configurations)

Pushed by
Add some missing combinations of conditions to GN processing. r=firefox-build-system-reviewers,andi
Add webrtc configs for arm and ppc64 linux. r=mjf
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 103 Branch

Since nightly and release are affected, beta will likely be affected too.
$For more information, please visit auto_nag documentation.

Flags: needinfo?(mh+mozilla)

Comment on attachment 9256948 [details]
Bug 1738845 - Add webrtc configs for arm and ppc64 linux.

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Broken builds for tier-3 downstreams.
  • User impact if declined:
  • Fix Landed on Version: 103
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): NPOTB
Attachment #9256948 - Flags: approval-mozilla-esr102?
Attachment #9282156 - Flags: approval-mozilla-esr102?

The patch landed in nightly and beta is affected.
:glandium, is this bug important enough to require an uplift?

  • If yes, please nominate the patch for beta approval.
  • If no, please set status-firefox102 to wontfix.

For more information, please visit auto_nag documentation.

Flags: needinfo?(mh+mozilla)

Comment on attachment 9256948 [details]
Bug 1738845 - Add webrtc configs for arm and ppc64 linux.

Approved for 102.1.0esr.

Attachment #9256948 - Flags: approval-mozilla-esr102? → approval-mozilla-esr102+
Attachment #9282156 - Flags: approval-mozilla-esr102? → approval-mozilla-esr102+
You need to log in before you can comment on or make changes to this bug.