1738845 - firefox-96.0 nightly fails to link webrtc: multiple definition of `WebRtcSpl_ComplexBitReverse';

Status: RESOLVED FIXED

(Core :: WebRTC, defect, P2)

Description

[tt_1]

Attachment: firefox-nightly-fails-in-webrtc-linking-build.log.gz

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

I downloaded latest nightly sources and compiled them on armv7a-linux

Actual results:

I got this bunch of errors while linking:

37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_gn/Unified_c_common_audio_c_gn0.o (symbol from plugin): in function WebRtc_InitBuffer': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_ComplexBitReverse'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_arm_asm_gn/complex_bit_reverse_arm.o:(.text+0x0): first defined here
37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_gn/Unified_c_common_audio_c_gn0.o (symbol from plugin): in function WebRtc_InitBuffer': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_FilterARFastQ12'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/common_audio_c_arm_asm_gn/filter_ar_fast_q12_armv7.o:(.text+0x0): first defined here
37:08.36 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/third_party/spl_sqrt_floor/spl_sqrt_floor_gn/spl_sqrt_floor_arm.o: in function WebRtcSpl_SqrtFloor': 37:08.36 (.text+0x0): multiple definition ofWebRtcSpl_SqrtFloor'; /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-96.0/work/firefox_build/toolkit/library/build/../../../third_party/libwebrtc/common_audio/third_party/spl_sqrt_floor/spl_sqrt_floor_gn/Unified_c_spl_sqrt_floor_gn0.o (symbol from plugin):(.text+0x0): first defined here
37:09.59 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: cannot find -landroid_support
37:09.59 /usr/libexec/gcc/armv7a-unknown-linux-gnueabihf/ld: cannot find -lunwind
37:10.62 collect2: error: ld returned 1 exit status

Expected results:

I expect the linking to work.

the compressed build log is attached, the error can be found right at the bottom.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::WebRTC' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Michael Froman [:mjf]]

tt_1, thanks for the report. This appears to be a build that is not in our normal Tier-1/2/3 builds. What machine are you building on?

Comment 3

[tt_1]

unless I'm misstaken, it is a tier-3 build: armv7a-linux

I'm using cross compiler, ain't nobody can follow nightly branch with a rpi2 native compile.

It's pretty obvious that the build script believes it is compiling for armv7-android, but I'm uncertain how to bend the scripts in the correct direction.

Current workaround is to go with --disable-webrtc

Comment 4

[Michael Froman [:mjf]]

If you can send me the .mozconfig file you're using to cross-compile on linux, I can probably get those build configs added without too much trouble.

Comment 5

[tt_1]

Thanks for the offer, I will upload both native and from cross-compile tomorrow.

Comment 6

[tt_1]

Attachment: mozconfig from cross-compile

I can't keep my promise of providing you with the native mozconfig as well, but I believe only MOZ_OBJDIR, x-includess and --host switch would differ. If you don't find the information you seek, please ping me again any time.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1654112

Comment 9

[Olivier Tilloy]

I believe https://hg.mozilla.org/mozilla-central/rev/dd3d80e100cea5b557ca3bfebf1872b0701ee3ca introduced the regression.

There are at least two incorrect assumptions in these build configs:

• That CONFIG["OS_TARGET"] == "Linux" implies CONFIG["CPU_ARCH"] != "arm". This explains the multiple definitions, because e.g. both complex_bit_reverse.c and complex_bit_reverse_arm.S are being built.

• That CONFIG["CPU_ARCH"] == "arm" implies CONFIG["OS_TARGET"] == "Android". This explains the cannot find -landroid_support errors.

This breaks building Ubuntu packages, as well as the snap package, for armhf (armv7).

Comment 10

[Olivier Tilloy]

I gave a go at the instructions in dom/media/webrtc/third_party_build/gn-configs/README.md on hardware running armhf, but step 6 fails when running gclient config https://github.com/mozilla/libwebrtc because depot_tools wants to bootstrap its environment by downloading pre-built artifacts from Google's servers, but those don't appear to exist for armhf. I'm not familiar enough with these tools, this will require further investigation.

Comment 11

[Olivier Tilloy]

For reference, here is the output:

$ ~/moz-libwebrtc# gclient config https://github.com/mozilla/libwebrtc
Errors:
  failed to resolve infra/3pp/tools/git/linux-armv6l@version:2.28.0.chromium.6 (line 27): no such tag
  failed to resolve infra/3pp/tools/cpython/linux-armv6l@version:2.7.17.chromium.22 (line 21): no such tag
  failed to resolve infra/3pp/tools/cpython3/linux-armv6l@version:3.8.0.chromium.8 (line 24): no such tag
~/depot_tools/bootstrap_python3: line 32: bootstrap-3.8.0.chromium.8_bin/python3/bin/python3: No such file or directory
cat: ~/depot_tools/python3_bin_reldir.txt: No such file or directory
[E2021-12-21T17:56:44.609360Z 1059 0 annotate.go:241] original error: no such file or directory
[E2021-12-21T17:56:44.609561Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.609624Z 1059 0 annotate.go:241] goroutine 1:
[E2021-12-21T17:56:44.609710Z 1059 0 annotate.go:241] #0 go.chromium.org/luci/vpython/python/interpreter.go:120 - python.(*Interpreter).GetVersion()
[E2021-12-21T17:56:44.609796Z 1059 0 annotate.go:241] #1 go.chromium.org/luci/vpython/venv/config.go:286 - venv.(*Config).resolvePythonInterpreter()
[E2021-12-21T17:56:44.609854Z 1059 0 annotate.go:241]   reason: failed to determine Python version for: ~/depot_tools//python3
[E2021-12-21T17:56:44.609909Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610003Z 1059 0 annotate.go:241] #2 go.chromium.org/luci/vpython/venv/config.go:186 - venv.(*Config).makeEnv()
[E2021-12-21T17:56:44.610088Z 1059 0 annotate.go:241]   reason: failed to resolve system Python interpreter
[E2021-12-21T17:56:44.610156Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610209Z 1059 0 annotate.go:241] #3 go.chromium.org/luci/vpython/venv/venv.go:150 - venv.With()
[E2021-12-21T17:56:44.610264Z 1059 0 annotate.go:241]   reason: failed to initialize empty probe environment
[E2021-12-21T17:56:44.610318Z 1059 0 annotate.go:241] 
[E2021-12-21T17:56:44.610372Z 1059 0 annotate.go:241] #4 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
[E2021-12-21T17:56:44.610429Z 1059 0 annotate.go:241] #5 go.chromium.org/luci/vpython/application/application.go:320 - application.(*application).mainImpl()
[E2021-12-21T17:56:44.610513Z 1059 0 annotate.go:241] #6 go.chromium.org/luci/vpython/application/application.go:408 - application.(*Config).Main.func1()
[E2021-12-21T17:56:44.610577Z 1059 0 annotate.go:241] #7 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
[E2021-12-21T17:56:44.610638Z 1059 0 annotate.go:241] #8 go.chromium.org/luci/vpython/application/application.go:407 - application.(*Config).Main()
[E2021-12-21T17:56:44.610696Z 1059 0 annotate.go:241] #9 vpython/main.go:109 - main.mainImpl()
[E2021-12-21T17:56:44.610749Z 1059 0 annotate.go:241] #10 vpython/main.go:115 - main.main()
[E2021-12-21T17:56:44.610805Z 1059 0 annotate.go:241] #11 runtime/proc.go:204 - runtime.main()
[E2021-12-21T17:56:44.610862Z 1059 0 annotate.go:241] #12 runtime/asm_arm.s:857 - runtime.goexit()

Comment 12

[tt_1]

There was an update to webrtc in the nightly branch, did it help with this bug? I can't test it on the spot, as I need to update my rust toolchain to a llvm-13 based one first.

Comment 13

[Mike Hommey [:glandium]]

Attachment: Bug 1738845 - Add webrtc configs for arm and ppc64 linux.

Adding arm alone breaks ppc64el.

Comment 14

[tt_1]

I updated to llvm-13 toolchain in the meantime, so will give the patch a try.

Can see some stray include dir, is this correct?

@@ -9069,6 +9451,7 @@
],
"include_dirs": [
"//",

•            "/tmp/central/obj-arm-unknown-linux-gnueabihf/third_party/libwebrtc/gn-output/gen/",
             "//third_party/abseil-cpp/"
         ],
         "libs": [],
  

Comment 15

[Mike Hommey [:glandium]]

Damn, forgot to re-run the fixer script on those jsons.

Comment 16

[tt_1]

Ok, I saw that you push forced the patch. Its seems to work just fine with my cross-gcc setup, but have no idea how it behaves at runtime. Can you please kindly test that on an actuall armv7a device?

Comment 17

[Olivier Tilloy]

I successfully rebuilt the firefox snap (96.0-2) on armhf with the following patches applied, and it seems to behave correctly at runtime, including WebRTC features (tested on a RPi4 running Ubuntu 20.04 armhf, with https://mozilla.github.io/webrtc-landing/gum_test.html):

• https://phabricator.services.mozilla.com/D133409
• https://phabricator.services.mozilla.com/D117104
• https://phabricator.services.mozilla.com/D134435
• https://hg.mozilla.org/mozilla-central/rev/3607c4937e84
• https://phabricator.services.mozilla.com/D134738

(the first 4 patches are pre-requisites for the last one to apply cleanly).

Comment 18

[Olivier Tilloy]

I have confirmed that D134738 fixes building (and running) Firefox 97.0 on armhf. What is missing to move forward with this and get the patch merged?

Comment 20

[Olivier Tilloy]

The patch that's up for review at D134738 doesn't apply cleanly on nightly any longer (it was applying until yesterday). Is there anything that we can do to help with moving this forward?

Comment 21

[Olivier Tilloy]

Mike, do the patches linked to bug 1773223 render D134738 unnecessary? Or will it need refreshing once all the former have landed?
If so, is it reasonable to expect an updated patch before the soft code freeze for 103.0 ?

The patch not applying any longer on the 102 branch means we cannot build Ubuntu packages for armhf, which is (for Ubuntu) a supported architecture.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/67ee3490fe22
Add webrtc configs for arm and ppc64 linux. r=mjf

Comment 23

[Cosmin Sabou [:CosminS]]

Backed out for causing build bustages.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=aFofmcymQfuaUogDoGLSTw.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=85f0cccbd2a8b007005aa155058ad9007323d83c

Failure logs: https://treeherder.mozilla.org/logviewer?job_id=381951166&repo=autoland

Comment 24

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/03c4ec9a5839
Backed out 11 changesets (bug 1738845, bug 1760484, bug 1773223) for causing build bustages and python failures. CLOSED TREE

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/653d2024a187
Add webrtc configs for arm and ppc64 linux. r=mjf

Comment 26

[Sandor Molnar[:smolnar]]

Backed out for causing multiple failures

Backout link: https://hg.mozilla.org/integration/autoland/rev/1bff014a2d90deeec6c821a7f1ee2a2fe3014bcb

Push with failures

Failure log mda

Failure log bc

Failure log wpt

Failure log crashtest

Comment 27

[Mike Hommey [:glandium]]

Attachment: Bug 1738845 - Add some missing combinations of conditions to GN processing.

Ideally, the code would handle things in a more general way that doesn't
require manually dealing with these lists, but this would require more
testing than there is time left before 102 releases.

While here, remove HOST_CPU_ARCH, which is always the same and thus
never appears in a condition.

This changes none of the generated moz.builds for the current
configuration (but changes the outcome when adding new configurations)

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4d109e7487ef
Add some missing combinations of conditions to GN processing. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/faf794de9900
Add webrtc configs for arm and ppc64 linux. r=mjf

Comment 29

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/4d109e7487ef
https://hg.mozilla.org/mozilla-central/rev/faf794de9900

Comment 30

[BugBot [:suhaib / :marco/ :calixte]]

Since nightly and release are affected, beta will likely be affected too.
$For more information, please visit auto_nag documentation.

Comment 31

[Mike Hommey [:glandium]]

Comment on attachment 9256948 [details]
Bug 1738845 - Add webrtc configs for arm and ppc64 linux.

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Broken builds for tier-3 downstreams.
• User impact if declined:
• Fix Landed on Version: 103
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): NPOTB

Comment 32

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:glandium, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox102 to wontfix.

For more information, please visit auto_nag documentation.

Comment 33

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9256948 [details]
Bug 1738845 - Add webrtc configs for arm and ppc64 linux.

Approved for 102.1.0esr.

Comment 34

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/251c39631403
https://hg.mozilla.org/releases/mozilla-esr102/rev/3447ce3987c3