1807993 - Perma [Tier 2] Android searchfox /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:30: error: undefined reference to 'Servo_AnimationValue_AddRef'

Status: RESOLVED FIXED

(Webtools :: Searchfox, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=401003939&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TFfYTOe5SMug5j0z4MaR5w/runs/0/artifacts/public/logs/live_backing.log

---

TEST-PASS | check_mozglue_order | libipcclientcerts.so
[task 2022-12-30T05:31:59.662Z] 05:31:59     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library'
[task 2022-12-30T05:31:59.663Z] 05:31:59     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library'
[task 2022-12-30T05:31:59.663Z] 05:31:59     INFO -  chmod +x libipcclientcerts.so
[task 2022-12-30T05:31:59.663Z] 05:31:59     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library'
[task 2022-12-30T05:31:59.664Z] 05:31:59     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library'
[task 2022-12-30T05:31:59.664Z] 05:31:59     INFO -  ../../../../../config/nsinstall -R -m 644 'libipcclientcerts.so' '../../../../../dist/bin'
[task 2022-12-30T05:31:59.664Z] 05:31:59     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library'
[task 2022-12-30T05:32:00.653Z] 05:32:00     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library'
[task 2022-12-30T05:32:00.653Z] 05:32:00     INFO -  /builds/worker/workspace/obj-build/_virtualenvs/build/bin/python -m mozbuild.action.dumpsymbols /builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library/libipcclientcerts.so /builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library/libipcclientcerts.so_syms.track
[task 2022-12-30T05:32:00.653Z] 05:32:00     INFO -  Running: /builds/worker/workspace/obj-build/_virtualenvs/build/bin/python /builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py -c --vcs-info --install-manifest=/builds/worker/workspace/obj-build/_build_manifests/install/dist_include,/builds/worker/workspace/obj-build/dist/include -s /builds/worker/checkouts/gecko /builds/worker/fetches/dump_syms/dump_syms /builds/worker/workspace/obj-build/dist/crashreporter-symbols /builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library/libipcclientcerts.so
[task 2022-12-30T05:32:00.653Z] 05:32:00     INFO -  Beginning work for file: /builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library/libipcclientcerts.so
[task 2022-12-30T05:32:00.654Z] 05:32:00     INFO -  Processing file: /builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library/libipcclientcerts.so
[task 2022-12-30T05:32:00.654Z] 05:32:00     INFO -  /builds/worker/fetches/dump_syms/dump_syms --inlines /builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library/libipcclientcerts.so
[task 2022-12-30T05:32:00.654Z] 05:32:00     INFO -  Finished processing /builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library/libipcclientcerts.so in 0.18s
[task 2022-12-30T05:32:00.654Z] 05:32:00     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/security/manager/ssl/ipcclientcerts/dynamic-library'
[task 2022-12-30T05:32:24.996Z] 05:32:24     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/toolkit/library/build'
[task 2022-12-30T05:32:24.998Z] 05:32:24     INFO -  /builds/worker/fetches/clang/bin/clang++ -Qunused-arguments -std=gnu++17 --target=arm-linux-androideabi --rtlib=libgcc -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -ftrivial-auto-var-init=pattern -isystem /builds/worker/fetches/android-ndk/sysroot/usr/include/arm-linux-androideabi -isystem /builds/worker/fetches/android-ndk/sysroot/usr/include --gcc-toolchain=/builds/worker/fetches/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=16 -mno-unaligned-access -fno-sized-deallocation -fno-aligned-new -fno-short-enums -fno-exceptions -fcrash-diagnostics-dir=/builds/worker/artifacts -stdlib=libstdc++ -I/builds/worker/fetches/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/fetches/android-ndk/sources/android/support/include -I/builds/worker/fetches/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=neon -mfloat-abi=softfp -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -gdwarf-4 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -Xclang -add-plugin -Xclang mozsearch-index -Xclang -plugin-arg-mozsearch-index -Xclang /builds/worker/checkouts/gecko -Xclang -plugin-arg-mozsearch-index -Xclang /builds/worker/workspace/obj-build/mozsearch_index -Xclang -plugin-arg-mozsearch-index -Xclang /builds/worker/workspace/obj-build -Oz -mno-outline -fno-omit-frame-pointer -funwind-tables -Werror  -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libxul.so -o libxul.so /builds/worker/workspace/obj-build/toolkit/library/build/libxul_so.list   --gcc-toolchain=/builds/worker/fetches/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -fuse-ld=gold -L/builds/worker/fetches/android-ndk/platforms/android-16/arch-arm/usr/lib -Wl,--rpath-link=/builds/worker/fetches/android-ndk/platforms/android-16/arch-arm/usr/lib --sysroot /builds/worker/fetches/android-ndk/platforms/android-16/arch-arm -mthumb -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 -Wl,--icf=safe -Wl,--hash-style=sysv -fstack-protector-strong -Wl,-rpath-link,/builds/worker/workspace/obj-build/dist/bin -Wl,-rpath-link,/usr/local/lib  ../../../security/nss/lib/crmf/crmf_crmf/libcrmf.a ../../../js/src/build/libjs_static.a /builds/worker/workspace/obj-build/thumbv7neon-linux-androideabi/debug/libgkrust.a ../../../security/libnss3.so ../../../config/external/lgpllibs/liblgpllibs.so ../../../mozglue/build/libmozglue.so   -landroid -llog -landroid_support -lunwind -lGLESv2 -lm -ldl -L/builds/worker/fetches/android-ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a -lc++_static -lc++abi
[task 2022-12-30T05:32:24.998Z] 05:32:24    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:30: error: undefined reference to 'Servo_AnimationValue_AddRef'
[task 2022-12-30T05:32:24.998Z] 05:32:24    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:30: error: undefined reference to 'Servo_AnimationValue_Release'
[task 2022-12-30T05:32:24.998Z] 05:32:24    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:30: error: undefined reference to 'Servo_AnimationValue_Release'
[task 2022-12-30T05:32:24.998Z] 05:32:24    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:30: error: undefined reference to 'Servo_AnimationValue_AddRef'
[task 2022-12-30T05:32:24.998Z] 05:32:24    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoBindingTypes.h:155: error: undefined reference to 'Servo_ComputedStyle_Release'
[task 2022-12-30T05:32:24.999Z] 05:32:24    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:27: error: undefined reference to 'Servo_DeclarationBlock_Release'
[task 2022-12-30T05:32:24.999Z] 05:32:24    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:27: error: undefined reference to 'Servo_DeclarationBlock_AddRef'
[task 2022-12-30T05:32:24.999Z] 05:32:24    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:27: error: undefined reference to 'Servo_DeclarationBlock_Release'
[task 2022-12-30T05:32:24.999Z] 05:32:24    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoBindingTypes.h:155: error: undefined reference to 'Servo_ComputedStyle_Release'
[task 2022-12-30T05:32:24.999Z] 05:32:24    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:27: error: undefined reference to 'Servo_DeclarationBlock_AddRef'
[task 2022-12-30T05:32:25.000Z] 05:32:25    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:26: error: undefined reference to 'Servo_StyleSheetContents_Release'
[task 2022-12-30T05:32:25.000Z] 05:32:25    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:44: error: undefined reference to 'Servo_FontFaceRule_Release'
[task 2022-12-30T05:32:25.000Z] 05:32:25    ERROR -  /builds/worker/workspace/obj-build/dist/include/mozilla/ServoArcTypeList.h:41: error: undefined reference to 'Servo_ContainerRule_Release'

Comment 1

[Emilio Cobos Álvarez (:emilio)]

I'm confused about why you think bug 1797752 regressed this (and if you know / bisected, then I'm a bit confused at how could it possibly do so...)

It didn't touch rust code at all, and it seems there's a lot of rust-defined symbols missing in the log (e.g., all the style crate symbols)? The linker change seems like a more likely culprit?

Comment 2

[Mike Hommey [:glandium]]

I haven't looked at what's going on, but it clearly started happening on your push: https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=searchfox&tochange=b0a9b9fd9d4f509a9eb83356376a8bf96f4bba10&fromchange=765ede8173e41ec3d0ef1f27cf03c805f1094ad8 and I further tested with bug 1806905 only, and it was green.

Comment 3

[Emilio Cobos Álvarez (:emilio)]

I'm confused, backing out that patch doesn't fix it. Reverting the linker patch doesn't either fwiw, but that's probably expected since the failing command has -fuse-ld=gold.

Mike, I'll try to look at this as time allows but it seems like the symbols from gkrust (well, from the style crate at least) are not getting linked... Do you know what might be going on?

I'll try to make the searchfox android debug builds more similar to regular android debug builds which do work... Some things I've noticed are:

• Different rust version (expected because of the save-analysis stuff).
• Different android version (android 5 for regular debug builds vs. 4 for searchfox).
• sccache vs. not (though I don't think that would matter)
• Mozsearch plug-in (also expected of course, and I'd be surprised if it was the culprit but you never know...).

Comment 4

[Intermittent Failures Robot]

40 failures in 1328 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #10 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 33
• mozilla-central: 7

Platform and build breakdown:

• android-4-1-armv7: 40
  • debug: 40

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1807993&startday=2022-12-26&endday=2023-01-01&tree=all

Comment 5

[Emilio Cobos Álvarez (:emilio)]

This "fixes" it (though it's a bit unfortunate because we'd lose the proper scip symbols it might be a good short-term fix): https://hg.mozilla.org/try/rev/689619bc06d05e7aeeb59b0a37eb8f739e4bc590

But do you know what can explain it?

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

Did something land to fix this? arm v7 builds have been passing since https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=searchfox&revision=764c051fe1d40356b1a565c163159afa7730cfd9 again.

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Nothing in the first successful push looks related.

Comment 8

[Emilio Cobos Álvarez (:emilio)]

https://hg.mozilla.org/mozilla-central/rev/b13639699fa72bccca9352362ea05b84d87ad866 touched style code. But why it failed before is still a mystery to me.

Comment 9

[Intermittent Failures Robot]

1 failures in 2607 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• android-4-1-armv7: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1807993&startday=2023-01-02&endday=2023-01-08&tree=all

Comment 10

[Mike Hommey [:glandium]]

It might have been a bug in the version of the gold linker in the NDK, because that's not happening when forcing the use of lld. I'll try to preemptively switch to a more recent version of gold.

Comment 11

[Mike Hommey [:glandium]]

This happens with the latest version of gold too (2.39), as well as bfd. Considering how innocuous changes both triggered this and untriggered it, there must be some kind of corner case happening in binutils. I'll file a bug there. In a not too distant future, we'll switch to lld, so this should be less of a problem. In the meanwhile, since this is gone, let's close the bug for now. I'll add a link to the binutils bug when I'm done filing it.

Comment 12

[Mike Hommey [:glandium]]

binutils bug: https://sourceware.org/bugzilla/show_bug.cgi?id=29980

Comment 13

[Andrew Sutherland [:asuth] (he/him)]

Thanks so much for digging into this and filing the upstream bug!

Comment 14

[Atila Butkovits]

Started failing again: https://treeherder.mozilla.org/logviewer?job_id=402659981&repo=mozilla-central&lineNumber=65984

Comment 15

[Mike Hommey [:glandium]]

An easy out here would be to force those builds to use lld. (--enable-linker=lld)

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Mike Hommey [:glandium] from comment #15)

An easy out here would be to force those builds to use lld. (--enable-linker=lld)

Thanks!

So I see at https://searchfox.org/mozilla-central/rev/738b761bb2847f609f9cacc550680071cdc53637/taskcluster/ci/searchfox/kind.yml#162-170 we have a custom-build-variant-cfg:

run:
    using: mozharness
    actions: [get-secrets, build]
    config:
        - builds/releng_base_android_64_builds.py
    script: "mozharness/scripts/fx_desktop_build.py"
    secrets: true
    custom-build-variant-cfg: arm-debug-searchfox
    tooltool-downloads: internal

Which seems to get mapped in https://searchfox.org/mozilla-central/rev/738b761bb2847f609f9cacc550680071cdc53637/testing/mozharness/mozharness/mozilla/building/buildbase.py#304

"arm-debug-searchfox": path_base + "%s_arm_debug_searchfox.py",

which is https://searchfox.org/mozilla-central/rev/738b761bb2847f609f9cacc550680071cdc53637/testing/mozharness/configs/builds/releng_sub_android_configs/64_arm_debug_searchfox.py#5-12

config = {
    "stage_platform": "android-arm-debug",
    "env": {
        "SCCACHE_DISABLE": "1",
    },
    "src_mozconfig": "mobile/android/config/mozconfigs/android-arm/debug-searchfox",
    "debug_build": True,
}

which is https://searchfox.org/mozilla-central/rev/738b761bb2847f609f9cacc550680071cdc53637/mobile/android/config/mozconfigs/android-arm/debug-searchfox#1-18

. "$topsrcdir/mobile/android/config/mozconfigs/common"

# Global options
ac_add_options --enable-debug

# Android
ac_add_options --target=arm-linux-androideabi

export MOZILLA_OFFICIAL=1

ac_add_options --with-branding=mobile/android/branding/nightly

# Save rust analysis (this requires unlocking the unstable features,
# which is done in the taskcluster task definition via RUSTC_BOOTSTRAP)
export RUSTFLAGS="-Zsave-analysis"
ac_add_options --enable-mozsearch-plugin

. "$topsrcdir/mobile/android/config/mozconfigs/common.override"

So we can put it in that last file.

I'm getting some guidance to avoid spending too much time on searchfox so I won't be able to get to this for a while.

Comment 17

[Intermittent Failures Robot]

11 failures in 3717 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10
• toolchains: 1

Platform and build breakdown:

• android-4-1-armv7: 11
  • debug: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1807993&startday=2023-01-16&endday=2023-01-22&tree=all

Comment 18

[Mike Hommey [:glandium]]

... and this ends up (unsurprisingly) affecting normal android builds too when we upgrade to 1.67.

Comment 19

[Mike Hommey [:glandium]]

(debug only, though)

Comment 20

[Mike Hommey [:glandium]]

Attachment: Bug 1807993 - Switch armv7 android debug builds to lld.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7263b929d6aa
Switch armv7 android debug builds to lld. r=firefox-build-system-reviewers,geckoview-reviewers,nalexander,owlish

Comment 22

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/7263b929d6aa

Comment 23

[Intermittent Failures Robot]

1 failures in 3276 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• macosx1015-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1807993&startday=2023-07-31&endday=2023-08-06&tree=all