1805371 - Firefox built with --with-system-nss fails to start with 'libnssutil3.so: cannot open shared object file: No such file or directory'

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Joonas Niilola]

Attachment: firefox-108.0:20221213-081130.log.xz

Steps to reproduce:

Try to build the latest release 108.0 with --with-system-nss configure flag, AND with MOZ_PGO=1.

Actual results:

The build ends up in an error,

XPCOMGlueLoad error for file /var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/dist/firefox/libnssutil3.so:
/var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/dist/firefox/libnssutil3.so: cannot open shared object file: No such file or directory
Couldn't load XPCOM.

Longer message:

11:55.51 ../../config/nsinstall -D ../../dist/linux-x86_64/xpi/
11:55.52 /var/tmp/portage/www-client/firefox-108.0/work/firefox_build/_virtualenvs/build/bin/python -m mozbuild.action.langpack_manifest --locales en-US --app-version 108.0 --max-app-ver 108.* --app-name 'Firefox' --l10n-basedir '/var/tmp/portage/www-client/firefox-108.0/work/     firefox_build/l10n-central' --defines /var/tmp/portage/www-client/firefox-108.0/work/firefox-108.0/toolkit/locales/en-US/defines.inc /var/tmp/portage/www-client/firefox-108.0/work/firefox-108.0/browser/locales/en-US/defines.inc  --langpack-eid '[e-mail].      org' --input ../../dist/xpi-stage/locale-en-US
11:55.88 /var/tmp/portage/www-client/firefox-108.0/work/firefox_build/_virtualenvs/build/bin/python -m mozbuild.action.zip -C ../../dist/xpi-stage/locale-en-US -x **/*.manifest -x **/*.js -x **/*.ini /var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/dist/   linux-x86_64/xpi/firefox-108.0.en-US.langpack.xpi chrome localization browser manifest.json
11:56.06 gmake[4]: Leaving directory '/var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/browser/locales'
11:56.06 gmake[3]: Leaving directory '/var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/browser/locales'
11:56.06 gmake[2]: Leaving directory '/var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/browser/installer'
11:56.07 gmake[2]: Entering directory '/var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/browser/installer'
11:56.07 gmake[2]: Nothing to be done for 'tools'.
11:56.07 gmake[2]: Leaving directory '/var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/browser/installer'
11:56.07 gmake[1]: Leaving directory '/var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/browser/installer'
11:56.07 gmake: Leaving directory '/var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented'
XPCOMGlueLoad error for file /var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/dist/firefox/libnssutil3.so:
/var/tmp/portage/www-client/firefox-108.0/work/firefox_build/instrumented/dist/firefox/libnssutil3.so: cannot open shared object file: No such file or directory
Couldn't load XPCOM.
Firefox exited with code 255 during profile initialization
Error running mach:

    ['build', '--verbose']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file build| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/var/tmp/portage/www-client/firefox-108.0/work/firefox_build/_virtualenvs/build/bin/python', '/var/tmp/portage/www-client/firefox-108.0/work/firefox-108.0/build/pgo/profileserver.py']' returned non-zero exit status 255.

  File "/var/tmp/portage/www-client/firefox-108.0/work/firefox-108.0/python/mozbuild/mozbuild/build_commands.py", line 218, in build
    subprocess.check_call(pgo_cmd, cwd=instr.topobjdir, env=pgo_env)
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)

Full build.log is attached. It's compressed because the original log is 17 MB.

Expected results:

Building with PGO and system-nss support should succeed. --with-system-nss works without PGO, and PGO works without --with-system-nss.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::General' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Joonas Niilola]

I found https://bugzilla.mozilla.org/show_bug.cgi?id=1799121 which had the tip to try passing --enable-tests and indeed it does look like passing that makes the build succeed in every scenario. I tried reverting https://hg.mozilla.org/releases/mozilla-release/rev/2d9f62d21cc62d84c99cdd4d5fc356b6c8833d30 which then brought me to the issue at #1799121 - but I still believe this is a bug as behaviour has definitely changed since 107.0.1.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

:jschanck, since you are the author of the regressor, bug 1799121, could you take a look? Also, could you set the severity field?

For more information, please visit auto_nag documentation.

Comment 5

[Mike Hommey [:glandium]]

I can see two ways out here:

• disable FaultyServer and the corresponding tests when building with system nss (and change the USE_LIBS and DEFINES to use system nss to link the other tests)
• change the nss gyp files to add an option to only build the nss static libraries and enable that option in the Firefox build system

Comment 6

[John Schanck [:jschanck]]

Attachment: Bug 1805371 - avoid building and running FaultyServer tests with system NSS. r=glandium

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1799121

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a5df6a127e9d
avoid building and running FaultyServer tests with system NSS. r=glandium,necko-reviewers,kershaw

Comment 9

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/a5df6a127e9d

Comment 10

[John Schanck [:jschanck]]

Comment on attachment 9308785 [details]
Bug 1805371 - avoid building and running FaultyServer tests with system NSS. r=glandium

Beta/Release Uplift Approval Request

• User impact if declined: Some builds using --with-system-nss will fail.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): There's no impact on official Firefox builds, which do not use --with-system-nss.
• String changes made/needed:
• Is Android affected?: Unknown

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9308785 [details]
Bug 1805371 - avoid building and running FaultyServer tests with system NSS. r=glandium

Approved for 109.0b9.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/25b323cc633b