1643258 - Firefox beta 78.0b1 requires glibc 2.18 on Linux, not 2.17 as announced

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Steve Common]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

Updated Firefox beta to 78.0b1 on Redhat Enterprise Linux 7 and tried to run it

Actual results:

Firefox no longer starts due to unannounced GLIBC 2.18 dependancy. GLIBC 2.18 is unavailable on Rehdat Enterprise linux 7 (2.17 max).

https://www.mozilla.org/en-US/firefox/78.0beta/system-requirements/

Requirement is given as GLIBC 2.17. Error messages when starting Firefox:
XPCOMGlueLoad error for file /usr/local/firefox/libxul.so: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /usr/local/firefox/libxul.so) Couldn't load XPCOM.

Expected results:

Firefox should have started normaly, as it did before updating to 78.0b1

Comment 1

[Clara Guerrero ( Need Info Brindusa Tot Please)]

Hi,

We do not have right environment set up in order to reproduce this issue but I will set the component for it and maybe one of our developers will be able to reproduce it on their end.

Thanks for the report.

Best regards,

Clara.

Comment 2

[Piotr Masek]

I have the exact same problem on an up-to-date Centos 7 machine, which is probably easier to set up, since it's free.
This bug also applies to FF 79 Nightly.

Comment 3

[Emilio Cobos Álvarez (:emilio)]

Is there any chance that you could run mozregression to figure out what regressed this?

Comment 4

[Darkspirit]

$ pip3 install --upgrade mozregression
$ ~/.local/bin/mozregression --good 77 --bad 78

Comment 5

[Piotr Masek]

Ok then, this is the result of mozregression:

6:08.87 INFO: Narrowed integration regression window from [6f03149b, 52cf30bf] (3 builds) to [6f03149b, 8621fde1] (2 builds) (~1 steps left)
6:08.87 INFO: No more integration revisions, bisection finished.
6:08.87 INFO: Last good revision: 6f03149b52094f6af4d842bfc40ce1ddf5a3b157
6:08.87 INFO: First bad revision: 8621fde19c3154f11758d8f9e0d2a22cfff1efed
6:08.87 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=6f03149b52094f6af4d842bfc40ce1ddf5a3b157&tochange=8621fde19c3154f11758d8f9e0d2a22cfff1efed

Comment 6

[Darkspirit]

Thanks!

Comment 7

[Mike Hommey [:glandium]]

Can you attach the full log when running with LD_DEBUG=all set in the environment?

Comment 8

[Piotr Masek]

Attachment: ld-debug.zip

log from LD_DEBUG=all on Centos 7

Comment 9

[Steve Common]

do you still "needinfo" from me or is Poitr's info sufficient (I think it is)?
I won't have access to the run environment on Red-hot Linux until Moday 15 June

Comment 10

[Mike Hommey [:glandium]]

Attachment: Bug 1643258 - Avoid the use of the __cxa_thread_atexit_impl symbol.

When linking a weak symbol in an object against a library where the
symbol is provided with a version, the final binary get a weak versioned
symbol reference.

It turns out weak versioned symbols still make the dynamic linker need
the symbol version, even if all symbols needed with that version are
weak.

Practically speaking, that means with bug 1634204, we now end up with
a weak versioned symbol reference to __cxa_thread_atexit_impl with
version GLIBC_2.18, and glibcs without the symbol can't fulfil that
version, even though the weak symbol is the only thing we need from that
version.

This means the check_binary changes in bug 1634204 are too
relaxed, so we revert them (although we keep the easier to read
conditions in check_dep_versions).

We also introduce a hack in stdc++compat.cpp (although it's not
technically entirely about libstdc++ compat) so that we avoid the weak
symbol reference while keeping the intended baseline for libstdc++ and
glibc.

Comment 11

[chris.moth]

This bug would be a lot less frustrating if I could revert to v77 and halt all updates. However, even after setting "Ask before updating", firefox updates to v78 in the next start and dies on CentOS7.

As others report:

XPCOMGlueLoad error for file /opt/firefox-developer/firefox/libxul.so:
/lib64/libc.so.6: version `GLIBC_2.18' not found (required by /opt/firefox-developer/firefox/libxul.so)
Couldn't load XPCOM.

Thanks for working on this - I so miss firefox developer edition!

I can imagine that many users are just seeing Firefox fail to start (since starting from GUI does not show this error message)... and sadly moving on to other browsers. Perhaps consider a GUI popup, a death rattle, in the event of a system level fail like this.

Even better, automatically revert to a prior version of firefox(!!!) and transmit a message to mozilla proactively.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5c7f4c2de55b
Avoid the use of the __cxa_thread_atexit_impl symbol. r=froydnj

Comment 13

[Razvan Maries]

Backed out for build bustages.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=linux%2Cx64%2Cdebug%2Cbuild-linux64-base-toolchains%2Fdebug%2C%28bb%29&revision=fc7a7af728149e531f5516624abca40f2cce1af6&selectedTaskRun=Iob3CXuHRlubU2WEeAFc0g.0

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306547621&repo=autoland&lineNumber=29572

Backout: https://hg.mozilla.org/integration/autoland/rev/9d296a4e3f82b9ea2462d772f5192a26b295091e

Comment 14

[Mike Hommey [:glandium]]

Attachment: Bug 1643258 - Disable stdc++ compat on base-toolchain builds.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/439eee50f354
Disable stdc++ compat on base-toolchain builds. r=firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/4c359d743fb2
Avoid the use of the __cxa_thread_atexit_impl symbol. r=froydnj

Comment 16

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/439eee50f354
https://hg.mozilla.org/mozilla-central/rev/4c359d743fb2

Comment 17

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e39dd3ccf240
Backed out 2 changesets for causing Bug 1646625.

Comment 18

[Atila Butkovits]

Backed out for causing Bug 1646625.

Backout link: https://hg.mozilla.org/integration/autoland/rev/e39dd3ccf240b1af8259681dbb1d3ab9795c26bf

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&revision=7f0b0cbecd946aee526a869853a46a14ee44b1f9&selectedTaskRun=Dde8tU5XQTq4Nf3Z24UBsA.0&searchStr=Linux%2Cx64%2CCCov%2Copt%2Cbuild-linux64-ccov%2Fopt%2C%28B%29

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306725659&repo=mozilla-central&lineNumber=29864

Comment 19

[Atila Butkovits]

Backout merged: https://hg.mozilla.org/mozilla-central/rev/e39dd3ccf240b1af8259681dbb1d3ab9795c26bf

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b707f591bef5
Avoid the use of the __cxa_thread_atexit_impl symbol. r=froydnj

Comment 21

[Dorel Luca [:dluca]]

Backed out changeset b707f591bef5 (bug 1643258) for build bustage. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306824592&repo=autoland&lineNumber=3735

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306824631&repo=autoland&lineNumber=3162

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=Zj8u83A_Tc-QuI0cjQ-g7Q.0&revision=b707f591bef54b35d90924104dd56d651e43078e

Backout:
https://hg.mozilla.org/integration/autoland/rev/aa7ed85575e0bde4adfd3f2c5f86d03a76222107

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/98efceb86ec5
Avoid the use of the __cxa_thread_atexit_impl symbol. r=froydnj

Comment 23

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/98efceb86ec5

Comment 25

[Mike Hommey [:glandium]]

Comment on attachment 9156875 [details]
Bug 1643258 - Avoid the use of the __cxa_thread_atexit_impl symbol.

Beta/Release Uplift Approval Request

• User impact if declined: Cannot start Firefox on some older Linux systems
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Trick to avoid using some symbols, which changes no functionality. If it broke anything, it would have been detected already. While there were several landings/backouts, none of them involved the trick breaking things, but some non-shipped builds using another symbol that wasn't expected to be used, for which another trick was added.
• String changes made/needed: N/A

Comment 26

[Julien Cristau [:jcristau]]

Comment on attachment 9156875 [details]
Bug 1643258 - Avoid the use of the __cxa_thread_atexit_impl symbol.

approved for 78 rc1

Comment 27

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/5f0dcca70da9

Comment 28

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/5f0dcca70da9f393f3800763306dc8f07821e84a