1855568 - Crash in [@ ld-linux-x86-64.so.2@0xbc73]

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

There had been isolated instances of these signatures before but they affect several Nightly users since 120.0a1 20230926092749 (list)

Mike, could this be related to the changes in bug 1854047 or bug 1839739?

Crash report: https://crash-stats.mozilla.org/report/index/56156b6a-12b7-40be-8d68-b5c5d0230927

Reason: SIGSEGV / SEGV_MAPERR

Top 10 frames of crashing thread:

0  ld-linux-x86-64.so.2  ld-linux-x86-64.so.2@0xbc73  
1  ld-linux-x86-64.so.2  ld-linux-x86-64.so.2@0x1476b  
2  ld-linux-x86-64.so.2  ld-linux-x86-64.so.2@0xf8d3  
3  ld-linux-x86-64.so.2  ld-linux-x86-64.so.2@0x1425f  
4  ld-linux-x86-64.so.2  ld-linux-x86-64.so.2@0x13c8a  
5  libnspr4.so  pr_LoadLibraryByPathname  nsprpub/pr/src/linking/prlink.c:584
5  libnspr4.so  PR_LoadLibraryWithFlags  nsprpub/pr/src/linking/prlink.c:386
6  libdl.so.2  libdl.so.2@0xeea  
7  libdl.so.2  libdl.so.2@0xe8f  
8  ld-linux-x86-64.so.2  ld-linux-x86-64.so.2@0xf8d3  

Comment 1

[Mike Hommey [:glandium]]

Mike, could this be related to the changes in bug 1854047 or bug 1839739?

I don't think so, but then, it's also hard to say anything, because those stack traces are 100% useless. NSS might as well not be involved at all.

Comment 2

[Mike Hommey [:glandium]]

Gabriele, can we get the symbols for those missing libs?

Comment 3

[Gabriele Svelto [:gsvelto]]

Lots of Debian 8 users... I think we can, let me check

Comment 4

[Gabriele Svelto [:gsvelto]]

Scraping is underway, let's hope the symbols we look for are still on archive.debian.org

Comment 5

[Gabriele Svelto [:gsvelto]]

I found the symbols, I'll try and reprocess the crashes.

Comment 6

[Gabriele Svelto [:gsvelto]]

Alright, here comes the new crash signature.

Comment 7

[Gabriele Svelto [:gsvelto]]

Reprocessed crashes look like this: https://crash-stats.mozilla.org/report/index/c5b8ca7c-ece4-41d3-9148-ff38e0230928

Reason: SIGSEGV / SEGV_MAPERR

Top 10 frames of crashing thread:

0  ld-linux.so.2  elf_machine_rel  /build/glibc-PNN5fi/glibc-2.19/sysdeps/i386/dl-machine.h:337
0  ld-linux.so.2  elf_dynamic_do_Rel  /build/glibc-PNN5fi/glibc-2.19/elf/do-rel.h:137
0  ld-linux.so.2  _dl_relocate_object  /build/glibc-PNN5fi/glibc-2.19/elf/dl-reloc.c:264
1  ld-linux.so.2  dl_open_worker  /build/glibc-PNN5fi/glibc-2.19/elf/dl-open.c:427
2  ld-linux.so.2  _dl_catch_error  /build/glibc-PNN5fi/glibc-2.19/elf/dl-error.c:187
3  libdl.so.2  dlopen_doit  /build/glibc-PNN5fi/glibc-2.19/dlfcn/dlopen.c:66
4  ld-linux.so.2  _dl_catch_error  /build/glibc-PNN5fi/glibc-2.19/elf/dl-error.c:187
5  libdl.so.2  __dlopen_check  /build/glibc-PNN5fi/glibc-2.19/dlfcn/dlopen.c:87
6  libxul.so  mozilla::MozAVLink  dom/media/platforms/ffmpeg/ffvpx/FFVPXRuntimeLinker.cpp:45
7  libxul.so  mozilla::FFVPXRuntimeLinker::Init  dom/media/platforms/ffmpeg/ffvpx/FFVPXRuntimeLinker.cpp:94

Comment 8

[:gerard-majax]

similar stack on https://github.com/LMMS/lmms/issues/1329 ?

Comment 9

[Dianna Smith [:diannaS]]

tracking because it seems to be alot of separate occurrences

Comment 10

[:gerard-majax]

(In reply to Dianna Smith [:diannaS] from comment #9)

tracking because it seems to be alot of separate occurrences

it's only debian 8 32-bits, and looks like a crash in libc. It's likely triggered by utility audio decoder that tries to load ffmpeg from the system. Media code will retry on process death, there should be throttling ...

Comment 11

[Mike Hommey [:glandium]]

What about the scientific linux ones, they have different things in their stack traces?

Comment 12

[:gerard-majax]

looking https://crash-stats.mozilla.org/signature/?product=Firefox&signature=elf_machine_rel&date=%3E%3D2023-02-01T05%3A07%3A00.000Z&date=%3C2023-09-29T05%3A07%3A00.000Z#aggregations we have 63% on RDD and 27% on Utility. Interestingly, it started to get reported with 20230926092749 ?

Comment 13

[Mike Hommey [:glandium]]

That would correlate with bug 1839739.

Comment 14

[Mike Hommey [:glandium]]

This the return of the revenge of https://sourceware.org/bugzilla/show_bug.cgi?id=14341

Comment 15

[Mike Hommey [:glandium]]

It doesn't affect libxul because it's not loaded with RTLD_NOW, while the code that loads mozavutil uses RTLD_NOW (likewise for some nss code).

Comment 16

[Gabriele Svelto [:gsvelto]]

(In reply to Mike Hommey [:glandium] from comment #11)

What about the scientific linux ones, they have different things in their stack traces?

I haven't had much chances with them yet. OpenSUSE ones seem unrelated and come from a single user, see this reprocessed crash.

Comment 17

[Gabriele Svelto [:gsvelto]]

CentOS 7 crashes are related and I could find the symbols, adding the signature and here's an example.

Comment 18

[Gabriele Svelto [:gsvelto]]

Old Ubuntu is also affected, same signature as CentOS 7, see this crash.

Comment 19

[Gabriele Svelto [:gsvelto]]

Found some symbols for older Debian versions too, expect some extra volume as Socorro reprocesses crashes and rebuilds its indexes with the new signatures.

Comment 20

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to topcrash signatures, which match the following criterion:

• Top 10 desktop browser crashes on nightly (startup)

For more information, please visit BugBot documentation.

Comment 21

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox120 (nightly). However, the bug still isn't assigned.

:ckerschb, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 22

[Mike Hommey [:glandium]]

The bug was fixed by backing out the causes of it. Let's keep the bug open for an actual fix that would allow relanding.

Comment 23

[Mike Hommey [:glandium]]

Attachment: Bug 1855568 - Refactor writes in relrhack.

In a manner similar to what's done for reads.

Comment 24

[Mike Hommey [:glandium]]

Attachment: Bug 1855568 - Refactor how the PT_DYNAMIC info is collected.

We're soon going to collect a little more, and the current way of doing
so is not super convenient.

Comment 25

[Mike Hommey [:glandium]]

Attachment: Bug 1855568 - Ensure that .relr.plt follows and is adjacent to .rel.dyn.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/871f72b6c738
Refactor writes in relrhack. r=sergesanspaille
https://hg.mozilla.org/integration/autoland/rev/b99fc81764f7
Refactor how the PT_DYNAMIC info is collected. r=sergesanspaille
https://hg.mozilla.org/integration/autoland/rev/d8a68b5a6687
Ensure that .relr.plt follows and is adjacent to .rel.dyn. r=sergesanspaille

Comment 27

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/871f72b6c738
https://hg.mozilla.org/mozilla-central/rev/b99fc81764f7
https://hg.mozilla.org/mozilla-central/rev/d8a68b5a6687