1847098 - Add version number of libraries (e.g. egl-wayland) to crash reports.

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, enhancement)

Description

[Brad Werth [:bradwerth]]

This would be very useful data to have when we investigate Wayland crashes.

Comment 1

[Brad Werth [:bradwerth]]

Perhaps the right place to add this is as additional information the App Notes part of the crash report. We already report when EGL is intialized. Maybe it would be fine to add a version number into that initialization string.

Comment 2

[Brad Werth [:bradwerth]]

Attachment: Bug 1847098: Add a string 'EGL v<version>' to the App Notes in GLLibraryEGL::Init.

This also drive-by changes GLLibraryEGL::Init to update the App Notes when
the call has completed successfully.

Comment 3

[Kelsey Gilbert [:jgilbert]]

What we probably want is the wayland module library version, but none of the Modules listed for Linux crash reports have versions, and that's probably what we want to fix.

Comment 4

[Gabriele Svelto [:gsvelto]]

What are the libraries you might be interested in specifically? We have some information about the library versions in the minidumps because we have the proc map; however not all libraries have versions encoded in their filename. In some cases the filename will just be something like libX11.so.6 which doesn't tell you much. Poking inside the binaries doesn't help either, at best you have a buildid which - with some patience - you can map back to the corresponding package and version.

Comment 5

[Ashley Hale [:ahale]]

We want to know the version of Mesa that is running when a crash occurs, because from a gfx standpoint it matters a great deal if Mesa had a regression we need to look into, differentiating crash reports by Mesa version is meaningful, much like what code path the driver is using for the chosen GPU is relevant to us.

Comment 6

[Gabriele Svelto [:gsvelto]]

The version is already available and can be aggregated upon, when looking at a crash signature go on the "Aggregations" tab and select "adapter driver version". You'll get something like this screenshot.

Comment 7

[Gabriele Svelto [:gsvelto]]

Attachment: Aggregating crashes on Mesa versions

Comment 8

[Kelsey Gilbert [:jgilbert]]

That's the active gpu driver version, but not necessarily the version of e.g. libwayland, which I fear is orthogonal.
We do get lucky in at least one of our compelling usecases, and the versions we need appear to be "accidentally" revealed via the crash stack frame source location paths, but it would be better if we could:

• Aggregate on versions of libraries
• See the versions of loaded libs that don't happen to reveal their versions via crash stack frame source location paths.

Comment 9

[Gabriele Svelto [:gsvelto]]

OK, I think that I might extract this information from the contents of /proc/<pid>/maps for the libraries that have it in their names, and surface that information in the modules tab. For other libraries there's not much we can do (for example libGLX.so, libGLdispatch.so and
libOpenGL.so will always use 0.0.0. I don't know why, probably compatibility reasons).

Comment 10

[Kelsey Gilbert [:jgilbert]]

We think it provide valuable assistance with sec-high bugs that we have, if you think it's actionable.

Comment 11

[:gerard-majax]

Quick check on my main process, ~152 libs, out of which we have soname with version on 134 of them and 18 without:

$ cat /proc/2281434/maps |egrep '\.so.*$' |grep -v bin/firefox | awk '{ print $6 }' | uniq | egrep '\.so$'
/usr/lib/x86_64-linux-gnu/libvulkan_intel.so
/usr/lib/x86_64-linux-gnu/libvulkan_intel_hasvk.so
/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so
/usr/lib/x86_64-linux-gnu/libvulkan_lvp.so
/usr/lib/x86_64-linux-gnu/libvulkan_virtio.so
/usr/lib/x86_64-linux-gnu/libcanberra-0.30/libcanberra-pulse.so
/usr/lib/x86_64-linux-gnu/gio/modules/libgioremote-volume-monitor.so
/usr/lib/x86_64-linux-gnu/onepin-opensc-pkcs11.so
/usr/lib/x86_64-linux-gnu/libVkLayer_MESA_device_select.so
/usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-16.1.so
/usr/lib/x86_64-linux-gnu/gtk-3.0/modules/libcanberra-gtk3-module.so
/usr/lib/x86_64-linux-gnu/dri/radeonsi_dri.so
/usr/lib/x86_64-linux-gnu/gtk-3.0/3.0.0/immodules/im-wayland.so
/usr/lib/x86_64-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-svg.so
/usr/lib/x86_64-linux-gnu/libelf-0.189.so
/usr/lib/x86_64-linux-gnu/gvfs/libgvfscommon.so
/usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so
/usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so
$ cat /proc/2281434/maps |egrep '\.so.*$' |grep -v bin/firefox | awk '{ print $6 }' | uniq | egrep -c '\.so$'
18
$ cat /proc/2281434/maps |egrep '\.so.*$' |grep -v bin/firefox | awk '{ print $6 }' | uniq | egrep -vc '\.so$'
134
$ cat /proc/2281434/maps |egrep '\.so.*$' |grep -v bin/firefox | awk '{ print $6 }' | uniq | wc -l 
152

Comment 12

[Gabriele Svelto [:gsvelto]]

Different packages use different conventions for naming, most of them used to adapt to libtool versioning, so for example:

• libglib-2.0.so.0.7800.3 is glib 2.78.3 (generally the minor version is multiplied by 100 and stored as the second number in the filename, I have some recollection that's what libtool calls the interface number, the last is the micro version but not always)
• libgtk-3.so.0.2406.32 is GTK+ 3.24.38 (note how the micro version is 06 + 32, there's a rule about that but I can't find it)

Others use plain versioning:

• libnvidia-egl-wayland.so.1.1.11 is egl-wayland 1.1.11

Others use no versioning at all:

• libGLX.so.0 is always libGLX.so.0.0.0 for all versions

Comment 13

[:gerard-majax]

(In reply to Gabriele Svelto [:gsvelto] from comment #12)

Different packages use different conventions for naming, most of them used to adapt to libtool versioning, so for example:

• libglib-2.0.so.0.7800.3 is glib 2.78.3 (generally the minor version is multiplied by 100 and stored as the second number in the filename, I have some recollection that's what libtool calls the interface number, the last is the micro version but not always)
• libgtk-3.so.0.2406.32 is GTK+ 3.24.38 (note how the micro version is 06 + 32, there's a rule about that but I can't find it)

That's only relevant if we want to present some "sensical" version to the user, but just collecting the version from the filename in this case is enough to perform some filtering (although imperfect). But at least if we do that server-side then we can issue updates of how we parse and expose the information in case we have some wrong heuristic.

Others use plain versioning:

• libnvidia-egl-wayland.so.1.1.11 is egl-wayland 1.1.11

Easy.

Others use no versioning at all:

• libGLX.so.0 is always libGLX.so.0.0.0 for all versions

We're stuck so far, but as of bug 1870823 we can also source the distro package information for the distributions we scrap from and that should get us an actionable version number, both pleasant to display and useful to track back the matching source code of libraries.

Comment 14

[:gerard-majax]

Attachment: Bug 1847098 - Report library version from filename at crash r?gsvelto!

Comment 15

[:gerard-majax]

Attachment: GitHub Pull Request for minidump-writer

Comment 16

[:gerard-majax]

https://crash-stats.mozilla.org/report/index/ecce9e39-ddb1-4595-a3e2-da7010240222#tab-modules was generated using the latest patches, and the version column makes sense and matches my local view as long as you ignore the second digit, B in A.B.C.D. e.g., libdbus-1 is 3.32.4

Comment 17

[:gerard-majax]

Attachment: GitHub Pull Request for rust-minidump

Comment 18

[:gerard-majax]

new scheme: https://crash-stats.mozilla.org/report/index/7214d5b5-998b-42af-9598-9b3d60240223#tab-modules

we end up using file_version_hi, file_version_lo and product_version_hi: anything can be a MAJOR.minor.release even pure strings. Checking on Ubuntu and Debian, we actually dont see any library exposing their filename version using strings:

ebook-dev-alp: /usr/share/doc/ebook-dev-alp/listings/chapter-2/libtest.so.gz
glibc-source: /usr/src/glibc/debian/local/etc/ld.so.conf
glibc-source: /usr/src/glibc/debian/patches/any/local-ldconfig-ignore-ld.so.diff
libc-bin: /etc/ld.so.conf
libcamera-ipa: /usr/lib/x86_64-linux-gnu/libcamera/ipa_ipu3.so.sign
libcamera-ipa: /usr/lib/x86_64-linux-gnu/libcamera/ipa_vimc.so.sign
libgdome2-ocaml-dev: /usr/lib/ocaml/stublibs/dllmlgdome.so.owner
libkcapi-dev: /usr/lib/x86_64-linux-gnu/.libkcapi.so.hmac
libpamtest0: /usr/lib/x86_64-linux-gnu/libpamtest.so.SOVERSION
librust-lddtree-dev: /usr/share/cargo/registry/lddtree-0.3.3/tests/ld.so.conf
libzia: /usr/lib/x86_64-linux-gnu/libzia/libzia-4.36.so.d
odoo-16: /usr/lib/python3/dist-packages/odoo/addons/point_of_sale/tools/posbox/overwrite_after_init/etc/ld.so.preload
odoo-16: /usr/lib/python3/dist-packages/odoo/addons/point_of_sale/tools/posbox/overwrite_before_init/etc/ld.so.preload
rear: /usr/share/rear/skel/default/etc/ld.so.conf
rust-1.68-src: /usr/src/rustc-1.68.2/src/ci/docker/host-x86_64/dist-mips-linux/patches/glibc/2.23/0001-MIPS-SPARC-fix-wrong-vfork-aliases-in-libpthread.so.patch
rust-src: /usr/src/rustc-1.71.1/src/ci/docker/host-x86_64/dist-mips-linux/patches/glibc/2.23/0001-MIPS-SPARC-fix-wrong-vfork-aliases-in-libpthread.so.patch
rust-src: /usr/src/rustc-1.72.1/src/ci/docker/host-x86_64/dist-mips-linux/patches/glibc/2.23/0001-MIPS-SPARC-fix-wrong-vfork-aliases-in-libpthread.so.patch
rust-src: /usr/src/rustc-1.73.0/src/ci/docker/host-x86_64/dist-mips-linux/patches/glibc/2.23/0001-MIPS-SPARC-fix-wrong-vfork-aliases-in-libpthread.so.patch
scribus-data: /usr/share/scribus/translations/scribus.so.qm
xppaut: /usr/share/doc/xppaut/examples/ode/example.so.gz
xppaut: /usr/share/doc/xppaut/examples/ode/getmax.so.gz
xppaut: /usr/share/doc/xppaut/examples/ode/getmin.so.gz
xppaut: /usr/share/doc/xppaut/examples/ode/t.so.gz

But technically it does work. There are some libraries that relies on libtool as mentionned earlier for their versions, and some have other semantic ; libabseil seems to use date stamps (YYYYMMJJ) as MAJOR which defeats the first implem where we assumed nothing more than u16. Moving to u32 allows us to handle enoug values that we should not have to worry.

Feeding this way exposes the version on crash-stats in the format 0.MAJOR.0.MINOR so we are loosing a bit of information, but it should be okay with https://github.com/rust-minidump/rust-minidump/pull/955

If any library has something else than pure digits, we would report version 0.0.0, and we fall into the same problem as libraries that don't report anything at all, so we will rely on bug 1870823 for that.

Comment 19

[:gerard-majax]

Attachment: WIP: Bug 1847098 - Update for https://github.com/rust-minidump/minidump-writer/pull/104

Depends on D202249

Comment 20

[Alex Franchuk [:afranchuk]]

afranchuk merged PR [rust-minidump/rust-minidump]: Bug 1847098 - Parse version number for Elf files (#955) in 9030821.

Comment 21

[:gerard-majax]

(In reply to Alex Franchuk [:afranchuk] from comment #20)

afranchuk merged PR [rust-minidump/rust-minidump]: Bug 1847098 - Parse version number for Elf files (#955) in 9030821.

I guess this needs release + updating the vendored crate on gecko side ?

Comment 22

[Alex Franchuk [:afranchuk]]

(In reply to :gerard-majax from comment #21)

(In reply to Alex Franchuk [:afranchuk] from comment #20)

afranchuk merged PR [rust-minidump/rust-minidump]: Bug 1847098 - Parse version number for Elf files (#955) in 9030821.

I guess this needs release + updating the vendored crate on gecko side ?

Yes. Conveniently, it was merged in an effort to publish and vendor a new release shortly :)

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/50ce57f9d5c5
Report library version from filename at crash r=gsvelto

Comment 24

[Serban Stanca [:SerbanS]]

Backed out for causing toolchain build bustages in linux_dumper.h.

• Backout link
• Push with failures
• Failure Log
• Failure line: /builds/worker/workspace/build/src/toolkit/crashreporter/breakpad-client/linux/minidump_writer/linux_dumper.h:60:12: fatal error: 'linux_utils.h' file not found

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4399c5a65886
Report library version from filename at crash r=gsvelto

Comment 26

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/4399c5a65886

Comment 27

[Iulian Moraru]

Backed out changeset 4399c5a65886 (bug 1847098) on gerard-majax's request.

• Push that was backed out

• Backout link

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/76ecc1dcf99e
Report library version from filename at crash r=gsvelto

Comment 29

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/76ecc1dcf99e