1779442 - Make symbols.mozilla.org usable as a debuginfod server

Status: NEW

(Tecken :: General, task)

Description

[Markus Stange [:mstange]]

It would be great if we could make https://symbols.mozilla.org/ work as a debuginfod server for Firefox binaries.

debuginfod is the Linux (elfutils) equivalent of Microsoft symbol servers: You can set an environment variable to a URL, and then gdb / lldb / perf and other tools can obtain native debug symbol files based on the build ID.

This way, you can debug and profile binaries which you didn't build yourself but for which the debugging artifacts are published on a server.

There are three URL schemes:

<server>/buildid/<buildid>/executable: The executable, i.e. the binary or .so file containing machine code but not necessarily symbols.
<server>/buildid/<buildid>/debuginfo: The .so.dbg file, i.e. an object containing symbol tables and DWARF sections, but not necessarily machine code.
<server>/buildid/<buildid>/source/<path>: The source code of the file whose native path is <path> in the corresponding debug symbols file.

Examples:
https://debuginfod.fedoraproject.org/buildid/7b0cdaf878ab4f99078439d864af70a5fd7b5a2c/executable
https://debuginfod.systemtap.org/buildid/101ecd8ba902186974b9d547f9bfa64b166b3bb9/debuginfo
https://debuginfod.fedoraproject.org/buildid/7b0cdaf878ab4f99078439d864af70a5fd7b5a2c/source/usr/src/debug/glibc-2.33/nptl/pthread_cond_wait.c

The environment variable is DEBUGINFOD_URLS.

Comment 1

[Markus Stange [:mstange]]

• Our debuginfo file is currently stored at https://symbols.mozilla.org/<name>/<debug_id>/<name>.dbg.gz. We would instead need to store it uncompressed (bug 1779439) and at a different path (no name, and using the build ID instead of the breakpad debug ID).
• We currently don't store the executable at all. This is bug 1429871.
• We can punt on the source code aspect of it for now.

Long term, for source code, I think it would be fun to do something like this: 1. Put a srcsrv section into the .so.dbg file that maps absolute paths to URLs, like we do for Windows pdbs. (originally suggested by Ted) We could call the section .debug_EXT_srcsrv. 2. When a path is requested, look up the corresponding URL in the corresponding .so.dbg srcsrv section, and then redirect to that URL.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Markus: Regarding "It would be great if...", it sounds like this support is a nice-to-have and not a change that has a clear value / impact on the organization. Is that right? If not, what's the value / impact?

Comment 3

[Markus Stange [:mstange]]

I agree it's more of a nice-to-have. There is some impact to the organization, but it's not super huge:

• Profiling regular Firefox Nightly builds on Linux with perf would give rich symbol information. The Performance and JS teams are planning to make increased use of perf in the near future. (But we'll probably mostly use perf on local builds of the JS shell.)
• If users report a Firefox hang on Linux, they can attach gdb and give us much more useful information that lets us track down the bug.

As for prioritization, the value in this bug mostly derives from the fact that it's a cheap, easy win. My thinking was the following: We have these native symbol files on the server anyway, so we might as well pick filenames which work well with any existing platform conventions, and debuginfod is that platform convention for Linux. And I'm touching this code in bug 1429871 anyway (that bug has value on its own), so this is a very cheap change to make.