1429871 - Upload binaries for non-Windows builds to the symbol server

Status: ASSIGNED

(Firefox Build System :: General, enhancement, P2)

Description

[(not currently active) Ted Mielczarek]

For several years now we've been uploading binaries to the symbol server for Windows builds (bug 528092). Microsoft's debuggers know how to use symbol servers to locate binaries when debugging a minidump, so this provides a really nice experience where a developer can download a minidump from crash-stats, open it in a debugger, and have everything Just Work without having to manually locate the matching binaries and whatnot.

I'd like to work on making this work for other platforms as well. The first step will be uploading the binaries to the symbol server. I'd like to upload them with a simpler path so that we can just GET `https://symbols.mozilla.org/<build id>/<filename>` for Linux binaries or `https://symbols.mozilla.org/<UUID>/<filename>` for Mac binaries. We might actually just want to use the UUID for mac (see bug 1139356).

Comment 1

[(not currently active) Ted Mielczarek]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0cbc73668dd901cb2f13b0d351564745d81e9873

Comment 2

[Gabriele Svelto [:gsvelto] (PTO)]

Bump this up, this is something we want to do.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Is "buildid" unique enough to prevent binaries from stomping on one another?

Also, what is "UUID"? Is that something that's unique to the build or something generated for every file?

Also, what do we think the order of magnitude size impact for uploads will be? Will this push more symbol uploads to > 1gb? This may require us to fix one or both of bug #1595365 and bug #1652835.

Comment 4

[(not currently active) Ted Mielczarek]

Hi Will! :)

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #3)

Is "buildid" unique enough to prevent binaries from stomping on one another?

Yes. Functionally it's a SHA-1 hash of parts of the binary. I'm not sure there's a great reference to point you at, but this RedHat page describes it succinctly:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/compiling-build-id

In practice you can assume that two binaries with the same build id are equivalent. For a build that is reproducible, you should be able to take the same source + set of build tools and produce bit-identical binaries that share the same build id. Firefox builds are reproducible, so you will see duplicate build ids, but that's a feature.

Also, what is "UUID"? Is that something that's unique to the build or something generated for every file?

Apple's linker inserts a UUID in the file headers of every binary it produces. They will be unique. (I can't remember how this interacts with reproducible builds at the moment.)

Also, what do we think the order of magnitude size impact for uploads will be? Will this push more symbol uploads to > 1gb? This may require us to fix one or both of bug #1595365 and bug #1652835.

Luckily binary files are way smaller than debug symbols! Several orders of magnitude smaller. What this bug is asking for is basically storing the contents of the Firefox installer that ships to end users (not all of it, just the executables and shared libraries) in the symbol store.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Hi Ted! :)

(In reply to (not currently active) Ted Mielczarek from comment #4)

Hi Will! :)

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #3)

Is "buildid" unique enough to prevent binaries from stomping on one another?

Yes. Functionally it's a SHA-1 hash of parts of the binary. I'm not sure there's a great reference to point you at, but this RedHat page describes it succinctly:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/compiling-build-id

Got it! I thought maybe we were talking buildid like the buildid we generate for builds. For example, 20210903235534 isn't a sha and I'm pretty sure it's not unique in the ways we're talking about.

I think this sounds fine. Also, it makes me think we should stand up a Symbolicator service for handling the downloading side of things. That may inform paths for things.

Comment 6

[Markus Stange [:mstange]]

I'd like to propose the following paths:

• For Linux: https://symbols.mozilla.org/buildid/<buildid>/executable, for debuginfod compatibility (bug 1779442). This file is stored uncompressed. Example: https://symbols.mozilla.org/buildid/5f6af34726221c47d0138dc4623a044fccf0733f/executable
• For macOS: https://symbols.mozilla.org/uuid/<UUID>/executable. Example: https://symbols.mozilla.org/uuid/D5419BDA-4A35-327C-A444-B64EBD8AB811/executable . And for the dSYM we could use /uuid/<UUID>.dSYM.tar.bz2 or /uuid/<UUID>.dSYM.yaa.

Do those paths sound good?

Comment 7

[Markus Stange [:mstange]]

(In reply to Markus Stange [:mstange] from comment #6)

• For macOS: https://symbols.mozilla.org/uuid/<UUID>/executable. Example: https://symbols.mozilla.org/uuid/D5419BDA-4A35-327C-A444-B64EBD8AB811/executable . And for the dSYM we could use /uuid/<UUID>.dSYM.tar.bz2 or /uuid/<UUID>.dSYM.yaa.

These paths are currently rejected by Tecken:

{"error": "Unrecognized file pattern. Should only be <module>/<hex>/<file> or <name>-symbols.txt and nothing else. (First unrecognized pattern was uuid/05E23BCB-EFB0-330B-809B-1EEAC8884B86/executable)"}

It doesn't like the dashes in the UUID.

Comment 8

[Markus Stange [:mstange]]

Attachment: Bug 1429871 - Store macOS dSYM bundles at /uuid/<code_id>/debuginfo.dSYM.tar.bz2, for use with DBGShellCommands. r=gsvelto

In the past, they were stored at /<debug_name>/<debug_id>/<debug_name>.dSYM.tar.bz2.

Example:
Before: https://symbols.mozilla.org/libmozglue.dylib/D13BD7EC88D830C49C6BC4B34F2D91560/libmozglue.dylib.dSYM.tar.bz2
After: https://symbols.mozilla.org/uuid/D13BD7EC88D830C49C6BC4B34F2D9156/debuginfo.dSYM.tar.bz2

See https://lldb.llvm.org/use/symbols.html for a description of DBGShellCommands.
A DBGShellCommands script is only supplied with the UUID, not with the filename.

Depends on D154294

Comment 9

[Markus Stange [:mstange]]

Attachment: Bug 1429871 - Upload the native executable for Linux/Android to /buildid/<code_id>/executable. r=gsvelto

Having the executable on the symbol server will allow the following:

• Debugging of minidumps from a different machine using gdb, with gdb
  downloading the executable and debuginfo via the debuginfod protocol
• In the future: crash stack unwinding using native unwind info from the
  binary
• In the future: Exposing assembly code via the symbolication API, for
  use in the Firefox profiler

Depends on D154297

Comment 10

[Markus Stange [:mstange]]

Attachment: Bug 1429871 - Upload the native executable for macOS to /uuid/<code_id>/executable. r=gsvelto

Having the executable on the symbol server will allow the following:

• Debugging of minidumps from a different machine using gdb, with gdb
  downloading the executable and debuginfo via the debuginfod protocol
• In the future: crash stack unwinding using native unwind info from the
  binary
• In the future: Exposing assembly code via the symbolication API, for
  use in the Firefox profiler

Depends on D154298

Comment 11

[Markus Stange [:mstange]]

https://treeherder.mozilla.org/jobs?repo=try&revision=2f794fa6d70369704647e923f5088a86b8085454

Comment 12

[Markus Stange [:mstange]]

This bug is currently blocked on bug 1784290 / PR #2598.