1779439 - Store native debug symbols for Linux as .so.dbg and not as .so.dbg.gz

Status: ASSIGNED

(Tecken :: General, task)

Description

[Markus Stange [:mstange]]

For native debug symbols on Linux, we currently compress them to .dbg.so.gz before stuffing them into a zip to upload them to the symbol server.

Then they get served at URLs like the following:
https://symbols.mozilla.org/libxul.so/47F36A5F2226471CD0138DC4623A044F0/libxul.so.dbg.gz

Could we serve them without the .gz extension? We can still use Content-Encoding: gzip for them, if the requesting client indicates support for it with the Accept-Encoding header.
Then they "look" uncompressed but still get transmitted in compressed form.

I think this is what we do for the .sym files - we upload them as .sym (in the zip), and they are served with a .sym extension, but they use Content-Encoding: gzip.

Then, as the next step, we can move them to URLs which are compatible with the debuginfod protocol:

https://symbols.mozilla.org/buildid/5f6af34726221c47d0138dc4623a044fccf0733f/debuginfo

The debuginfod protocol requires the downloaded files to be uncompressed .so.dbg files.

Comment 1

[Markus Stange [:mstange]]

Attachment: Bug 1779439 - Use exist_ok=True instead of catching exceptions. r=gsvelto

exist_ok was added in Python 3.2. It is already used by uploadsymbols.py.

Comment 2

[Markus Stange [:mstange]]

Attachment: Bug 1779439 - Obtain the ELF build ID on Linux, from the INFO CODE_ID row. r=gsvelto

Depends on D154292

Comment 3

[Markus Stange [:mstange]]

Attachment: Bug 1779439 - Store Linux/Android native debuginfo uncompressed at /buildid/<code_id>/debuginfo, for debuginfod compatibility. r=gsvelto

In the past, it was stored at /<debug_name>/<debug_id>/<debug_name>.so.gz.

Example:
Before: https://symbols.mozilla.org/libxul.so/47F36A5F2226471CD0138DC4623A044F0/libxul.so.dbg.gz
After: https://symbols.mozilla.org/buildid/5f6af34726221c47d0138dc4623a044fccf0733f/debuginfo

Depends on D154293

Comment 4

[Markus Stange [:mstange]]

Hmm, so the upload-symbols job completed successfully in the try push, but I can't find the files on the server.

Does Tecken have another file name allow list that I'm running into?

For example, this file exists: https://symbols.mozilla.org/try/ssltunnel/B6D4DCC11470DC818FF28B66C95000600/ssltunnel.sym
But this file does not: https://symbols.mozilla.org/try/buildid/c1dcd4b6701481dc8ff28b66c9500060adddf31d/debuginfo
Both were present in the same .zip file.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

The second item doesn't match the zip file structure, so my guess is that Tecken is dropping those files. I'd need to check the code to be sure. I won't get to that for a bit. Keeping the needinfo until I do.

Comment 6

[Markus Stange [:mstange]]

Ah, I think I've found the explanation:

DOWNLOAD_FILE_EXTENSIONS_ALLOWED = _config(
    "DOWNLOAD_FILE_EXTENSIONS_ALLOWED",
    default=".sym,.dl_,.ex_,.pd_,.dbg.gz,.tar.bz2",
    parser=ListOf(str),
    doc=(
        "A list of file extensions that if a file is NOT one of these extensions "
        "we can immediately return 404 and not bother to process for anything "
        "else.\n\n"
        "It's case sensitive and has to be lower case.  As a way to get marginal "
        "optimization of this, make sure '.sym' is first in the list since it's "
        "the most common."
    ),
)

The files are probably on the server, but the URLs don't serve the redirect because they take the early-out-404 path.

Comment 7

[Markus Stange [:mstange]]

I've filed bug 1784290 to take care of that.