1806123 - Symbols fail to upload when `symbols.zip` exceeds 2GB

Status: VERIFIED FIXED

(Firefox Build System :: General, defect)

Description

[Ben Dean-Kawamura [:bdk]]

While doing a try build, the build-macosx64-fuzzing/debug-upload-symbols is failing with OverflowError: string longer than 2147483647 bytes when uploading symbols.zip.

It looks like this comes from an issue from requests library when uploading multi-part data over 2GB (the comments towards the bottom are most relevant to this case).

Comment 1

[Ben Dean-Kawamura [:bdk]]

The requests library issue can be fixed by using the streaming multi-part encoder from requests_toolbelt. However, we then run into a server error: 413: Request Entity Too Large.

I think there are a few potential fixes, but it's beyond what I can do at this point:

• Increase the server limit
• Use bzip2 to compress the .sym files rather than rely on the zlib compression from the zip file. I think this would bring the file size under the limit, but maybe requires updating the ingestion code.
• Uploading the source .tar.zst file would also work, since that file is under 2GB. This would also save a lot of CPU cycles.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

All of the options in comment #1 need work to be done on the Mozilla Symbols Server (aka Tecken). I was hoping to get to reworking the upload view in October 2022, but never got a chance. I don't know when I'm going to get to that. However, supporting other compression formats for uploading symbols is one of the things I wanted to look at.

As a side note, Mozilla Symbols Server doesn't know anything about builds, so there's no reason we can't split files across multiple symbols.zip files. If we did that, the files would be smaller.

Comment 3

[Mike Hommey [:glandium]]

It'd also be great is we didn't have to use a zip for these transactions because zip compression is slow (incidentally, the original .tar.zst is smaller than 2GB)

Comment 4

[Ben Dean-Kawamura [:bdk]]

Attachment: Bug 1806123 - Refactoring upload_symbols.py

The functionality is the same, but the code is now split into several
functions. This is preparation for more changes to fix bug 1806123.

Comment 5

[Ben Dean-Kawamura [:bdk]]

Attachment: Bug 1806123 - Split the symbols.zip upload into chunks.

This avoids hitting the 2GB limit that both the requests library and the
server have.

Depends on D165035

Comment 6

[Ben Dean-Kawamura [:bdk]]

(In reply to Mike Hommey [:glandium] from comment #3)

It'd also be great is we didn't have to use a zip for these transactions because zip compression is slow (incidentally, the original .tar.zst is smaller than 2GB)

I think this would be great, but I wasn't sure how to implement it so I just went with splitting up the zip file in upload_symbols..py.

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1806123 - Move zip files cleanup outwards.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5e99a63a268e
Refactoring upload_symbols.py r=glandium
https://hg.mozilla.org/integration/autoland/rev/d7d5bc97d505
Move zip files cleanup outwards. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/15e7bee14512
Split the symbols.zip upload into chunks. r=bdk,firefox-build-system-reviewers,ahochheiden

Comment 10

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/5e99a63a268e
https://hg.mozilla.org/mozilla-central/rev/d7d5bc97d505
https://hg.mozilla.org/mozilla-central/rev/15e7bee14512

Comment 11

[Iulian Moraru]

This seems to work in today's early beta simulation.
Just a note here: the job didn't run by default(the last time Symf ran by default on a beta simulation was in 26th Dec), to see if this works I added the job manually on today's beta simulation. If there is a difference when running by default vs when this is manually added, please revert the flags here and NI sheriffs so we would know to keep an eye on a default run.
Thank you!