Closed Bug 1654994 Opened 4 months ago Closed 4 months ago

Move debug info compression to upload-symbols task

Categories

(Firefox Build System :: Task Configuration, task)

task

Tracking

(firefox81 fixed)

RESOLVED FIXED
81 Branch
Tracking Status
firefox81 --- fixed

People

(Reporter: glandium, Assigned: glandium)

References

(Blocks 1 open bug)

Details

Attachments

(6 files, 2 obsolete files)

Instead of having symbolstore compress the individual debug info files, which takes a long time but is not necessary for tests, move that to the upload-symbols task, since what really wants those files compressed is the symbols server (until the symbols server doesn't need that, per bug 1635150).
But since that would mean large artifacts, still compress, but with something faster: zstd.

Because the symbols server can only handle the .zip format for the
symbols archives, we make upload_symbols itself munge what it takes from
builds if they are .tar.zst, so that the symbols server is happy.

Eventually, the symbols server will be able to deal with those archives
on its own.

It is worth noting that this conversion is actually taking more time
than builds spend for the same archive because nothing is parallelized,
while in builds the compression of e.g. libxul.so and libxul-gtest
happen in parallel.

However, overall, we'll still gain from switching to this new code path
because builds currently always compress the debug info, even when it's
not going to be uploaded to the symbols server.

When adding a kind dependency for e.g. fetches, it's not desirable that
the template is applied for the tasks in that kind. So allow to list
kinds to ignore.

Now that it is not actually used anywhere during the build.

This has the unfortunate side effect of losing the optimization we added where the upload-symbols task didn't have to download the symbols.zip at all, and could simply pass the URL along to Tecken.

Attachment #9166763 - Attachment is obsolete: true
Attachment #9166767 - Attachment is obsolete: true
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/435b90f6ca70
Make full symbols archive opt-in rather than opt-out. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/7131943d8db9
Support creating .tar.zst archives instead of .zip for symbol archives. r=firefox-build-system-reviewers,rstewart,mhentges
https://hg.mozilla.org/integration/autoland/rev/9fca8b0bf8c1
Support using .tar.zst symbol archives in mach artifact. r=nalexander,firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/77f1b667baec
Allow upload_symbols to handle .tar.zst symbols archives. r=firefox-build-system-reviewers,rstewart,mhentges
https://hg.mozilla.org/integration/autoland/rev/45ee50e76d61
Switch full symbols archive to tar.zst. r=firefox-build-system-reviewers,rstewart,tomprince
https://hg.mozilla.org/integration/autoland/rev/cd3e1074b785
Stop setting up makecab for the build. r=firefox-build-system-reviewers,rstewart

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: General → Task Configuration
Status: RESOLVED → REOPENED
Flags: needinfo?(mh+mozilla)
Resolution: FIXED → ---
Target Milestone: 81 Branch → ---
Flags: needinfo?(mh+mozilla)
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/ab117221aeae
Make full symbols archive opt-in rather than opt-out. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/fa6854f9e0c9
Support creating .tar.zst archives instead of .zip for symbol archives. r=firefox-build-system-reviewers,rstewart,mhentges
https://hg.mozilla.org/integration/autoland/rev/32dfcdcbe2ac
Support using .tar.zst symbol archives in mach artifact. r=nalexander,firefox-build-system-reviewers,rstewart
https://hg.mozilla.org/integration/autoland/rev/c31ebb9aae4a
Allow upload_symbols to handle .tar.zst symbols archives. r=firefox-build-system-reviewers,rstewart,mhentges
https://hg.mozilla.org/integration/autoland/rev/ee8fad62b8ab
Switch full symbols archive to tar.zst. r=firefox-build-system-reviewers,rstewart,tomprince
https://hg.mozilla.org/integration/autoland/rev/183b80cb2e0a
Stop setting up makecab for the build. r=firefox-build-system-reviewers,rstewart
Regressions: 1656785

== Change summary for alert #26667 (as of Thu, 06 Aug 2020 08:04:05 GMT) ==

Improvements:

10% build times linux64 debug base-toolchains-clang taskcluster-c5d.4xlarge 1,520.00 -> 1,362.08
9% build times linux64 debug base-toolchains-clang taskcluster-c5d.4xlarge 1,469.77 -> 1,341.09
5% build times osx-shippable opt nightly taskcluster-c5d.4xlarge 2,499.77 -> 2,379.66

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=26667

Regressions: 1668317
You need to log in before you can comment on or make changes to this bug.