1668317 - at some point in the last two months, dlls stopped being compressed

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

At some point, the build system extracts symbols from binaries it's just built and then later uploads them to Tecken. For Windows builds, it runs makecab on files to compress them. This happens before uploading.

At some point in the last 3 months (I can spend more time to figure out exactly when things changed), .dll files stopped being compressed. I did some spot checking and found these two uploads--somewhere between those two, things changed.

This upload from July 8th, 2020 has compressed dlls:

https://symbols.mozilla.org/uploads/upload/145567

It has this record:

v1/xul.dll/5F064109699B000/xul.dl_  41.7 MB

This upload from August 7th, 2020 does not have compressed dlls:

https://symbols.mozilla.org/uploads/upload/165577

It has this record:

v1/xul.dll/5F2D86426EB9000/xul.dll    109.9 MB

The consequence of this is that VS can no longer download dlls. It's trying to download:

https://symbols.mozilla.org/xul.dll/5F517CAB659C000/xul.dll

But "dll" is not in the allowed extensions list, so it gets back a 404 (ignored) message.

This bug covers fixing something so VS works again.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

cc:ing everyone involved.

Braindump:

I think (but I'm not sure because I don't work on any of that stuff) the work done in this bug caused the problem:

https://bugzilla.mozilla.org/show_bug.cgi?id=1654994

That moved where makecab was being called. Maybe we stopped calling it on dlls?

Tecken has a setting for "allowed extensions" which it'll let people download. "dl_" is in the list, but "dll" is not. So the dll files are in the symbols bucket, but because they're not compressed, they have "dll" as an extension and not "dl_" and thus requesting them yields the 404 (ignored) message. That's what Jens and Simon are hitting.

Bug #1635150 covers figuring out whether Tecken can handle uploads of non-breakpad symbols files that are uncompressed. I didn't have time to look into that.

Options I can think of:

1. We change the setting in Tecken to also allow downloading of "dll". I don't know how uncompressed files affects storage costs. I don't know if there are other files that need to be allowed.
2. We fix upload_symbols to call makecab on dll files. https://searchfox.org/mozilla-central/source/toolkit/crashreporter/tools/upload_symbols.py#135
3. Some third option?
4. All of the above?

Comment 2

[Jens Stutte [:jstutte]]

Tentatively setting regressed by 1654994 and ni to glandium:. Thanks!

Comment 3

[Mike Hommey [:glandium]]

Attachment: Bug 1668317 - Also compress .exe and .dll files for the symbol server.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/37a2334aacb4
Also compress .exe and .dll files for the symbol server. r=firefox-build-system-reviewers,mhentges,rstewart DONTBUILD

Comment 5

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/37a2334aacb4

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

If I understand what happened correctly, this will fix the nightly channel, but not the beta channel or release channel. Do both those channels also have the issue? If so, can this get uplifted?

Comment 7

[Mike Hommey [:glandium]]

I can confirm it worked: https://symbols.mozilla.org/xul.dll/5F7B502C6A9B000/xul.dl_

Comment 8

[Mike Hommey [:glandium]]

Comment on attachment 9178913 [details]
Bug 1668317 - Also compress .exe and .dll files for the symbol server.

Beta/Release Uplift Approval Request

• User impact if declined: Visual Studio can't load the symbols for crash reports.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It doesn't affect Firefox directly, and has been verified to work.
• String changes made/needed: n/a

Comment 9

[Julien Cristau [:jcristau]]

Comment on attachment 9178913 [details]
Bug 1668317 - Also compress .exe and .dll files for the symbol server.

approved for 82.0b8

Comment 10

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/a6d07a29d241

Comment 11

[Pascal Chevrel:pascalc]

Comment on attachment 9178913 [details]
Bug 1668317 - Also compress .exe and .dll files for the symbol server.

Approved for 81.0.2

Comment 12

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-release/rev/b967a3966ba1