1618923 - windows spidermonkey jobs take 15 minutes to upload llvm-symbolizer.sh for every SM(*) job

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we run many spidermonkey jobs, on windows we run:

• 6/push on mozilla-central
• 4/push on autoland when scheduled (about 50% of the pushes)

in total we are looking at about 225 jobs/day.

these are all run on builders which are expensive costs.

This bug is about the ~15 minutes to upload llvm-symbolizer.sh for each of these jobs.

I see in bug 1351559 this was setup.

I want to know:

• what we use llvm-sanitizer.gz for
• as this is periodic, I assume this doesn't need to be for every push
• can we reduce this to be a single build type and once/day or at worse case every m-c build?
• do we need this on all platforms (currently on linux and windows)

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:decoder, would you be able to help me answer these questions?

Comment 2

[Geoff Brown [:gbrown]]

Also interesting: most logs show something like

[task 2020-02-27T11:12:57.182Z] + gzip -c Z:/task_1582797506/fetches/clang/bin/llvm-symbolizer
[task 2020-02-27T11:12:57.211Z] gzip: Z:/task_1582797506/fetches/clang/bin/llvm-symbolizer: No such file or directory

and the artifact links for llvm-symbolizer.gz appear to be useless: NoSuchKey error.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

ack, that would be evidence that these artifacts are not used at all

Comment 4

[Steve Fink [:sfink] [:s:]]

I'm not following what's happening here. Portion of the log:

[task 2020-03-04T18:03:49.381Z] + gzip -c Z:/task_1583340662/fetches/clang/bin/llvm-symbolizer
[task 2020-03-04T18:03:49.395Z] gzip: Z:/task_1583340662/fetches/clang/bin/llvm-symbolizer: No such file or directory
[taskcluster 2020-03-04T18:03:56.243Z] Uploading artifact public/build/llvm-symbolizer.gz from file public\build\llvm-symbolizer.gz with content encoding "identity", mime type "application/x-gzip" and expiry 2021-03-04T16:51:47.303Z
[taskcluster 2020-03-04T18:18:11.976Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/Rr7d-QfVQ2O7nMNk6VZGfw/0/on/aws/i-02482d5773311fa11/until/1583347092.517

Then when attempting to download the artifact, I get the NoSuchKey error.

The first obvious problem is that on Windows this would be llvm-symbolizer.exe, which is why the gzip line is failing. But then what does it spend 15 minutes uploading? Does something else produce it? I would expect an error message if it simply didn't exist. But maybe not, and maybe the 15 minutes is actually being spent doing something completely unrelated that doesn't produce a log message?

Anyway, it's clear that this isn't used on Windows. We also have much better facilities for producing toolchain artifacts like this now, so even if the linux version is being used, this particular upload should just be killed. :decoder will have to answer whether it's needed now or not. It was always meant to be a cheap hack to make something available in an easy way to avoid retrieving it in a more proper way. (:gkw requested it originally.) I hadn't imagined it would take any significant amount of time.

On a sample linux64 job https://firefoxci.taskcluster-artifacts.net/KUc-s6V3Q6iaHfkpb7WNRg/0/public/logs/live_backing.log it looks like it takes 0.1sec to compress with gzip, and there's no log message for the upload but the task finishes in less than 1 second after the gzip compression. I'm not sure how to reconcile the Windows and Linux behavior here.

Comment 5

[Steve Fink [:sfink] [:s:]]

Attachment: Bug 1618923 - Fix llvm-symbolizer filename on Windows

Comment 6

[Steve Fink [:sfink] [:s:]]

I'll do this by requesting review from decoder for the trivial change, so if he says it's not needed he can just r+.

Comment 7

[Christian Holler (:decoder)]

LLVM Symbolizer only needs to be uploaded for ASan and TSan builds, not for regular builds (for now it is also okay to only do this on Linux, I am not even sure if we have Windows ASan SM builds).

In general, it is important to have the right version of llvm-symbolizer deployed with an ASan or TSan binary to ensure that it's trace symbolizing works. This is also required by our test CI.

Comment 8

[Christian Holler (:decoder)]

(In reply to Christian Holler (:decoder) from comment #7)

On a sample linux64 job https://firefoxci.taskcluster-artifacts.net/KUc-s6V3Q6iaHfkpb7WNRg/0/public/logs/live_backing.log it looks like it takes 0.1sec to compress with gzip, and there's no log message for the upload but the task finishes in less than 1 second after the gzip compression. I'm not sure how to reconcile the Windows and Linux behavior here.

It sounds to me like the measurements might be broken somehow and this doesn't actually take up time at all. In fact I would be surprised if any of this would take 15 minutes.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

it sounds like we need to remove the code for gzip and making an artifact of llvm-symbolizer on windows.

In addition we need to fix the upload, there is no reason we should spend 15 minutes on an invalid/non-existent artifact uploading. That hints at a timeout or some error condition. Here is the code that does the uploading:
https://github.com/taskcluster/taskcluster/blob/42abb6881e6326c6e1532b240abff3ed7213013a/workers/generic-worker/artifacts.go#L175

Comment 10

[Pete Moore [:pmoore][:pete]]

I suspect the archive is created as a zero byte file since the file it is meant to archive does not exist, and that ec2 doesn't allow us to upload 0 byte files, and that we retry when it fails to upload.

The quick workaround is probably to delete the zero byte file if the archive operation fails, since then we'll immediately upload an error artifact, rather than trying to upload a zero byte file. In bug 1620191 we can adapt generic-worker to refuse to publish 0 byte artifacts when using s3 as a backing store (which currently is the only backing store it supports). Note other backing stores in future may allow a zero byte file.

Comment 11

[Steve Fink [:sfink] [:s:]]

If I fix it to work on Windows as well, it eliminates the 15 minute delay (and the total time to compress and upload is still under 1 second.) Given that having the right version is useful on linux, and conceivably might become useful on Windows, it seems worth landing a fix to make it stop looking for the wrong filename. Or at least, it seems to me like having it be platform-independent seems worth the extra second.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

1 second isn't the end of the world- I always default to not generating extra data if we don't need it. I suspect the storage cost is cheap, so either way is probably fine here.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/14a6cc295897
Fix llvm-symbolizer filename on Windows r=decoder

Comment 14

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/14a6cc295897