Write sccache log directly to artifact directory

RESOLVED FIXED in Firefox -esr60

Status

enhancement
RESOLVED FIXED
9 months ago
4 months ago

People

(Reporter: ted, Assigned: ted)

Tracking

Trunk
mozilla64
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 fixed, firefox64 fixed)

Details

Attachments

(2 attachments)

We've seen some sccache failures that are hard to diagnose because currently the sccache log gets copied to the artifact directory only after a successful build. I believe this was mostly so that we could gzip it for uploading, but the logs aren't that big anyway, and some Taskcluster client implementations will store .log artifacts gzip-compressed, so I don't think we should worry about that detail.
Still working out some kinks here, apparently nothing creates the artifact directory before the start of the build, and my attempt to do so on Windows didn't work properly, but one positive result here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=70d56b46ec6f221a9693d7404849b12d81bc5ae3&selectedJob=198690831

This job timed out in mozharness but produced an sccache.log.

Less successful:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=70d56b46ec6f221a9693d7404849b12d81bc5ae3&selectedJob=198690813

This job hit "[taskcluster:error] Task timeout after 3600 seconds. Force killing container.", so it didn't upload any artifacts. :-/
I don't see anything suspicious in that sccache.log, except for the fact that we're not caching any compiles (bug 1484897 comment 11).
Currently the BuildScript mozharness base class has an `ensure_upload_path`
action that we run after the build. This patch changes it to no longer be
an action, and instead simply run as part of `preflight_build` so that the
artifact directory will be created before starting the build. This makes
it more straightforward to write things like log files directly into this
directory without worrying about whether it exists.
This patch makes it so we write the sccache log directly to the artifact
directory, so that it will be uploaded even if the build fails. It also makes
the log slightly more verbose. Both of these should help with diagnosing
sccache failures in CI.

The sccache log will no longer be explicitly gzip compressed, but some
Taskcluster client implementations will store it as gzip compressed.
The last try push shows this mostly working, although  the caveat in comment 3 about taskcluster timeouts still apply. (The try push includes an sccache update that is causing the timeouts.)
Comment on attachment 9010022 [details]
bug 1490325 - make mozharness BuildScript.ensure_upload_path part of preflight_build. r?build

Nick Alexander :nalexander [he/him] has approved the revision.
Attachment #9010022 - Flags: review+
Comment on attachment 9010024 [details]
bug 1490325 - write sccache log directly to artifact directory, make logs slightly more verbose. r?build

Michael Shal [:mshal] has approved the revision.
Attachment #9010024 - Flags: review+

Comment 14

9 months ago
Pushed by tmielczarek@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1ab93555c039
make mozharness BuildScript.ensure_upload_path part of preflight_build. r=firefox-build-system-reviewers,nalexander
https://hg.mozilla.org/integration/autoland/rev/0780fdf31ce1
write sccache log directly to artifact directory, make logs slightly more verbose. r=mshal

Comment 15

9 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/1ab93555c039
https://hg.mozilla.org/mozilla-central/rev/0780fdf31ce1
Status: NEW → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Duplicate of this bug: 1331794
You need to log in before you can comment on or make changes to this bug.