1382849 - Toolchain tasks should use a cache/volume for work

Status: NEW

(Firefox Build System :: General, enhancement)

Description

[Gregory Szorc [:gps]]

Currently, at least the clang toolchain task is building from an aufs filesystem in /home/worker/workspace. This likely slows down operations significantly due to aufs badness. See bug 1291940.

/home/worker/workspace (or wherever intensive work is performed) should be a host-mounted volume/cache backed by ext4 so aufs doesn't interfere.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1382849 - Add workspace cache to toolchain docker-worker jobs.

Review commit: https://reviewboard.mozilla.org/r/159518/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/159518/

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 8888538 [details]
Bug 1382849 - Add workspace cache to toolchain docker-worker jobs.

https://reviewboard.mozilla.org/r/159518/#review164880

Comment 3

[Mike Hommey [:glandium]]

Comment on attachment 8888538 [details]
Bug 1382849 - Add workspace cache to toolchain docker-worker jobs.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/159518/diff/1-2/

Comment 4

[Mike Hommey [:glandium]]

Comment on attachment 8888538 [details]
Bug 1382849 - Add workspace cache to toolchain docker-worker jobs.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/159518/diff/2-3/

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8888538 [details]
Bug 1382849 - Add workspace cache to toolchain docker-worker jobs.

https://reviewboard.mozilla.org/r/159518/#review164890

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0ee08a9803fc
Add workspace cache to toolchain docker-worker jobs. r=dustin,gps

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Backed out for failures like https://treeherder.mozilla.org/logviewer.html#?job_id=116197099&repo=autoland

Comment 8

[Mike Hommey [:glandium]]

So the problem here is that we're just abusing caches to have access to faster volumes. Ideally, we'd need a non-caching cache.  In fact, we'd want the same thing for try, so that try builds can stop doing this:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/job/common.py#21

and get fast I/O.

Dustin, what do you think? How far along is taskcluster-worker, and would it solve those aufs problems without abusing caches? If so, what should we do in the interim?

Comment 9

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b1e444b1b4c7
Backed out changeset 0ee08a9803fc for cctools bustage a=backout

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

tc-worker wouldn't solve aufs, and I'm not sure if it has a "non-caching cache" option.  At any rate, there's no docker engine for tc-worker yet, and writing such a thing and then switching to it is probably not practical until the migration from buildbot is complete.

At the moment, I suspect a simple 'rm -rf' would do the trick here?