Use cache for workspace checkout on Try

RESOLVED FIXED

Status

Taskcluster
Task Configuration
RESOLVED FIXED
9 months ago
9 months ago

People

(Reporter: gps, Assigned: gps)

Tracking

(Blocks: 1 bug)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(3 attachments)

(Assignee)

Description

9 months ago
lint tasks use a workspace cache to hold the source checkout. On Try (level 1 scopes), this cache is removed and we use local Docker container storage for the checkout. This is both slow (because AUFS is slower than host-native ext3/4 volume mount) and buggy (I see intermittent failures in automation when hg does checkouts).

We should use a cache on Try / level 1 scopes so these issues aren't present.

My original patches in bug 1247168 had this implemented. Then I removed it to make the review simpler and less contentious. Unfortunately, we are seeing intermittent failures due to AUFS, so this needs to be addressed.
(Assignee)

Comment 1

9 months ago
https://public-artifacts.taskcluster.net/Wd041B0aQjOoHqgPer_nng/0/public/logs/live_backing.log is an example log showing checkout failure due to aufs bugs. I haven't seen one of these failures when using a host-mounted cache/volume.
Maybe this just means adding a CLOBBER_OBJDIR env var that build.sh treats as an indication it should `rm -rf` the objdir?  Then we can set that variable when config.params['project'] == 'try'
(Assignee)

Comment 3

9 months ago
Or we can trust Mercurial will reset the checkout to a clean slate and not clobber on Try. If nothing else, this will make Try behave like all the other repos :)
Maybe I'm confused, but I thought clobbering meant deleting the objdir and any stray files in the repo, while I assume mercurial is only cleaning up the repo.  I think we would want the latter on all trees, but i assume on non-try trees that we want to take advantage of the objdir, right?
(Assignee)

Comment 5

9 months ago
It depends where the "objdir" is. If the objdir is under the source checkout, `hg robustcheckout --purge` will nuke it because it purges *all* untracked and ignored falls.

The lint tasks/images don't really have "objdirs." They may leave state under the source checkout. But that state will get nuked by `hg robustcheckout --purge`.

The only thing we should potentially save the objdir for are build tasks. And the "objdir" in this case should be running out of a separate directory or cache from the source checkout. This bug doesn't deal with build tasks - just the ones based on the lint image. So let's not worry about "objdirs" here.
Got it, sorry for my confusion
(Assignee)

Comment 7

9 months ago
Created attachment 8776070 [details]
Bug 1289643 - Add $ to regular expression in hg-shared cache name;

All the other expressions are anchored via ^ and $.

Review commit: https://reviewboard.mozilla.org/r/67996/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/67996/
Attachment #8776070 - Flags: review?(dustin)
Attachment #8776071 - Flags: review?(dustin)
Attachment #8776072 - Flags: review?(dustin)
(Assignee)

Comment 8

9 months ago
Created attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

Currently, TaskCluster tasks tend to use the "workspace" directory as
a cache that manages the source checkout *and* additional state.

Historically at Mozilla, we've lumped "source checkout" and "workspace"
(sometimes known as an "objdir") into the same directory. This is
not ideal. Ideally, there is an immutable, read-only source checkout
and all files produced from that source live in a separate directory.

In this commit, the "workspace" directory for the "lint" image has been
renamed to "checkouts" and all tasks using the image have been updated
accordingly. By having "checkout" in the name, we clearly identify this
cache as being relevant to source checkouts, which IMO can serve a
different role from "workspaces." This distinction is important, as the
next commit will prevent the "checkouts" cache from getting optimized
out in certain tasks.

To hammer this point home, documentation on common caches has been
introduced.

Review commit: https://reviewboard.mozilla.org/r/67998/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/67998/
(Assignee)

Comment 9

9 months ago
Created attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

Previously, the VCS checkout cache would be optimized away on Try. This
meant every task on Try had to create its own checkout from scratch. This
meant writing ~150,000 files. By allowing use of the cache, we minimize
the amount of work tasks need to perform on Try and therefore make tasks
complete faster, on average.

Review commit: https://reviewboard.mozilla.org/r/68000/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/68000/
(Assignee)

Comment 10

9 months ago
Comment on attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/67998/diff/1-2/
(Assignee)

Comment 11

9 months ago
Comment on attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/68000/diff/1-2/
(Assignee)

Comment 12

9 months ago
Comment on attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/67998/diff/2-3/
(Assignee)

Comment 13

9 months ago
Comment on attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/68000/diff/2-3/
(Assignee)

Updated

9 months ago
Blocks: 1290531
Attachment #8776070 - Flags: review?(dustin) → review+
Comment on attachment 8776070 [details]
Bug 1289643 - Add $ to regular expression in hg-shared cache name;

https://reviewboard.mozilla.org/r/67996/#review65374
Comment on attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

https://reviewboard.mozilla.org/r/67998/#review65376

::: taskcluster/docs/index.rst:32
(Diff revision 3)
>      kinds
>      transforms
>      yaml-templates
>      how-tos
>      docker-images
> +    caches

I feel like the "how tos" should be the last element, to be honest.  But if you have a specific reason to put `docker-images` and `caches` in this order that's fine.
Attachment #8776071 - Flags: review?(dustin) → review+
Comment on attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

https://reviewboard.mozilla.org/r/68000/#review65380
Attachment #8776072 - Flags: review?(dustin) → review+

Comment 17

9 months ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0beb6afa58b7
Add $ to regular expression in hg-shared cache name; r=dustin
https://hg.mozilla.org/integration/autoland/rev/50b87c38d18d
Change path for checkouts from "workspace" to "checkouts"; r=dustin
https://hg.mozilla.org/integration/autoland/rev/6ea38e685679
Allow level-{{level}}-checkouts cache to be used on Try; r=dustin

Comment 18

9 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/0beb6afa58b7
https://hg.mozilla.org/mozilla-central/rev/50b87c38d18d
https://hg.mozilla.org/mozilla-central/rev/6ea38e685679
Status: ASSIGNED → RESOLVED
Last Resolved: 9 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.