1289643 - Use cache for workspace checkout on Try

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Gregory Szorc [:gps]]

lint tasks use a workspace cache to hold the source checkout. On Try (level 1 scopes), this cache is removed and we use local Docker container storage for the checkout. This is both slow (because AUFS is slower than host-native ext3/4 volume mount) and buggy (I see intermittent failures in automation when hg does checkouts).

We should use a cache on Try / level 1 scopes so these issues aren't present.

My original patches in bug 1247168 had this implemented. Then I removed it to make the review simpler and less contentious. Unfortunately, we are seeing intermittent failures due to AUFS, so this needs to be addressed.

Comment 1

[Gregory Szorc [:gps]]

https://public-artifacts.taskcluster.net/Wd041B0aQjOoHqgPer_nng/0/public/logs/live_backing.log is an example log showing checkout failure due to aufs bugs. I haven't seen one of these failures when using a host-mounted cache/volume.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Maybe this just means adding a CLOBBER_OBJDIR env var that build.sh treats as an indication it should `rm -rf` the objdir?  Then we can set that variable when config.params['project'] == 'try'

Comment 3

[Gregory Szorc [:gps]]

Or we can trust Mercurial will reset the checkout to a clean slate and not clobber on Try. If nothing else, this will make Try behave like all the other repos :)

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Maybe I'm confused, but I thought clobbering meant deleting the objdir and any stray files in the repo, while I assume mercurial is only cleaning up the repo.  I think we would want the latter on all trees, but i assume on non-try trees that we want to take advantage of the objdir, right?

Comment 5

[Gregory Szorc [:gps]]

It depends where the "objdir" is. If the objdir is under the source checkout, `hg robustcheckout --purge` will nuke it because it purges *all* untracked and ignored falls.

The lint tasks/images don't really have "objdirs." They may leave state under the source checkout. But that state will get nuked by `hg robustcheckout --purge`.

The only thing we should potentially save the objdir for are build tasks. And the "objdir" in this case should be running out of a separate directory or cache from the source checkout. This bug doesn't deal with build tasks - just the ones based on the lint image. So let's not worry about "objdirs" here.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Got it, sorry for my confusion

Comment 7

[Gregory Szorc [:gps]]

Attachment: Bug 1289643 - Add $ to regular expression in hg-shared cache name;

All the other expressions are anchored via ^ and $.

Review commit: https://reviewboard.mozilla.org/r/67996/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/67996/

Comment 8

[Gregory Szorc [:gps]]

Attachment: Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

Currently, TaskCluster tasks tend to use the "workspace" directory as
a cache that manages the source checkout *and* additional state.

Historically at Mozilla, we've lumped "source checkout" and "workspace"
(sometimes known as an "objdir") into the same directory. This is
not ideal. Ideally, there is an immutable, read-only source checkout
and all files produced from that source live in a separate directory.

In this commit, the "workspace" directory for the "lint" image has been
renamed to "checkouts" and all tasks using the image have been updated
accordingly. By having "checkout" in the name, we clearly identify this
cache as being relevant to source checkouts, which IMO can serve a
different role from "workspaces." This distinction is important, as the
next commit will prevent the "checkouts" cache from getting optimized
out in certain tasks.

To hammer this point home, documentation on common caches has been
introduced.

Review commit: https://reviewboard.mozilla.org/r/67998/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/67998/

Comment 9

[Gregory Szorc [:gps]]

Attachment: Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

Previously, the VCS checkout cache would be optimized away on Try. This
meant every task on Try had to create its own checkout from scratch. This
meant writing ~150,000 files. By allowing use of the cache, we minimize
the amount of work tasks need to perform on Try and therefore make tasks
complete faster, on average.

Review commit: https://reviewboard.mozilla.org/r/68000/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/68000/

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/67998/diff/1-2/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/68000/diff/1-2/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/67998/diff/2-3/

Comment 13

[Gregory Szorc [:gps]]

Comment on attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/68000/diff/2-3/

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8776070 [details]
Bug 1289643 - Add $ to regular expression in hg-shared cache name;

https://reviewboard.mozilla.org/r/67996/#review65374

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

https://reviewboard.mozilla.org/r/67998/#review65376

::: taskcluster/docs/index.rst:32
(Diff revision 3)
>      kinds
>      transforms
>      yaml-templates
>      how-tos
>      docker-images
> +    caches

I feel like the "how tos" should be the last element, to be honest.  But if you have a specific reason to put `docker-images` and `caches` in this order that's fine.

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

https://reviewboard.mozilla.org/r/68000/#review65380

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0beb6afa58b7
Add $ to regular expression in hg-shared cache name; r=dustin
https://hg.mozilla.org/integration/autoland/rev/50b87c38d18d
Change path for checkouts from "workspace" to "checkouts"; r=dustin
https://hg.mozilla.org/integration/autoland/rev/6ea38e685679
Allow level-{{level}}-checkouts cache to be used on Try; r=dustin

Comment 18

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/0beb6afa58b7
https://hg.mozilla.org/mozilla-central/rev/50b87c38d18d
https://hg.mozilla.org/mozilla-central/rev/6ea38e685679