Last Comment Bug 1289643 - Use cache for workspace checkout on Try
: Use cache for workspace checkout on Try
Status: RESOLVED FIXED
:
Product: Taskcluster
Classification: Other
Component: Task Configuration (show other bugs)
: unspecified
: Unspecified Unspecified
-- normal
: ---
Assigned To: Gregory Szorc [:gps] (away until 2017-03-20)
:
:
Mentors:
Depends on: 1247168
Blocks: 1286336 1290531
  Show dependency treegraph
 
Reported: 2016-07-26 18:34 PDT by Gregory Szorc [:gps] (away until 2017-03-20)
Modified: 2016-08-02 08:12 PDT (History)
2 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---

MozReview Requests
Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:
Show discarded requests

Attachments
Bug 1289643 - Add $ to regular expression in hg-shared cache name; (58 bytes, text/x-review-board-request)
2016-07-29 11:12 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; (58 bytes, text/x-review-board-request)
2016-07-29 11:12 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try; (58 bytes, text/x-review-board-request)
2016-07-29 11:12 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review

Description User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-26 18:34:47 PDT
lint tasks use a workspace cache to hold the source checkout. On Try (level 1 scopes), this cache is removed and we use local Docker container storage for the checkout. This is both slow (because AUFS is slower than host-native ext3/4 volume mount) and buggy (I see intermittent failures in automation when hg does checkouts).

We should use a cache on Try / level 1 scopes so these issues aren't present.

My original patches in bug 1247168 had this implemented. Then I removed it to make the review simpler and less contentious. Unfortunately, we are seeing intermittent failures due to AUFS, so this needs to be addressed.
Comment 1 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-26 18:36:38 PDT
https://public-artifacts.taskcluster.net/Wd041B0aQjOoHqgPer_nng/0/public/logs/live_backing.log is an example log showing checkout failure due to aufs bugs. I haven't seen one of these failures when using a host-mounted cache/volume.
Comment 2 User image Dustin J. Mitchell [:dustin] 2016-07-27 08:01:50 PDT
Maybe this just means adding a CLOBBER_OBJDIR env var that build.sh treats as an indication it should `rm -rf` the objdir?  Then we can set that variable when config.params['project'] == 'try'
Comment 3 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-28 14:31:31 PDT
Or we can trust Mercurial will reset the checkout to a clean slate and not clobber on Try. If nothing else, this will make Try behave like all the other repos :)
Comment 4 User image Dustin J. Mitchell [:dustin] 2016-07-28 14:40:26 PDT
Maybe I'm confused, but I thought clobbering meant deleting the objdir and any stray files in the repo, while I assume mercurial is only cleaning up the repo.  I think we would want the latter on all trees, but i assume on non-try trees that we want to take advantage of the objdir, right?
Comment 5 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-29 09:49:52 PDT
It depends where the "objdir" is. If the objdir is under the source checkout, `hg robustcheckout --purge` will nuke it because it purges *all* untracked and ignored falls.

The lint tasks/images don't really have "objdirs." They may leave state under the source checkout. But that state will get nuked by `hg robustcheckout --purge`.

The only thing we should potentially save the objdir for are build tasks. And the "objdir" in this case should be running out of a separate directory or cache from the source checkout. This bug doesn't deal with build tasks - just the ones based on the lint image. So let's not worry about "objdirs" here.
Comment 6 User image Dustin J. Mitchell [:dustin] 2016-07-29 09:56:28 PDT
Got it, sorry for my confusion
Comment 7 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-29 11:12:39 PDT
Created attachment 8776070 [details]
Bug 1289643 - Add $ to regular expression in hg-shared cache name;

All the other expressions are anchored via ^ and $.

Review commit: https://reviewboard.mozilla.org/r/67996/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/67996/
Comment 8 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-29 11:12:39 PDT
Created attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

Currently, TaskCluster tasks tend to use the "workspace" directory as
a cache that manages the source checkout *and* additional state.

Historically at Mozilla, we've lumped "source checkout" and "workspace"
(sometimes known as an "objdir") into the same directory. This is
not ideal. Ideally, there is an immutable, read-only source checkout
and all files produced from that source live in a separate directory.

In this commit, the "workspace" directory for the "lint" image has been
renamed to "checkouts" and all tasks using the image have been updated
accordingly. By having "checkout" in the name, we clearly identify this
cache as being relevant to source checkouts, which IMO can serve a
different role from "workspaces." This distinction is important, as the
next commit will prevent the "checkouts" cache from getting optimized
out in certain tasks.

To hammer this point home, documentation on common caches has been
introduced.

Review commit: https://reviewboard.mozilla.org/r/67998/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/67998/
Comment 9 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-29 11:12:39 PDT
Created attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

Previously, the VCS checkout cache would be optimized away on Try. This
meant every task on Try had to create its own checkout from scratch. This
meant writing ~150,000 files. By allowing use of the cache, we minimize
the amount of work tasks need to perform on Try and therefore make tasks
complete faster, on average.

Review commit: https://reviewboard.mozilla.org/r/68000/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/68000/
Comment 10 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-29 11:19:55 PDT
Comment on attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/67998/diff/1-2/
Comment 11 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-29 11:19:55 PDT
Comment on attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/68000/diff/1-2/
Comment 12 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-29 11:31:01 PDT
Comment on attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/67998/diff/2-3/
Comment 13 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-29 11:31:01 PDT
Comment on attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/68000/diff/2-3/
Comment 14 User image Dustin J. Mitchell [:dustin] 2016-08-01 09:38:04 PDT
Comment on attachment 8776070 [details]
Bug 1289643 - Add $ to regular expression in hg-shared cache name;

https://reviewboard.mozilla.org/r/67996/#review65374
Comment 15 User image Dustin J. Mitchell [:dustin] 2016-08-01 09:41:08 PDT
Comment on attachment 8776071 [details]
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts";

https://reviewboard.mozilla.org/r/67998/#review65376

::: taskcluster/docs/index.rst:32
(Diff revision 3)
>      kinds
>      transforms
>      yaml-templates
>      how-tos
>      docker-images
> +    caches

I feel like the "how tos" should be the last element, to be honest.  But if you have a specific reason to put `docker-images` and `caches` in this order that's fine.
Comment 16 User image Dustin J. Mitchell [:dustin] 2016-08-01 09:43:03 PDT
Comment on attachment 8776072 [details]
Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try;

https://reviewboard.mozilla.org/r/68000/#review65380
Comment 17 User image Pulsebot 2016-08-01 09:55:58 PDT
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0beb6afa58b7
Add $ to regular expression in hg-shared cache name; r=dustin
https://hg.mozilla.org/integration/autoland/rev/50b87c38d18d
Change path for checkouts from "workspace" to "checkouts"; r=dustin
https://hg.mozilla.org/integration/autoland/rev/6ea38e685679
Allow level-{{level}}-checkouts cache to be used on Try; r=dustin

Note You need to log in before you can comment on or make changes to this bug.