Last Comment Bug 1247168 - Use pooled/shared storage and robustcheckout extension
: Use pooled/shared storage and robustcheckout extension
Status: RESOLVED FIXED
:
Product: Taskcluster
Classification: Other
Component: General (show other bugs)
: unspecified
: Unspecified Unspecified
P1 normal
: ---
Assigned To: Gregory Szorc [:gps] (away until 2017-03-20)
:
:
Mentors:
: 1288377 (view as bug list)
Depends on: 1288215 1288377 1288567
Blocks: 1255017 1259456 1286336 1289249 1289643 1298947
  Show dependency treegraph
 
Reported: 2016-02-09 21:31 PST by Mike Hommey [:glandium]
Modified: 2016-08-29 13:49 PDT (History)
5 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---

MozReview Requests
Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:
Show discarded requests

Attachments
Bug 1247168 - Reformat requirements.txt; (58 bytes, text/x-review-board-request)
2016-07-19 15:25 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Install Mercurial 3.8.4 in lint image; (58 bytes, text/x-review-board-request)
2016-07-19 15:25 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Don't remove hg-shared scopes; (58 bytes, text/x-review-board-request)
2016-07-19 15:25 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Make mozilla-unified the base repository for Firefox; (58 bytes, text/x-review-board-request)
2016-07-19 15:25 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Convert lint image and tasks to use robustcheckout; (58 bytes, text/x-review-board-request)
2016-07-19 15:25 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Update Mercurial version and add robustcheckout; (58 bytes, text/x-review-board-request)
2016-07-19 15:25 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review-
Details | Review
Bug 1247168 - Use robustcheckout in desktop-build task; (58 bytes, text/x-review-board-request)
2016-07-19 15:25 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review-
Details | Review
Bug 1247168 - Upgrade to pip 8.1.2 and setuptools 24.0.3 in decision image; (58 bytes, text/x-review-board-request)
2016-07-19 16:01 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Use robustcheckout in decision image; (58 bytes, text/x-review-board-request)
2016-07-19 16:01 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Change decision image to Ubuntu 16.04; (58 bytes, text/x-review-board-request)
2016-07-20 10:43 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Don't install virtualenv in the decision image; (58 bytes, text/x-review-board-request)
2016-07-20 10:43 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Install Mercurial 3.8.4 from .deb files; (58 bytes, text/x-review-board-request)
2016-07-20 10:43 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Stop installing most system packages; (58 bytes, text/x-review-board-request)
2016-07-20 10:43 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Vendor tooltool.py; (58 bytes, text/x-review-board-request)
2016-07-20 10:43 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Configure Mercurial and install global hgrc; (58 bytes, text/x-review-board-request)
2016-07-20 10:43 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Tag and use version 0.1.2 of the decision image; (58 bytes, text/x-review-board-request)
2016-07-20 10:43 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Actually use workspace cache in decision task; (58 bytes, text/x-review-board-request)
2016-07-20 16:46 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Use a cache for repo checkout in lint tasks; (58 bytes, text/x-review-board-request)
2016-07-20 16:46 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Bump centos6-build and centos6-build-upd image versions; (58 bytes, text/x-review-board-request)
2016-07-20 16:46 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Create worker user/group in decision image; (58 bytes, text/x-review-board-request)
2016-07-21 13:38 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Introduce scripts for running action and decision tasks; (58 bytes, text/x-review-board-request)
2016-07-21 13:38 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Add a script to perform a checkout then run a command; (58 bytes, text/x-review-board-request)
2016-07-21 13:38 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review
Bug 1247168 - Add special Dockerfile syntax to add arbitrary files to context; (58 bytes, text/x-review-board-request)
2016-07-21 16:32 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
no flags Details | Review
Bug 1247168 - Use vendored tooltool in lint image; (58 bytes, text/x-review-board-request)
2016-07-22 13:59 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
dustin: review+
Details | Review

Description User image Mike Hommey [:glandium] 2016-02-09 21:31:59 PST
tc-vcs is downloading tarballs of repositories as a first step. It should stop doing that for mercurial, because mercurial will do it much more efficiently with bundles (requires version 3.6, iirc).
Comment 1 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-02-10 16:52:12 PST
tc-vcs is currently snapshotting the .hg and working copy directories into tar.gz files and shipping those around. .hg won't compress very well because most content therein is already zlib compressed. So there's overhead doing the decompress of those files.

The working directory will compress well. However, I'm not convinced that `tar` is faster than hg when a streaming clone is involved.

A streaming hg clone is effectively a tar file (no compression). When using S3, it will complete as fast as the network and I/O can send/write bytes. I've seen EC2 machines apply streaming clones at 90 MB/s. I've seen Try builds apply stream clone bundles at 25-35 MB/s. But that's with less beefy and shared EC2 instances. This appears to be on par or faster than the tarball downloads from https://queue.taskcluster.net/.

For the working copy, `hg up` will use multiple processes for writing files. This theoretically has a leg up on `tar` which uses a single I/O thread. However, my experience is your filesystem will be the limiting factor here, so `hg up` and `tar` tie in terms of performance.

On Windows, streaming clones with hg 3.7+ should *always* be faster than tar because Mercurial will use multiple I/O threads for writing files. It turns out writing thousands of files on NTFS is inherently slow. On my Windows 10 desktop at home, `hg clone` is 3-4x faster than 7-zip.exe because hg uses multiple threads.

So, yes, I second your opinion that tc-vcs is being too smart for its own good. It feels like [now] unjustifiable complexity.
Comment 2 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-18 14:12:02 PDT
Since this bug was filed, we now have a "robustcheckout" Mercurial extension that does checkouts, well, robustly. It is already being used by mozharness and has made Mercurial interactions on buildbot infra more efficient.

https://hg.mozilla.org/try/rev/0c8c9fd64fd167466d4d4312db8ed8691c8a414d contains hacks getting this to work in TC.

I wanted to replace tc-vcs with robustcheckout. But speaking with Dustin in London, I believe his suggestion was to change tc-vcs to use robustcheckout. The benefit of that approach is tc-vcs could maintain version control agnosticism. This is a benefit because theoretically you could point TC at a Git repo of the Firefox repo and it would "just work." Apparently that was one of the initial goals of tc-vcs (although AFAICT nobody has done this with Firefox and there are no plans to do this so it is arguably cart coming before the horse). At this point, I'd be inclined to rip out tc-vcs and replace it with something simpler because tc-vcs is just adding complexity.
Comment 3 User image Dustin J. Mitchell [:dustin] 2016-07-18 14:32:51 PDT
I don't know if my feelings were that strong -- I like the agnosticism, but it's certainly cart-before-horse.  You should feel free to rip out tc-vcs and replace it wholesale with appropriate hg invocations (noting, though, that mulet builds still use tc-vcs to do git checkouts.. and for some unknown reason a few things also check out tooltool from github on every run).

The tc-vcs caches are a major PITA, as they must be primed manually every time someone adds a new repo.  So I (and I suspect :garndt) am in favor of this project!
Comment 4 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-18 15:10:42 PDT
/me slyly removes "tc-vcs" from summary line
Comment 5 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-18 18:05:07 PDT
Dustin: could you please enlighten me about the security model (read: scopes) of TaskCluster and how sharing VCS storage may impact things? (We had talked about this briefly on IRC a few weeks ago but I had just gotten back from Europe, was jet lagged, and didn't want to talk about it at the time.)

Essentially, you point the robustcheckout extension at a cache base directory. When it clones, it looks at the SHA-1 of the root revision in the remote repo and uses the SHA-1 as the repo directory in the cache. e.g. /hg-shared/8ba995b74e18334ab3707f27e9eb8f4e37ba3d29 would be the repo directory for mozilla-central, fx-team, inbound, try, etc because they all share the same root commit. It uses `hg share` under the hood to link a checkout back to the cache directory. The theory is that each revision is only stored once because you only have 1 copy for each logical group of repos.

I guess the question is whether we can share a cache across scopes or whether we'd need to create separate caches for separate scopes. The checkouts would be independent. So the only security risks I see from allowing a less-privileged scope to share a cache with a higher-privileged scope are:

a) a less-privileged scope could contaminate the cache, leading to the higher-privilege scope executing untrusted code or some such
b) a less-privileged scope could gain access to sensitive data that a higher-privilege scope added to the cache

I'm not sure if "b" is a scenario we have in Firefox automation. Do we have "hidden" tasks that run on the same TC workers as less-privileged tasks? (I hope not because I don't trust Docker that much.)

For "a," I trust Mercurial because it verifies SHA-1s on read (like when it does checkouts). However, if you are paranoid, then you don't trust Mercurial, even if I say we can trust it.
Comment 6 User image Dustin J. Mitchell [:dustin] 2016-07-19 07:02:51 PDT
I am a little paranoid :)

The basis of our separation is scm levels -- 1 being try through 3 being integration and release branches.  Every level has a role, i.e. a set of scopes, which control what it can accesse.  For level 1, that is
  https://tools.taskcluster.net/auth/roles/#moz-tree:level:1
the interesting scope in there is
  docker-worker:cache:level-1-*
which means that level-1 tasks can access caches with the prefix "level-1-*".  So that separation is enforced at the docker level (via volume mounts).  We could define a cache named "level-1-hgshare" that would be shared among level-1 tasks, but not shared with level 2 or 3.

In bug 1220686 there is work underway to use separate workerTypes, and thus separate instances, for each level, which will avoid depending entirely on docker/kernel security.  Mercurial's own security model is a great additional layer here.

Regarding the risks you see: the above should address both, but I can't fathom what secret data might be added to an hgshared repository so I might be missing something for (b).

All of this rests on the decision task being created with a revision and level which match.  That is, we want to be sure we never start a level-3 decision task with a changeset that was pushed by someone with only level-1 access.  To date, we've determined the "level of the changeset" by the repository to which the push was made -- pushes to try trigger level-1 decision tasks, while pushes to mozilla-inbound trigger level-3 decision tasks.  I think that's out of scope for this bug, but it's an important and related point so IMHO worth noting.
Comment 7 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 15:25:10 PDT
Created attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

Visual aligning makes diffs harder to read. Use line continuations
to avoid this. Also make the package list alphabetical.

Review commit: https://reviewboard.mozilla.org/r/65346/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65346/
Comment 8 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 15:25:10 PDT
Created attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

More control and more deterministic than installing via apt. This also
upgrades the Mercurial used by the image from 3.7.x.

Since the OS won't provide a global hgrc, we create one ourselves.

Review commit: https://reviewboard.mozilla.org/r/65348/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65348/
Comment 9 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 15:25:10 PDT
Created attachment 8772604 [details]
Bug 1247168 - Don't remove hg-shared scopes;

These scopes will be introduced in subsequent commits. We don't want
them to get removed, just like we don't want the tc-vcs caches to get
removed.

Review commit: https://reviewboard.mozilla.org/r/65350/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65350/
Comment 10 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 15:25:10 PDT
Created attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

https://hg.mozilla.org/mozilla-unified contains heads from all the
major Firefox repos. In addition, it is encoded on the server in such a
way that it is several hundred megabytes smaller despite containing
30,000+ more changesets. This means faster clones and faster
operations.

Cloning this repo does require a new version of Mercurial. So if any
TC tasks are using an ancient Mercurial - one that is vulnerable to
known CVE issues in fact - this will flush them out.

Review commit: https://reviewboard.mozilla.org/r/65352/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65352/
Comment 11 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 15:25:10 PDT
Created attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

The robustcheckout Mercurial extension does a clone+checkout optimally.
Read the bug for more on it.

robustcheckout is already used by mozharness automation. It has resulted
in a significant reduction in I/O usage and utilization in automation.

This commit replaces tc-vcs with the robustcheckout equivalent.

We replace the existing tc-vcs scope and cache with a new one.
Because Dustin and I are paranoid, we maintain separate caches per
SCM level - even though we could arguably share the same cache. Defense
in depth.

Robustcheckout (when used with --sharebase) pools storage for related
repos automatically. i.e. changesets from inbound and central will
be in the same store. This means you likely only have one copy of
each changeset per cache. This can result in significant space savings.
And, since there are fewer copies floating around, hg.mozilla.org
and various network appliances are working less too!

Since tc-vcs is no longer used, we stop it from being installed.

Review commit: https://reviewboard.mozilla.org/r/65354/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65354/
Comment 12 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 15:25:10 PDT
Created attachment 8772607 [details]
Bug 1247168 - Update Mercurial version and add robustcheckout;

We should be using the latest stable release of Mercurial in automation
to get performance gains and to squash bugs.

This commit installs Mercurial 3.8.4 and the robustcheckout
extension on the centos6-build image. It also installs a global hgrc
similar to the one we used for the lint image.

The centos6-build-upd image has been updated to bump image versions.

Review commit: https://reviewboard.mozilla.org/r/65356/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65356/
Comment 13 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 15:25:10 PDT
Created attachment 8772608 [details]
Bug 1247168 - Use robustcheckout in desktop-build task;

Like we've done for the lint tasks, we convert desktop-build to use
the robustcheckout extension instead of tc-vcs.

The logic is slightly more complicated because of various
environment variables.

Some unused environment variables have been removed.

Review commit: https://reviewboard.mozilla.org/r/65358/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65358/
Comment 14 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 15:35:54 PDT
Comment on attachment 8772608 [details]
Bug 1247168 - Use robustcheckout in desktop-build task;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/1-2/
Comment 15 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:01:14 PDT
Created attachment 8772622 [details]
Bug 1247168 - Upgrade to pip 8.1.2 and setuptools 24.0.3 in decision image;

Before I update this image to change VCS interactions, I thought I'd
bloat scope to update some Python packaging versions.

I dropped the virtualenv package because AFAICT nobody is using it.
In fact, there are a number of things in the decision image I don't
think should belong. Curious.

Review commit: https://reviewboard.mozilla.org/r/65374/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65374/
Comment 16 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:01:14 PDT
Created attachment 8772623 [details]
Bug 1247168 - Use robustcheckout in decision image;

Similar to other commits. We uprade to Mercurial 3.8.4. We install
robustcheckout extension and configure the /etc/mercurial/hgrc file.
We change the task to use `hg robustcheckout` instead of tc-vcs.

Review commit: https://reviewboard.mozilla.org/r/65376/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65376/
Comment 17 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:01:14 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/1-2/
Comment 18 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:01:14 PDT
Comment on attachment 8772604 [details]
Bug 1247168 - Don't remove hg-shared scopes;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/1-2/
Comment 19 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:01:14 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/1-2/
Comment 20 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:01:14 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/1-2/
Comment 21 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:01:14 PDT
Comment on attachment 8772607 [details]
Bug 1247168 - Update Mercurial version and add robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/1-2/
Comment 22 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:01:14 PDT
Comment on attachment 8772608 [details]
Bug 1247168 - Use robustcheckout in desktop-build task;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/2-3/
Comment 23 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:25:36 PDT
https://reviewboard.mozilla.org/r/65352/#review62404

This change didn't do what I wanted everywhere: Try jobs are still using mozilla-central as the base repo. I guess I'll need to seek out and destroy more references...
Comment 24 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:47:56 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/2-3/
Comment 25 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:47:56 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/2-3/
Comment 26 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:47:56 PDT
Comment on attachment 8772607 [details]
Bug 1247168 - Update Mercurial version and add robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/2-3/
Comment 27 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:47:56 PDT
Comment on attachment 8772608 [details]
Bug 1247168 - Use robustcheckout in desktop-build task;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/3-4/
Comment 28 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:47:56 PDT
Comment on attachment 8772622 [details]
Bug 1247168 - Upgrade to pip 8.1.2 and setuptools 24.0.3 in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65374/diff/1-2/
Comment 29 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-19 16:47:56 PDT
Comment on attachment 8772623 [details]
Bug 1247168 - Use robustcheckout in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/1-2/
Comment 30 User image Dustin J. Mitchell [:dustin] 2016-07-20 07:35:30 PDT
Comment on attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

https://reviewboard.mozilla.org/r/65346/#review62538
Comment 31 User image Dustin J. Mitchell [:dustin] 2016-07-20 07:36:21 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

https://reviewboard.mozilla.org/r/65348/#review62540
Comment 32 User image Dustin J. Mitchell [:dustin] 2016-07-20 07:37:01 PDT
Comment on attachment 8772604 [details]
Bug 1247168 - Don't remove hg-shared scopes;

https://reviewboard.mozilla.org/r/65350/#review62542
Comment 33 User image Dustin J. Mitchell [:dustin] 2016-07-20 07:37:57 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

https://reviewboard.mozilla.org/r/65352/#review62544

This is great!
Comment 34 User image Dustin J. Mitchell [:dustin] 2016-07-20 07:41:25 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

https://reviewboard.mozilla.org/r/65354/#review62546
Comment 35 User image Dustin J. Mitchell [:dustin] 2016-07-20 07:42:53 PDT
https://reviewboard.mozilla.org/r/65348/#review62548

::: testing/docker/lint/system-setup.sh:73
(Diff revision 2)
>  npm install -g taskcluster-vcs-v2.3.12.tar.gz
>  
>  
>  ###
>  # Flake8 Setup
>  ###

This comment is no longer accurate.  Either change it to reflect that this is installing python packages in bulk, or use a separate pip invocation to install mercurial.

(sorry I missed this on the first go-round)
Comment 36 User image Dustin J. Mitchell [:dustin] 2016-07-20 07:46:12 PDT
Comment on attachment 8772607 [details]
Bug 1247168 - Update Mercurial version and add robustcheckout;

https://reviewboard.mozilla.org/r/65356/#review62552

::: testing/docker/centos6-build-upd/VERSION:1
(Diff revision 3)
> -0.1.6.20160329195300
> +0.1.7.201607192200

You'll also need to change desktop-build/Dockerfile to refer to this version

::: testing/docker/centos6-build/VERSION:1
(Diff revision 3)
> -0.1.6
> +0.1.7

:egoktas just released 0.1.7 to install something else, so this will need to be 0.1.8 after rebase (bug 1272629)
Comment 37 User image Dustin J. Mitchell [:dustin] 2016-07-20 08:31:20 PDT
Comment on attachment 8772622 [details]
Bug 1247168 - Upgrade to pip 8.1.2 and setuptools 24.0.3 in decision image;

https://reviewboard.mozilla.org/r/65374/#review62562

This image hasn't gotten a lot of attention, so thanks for updating it!
Comment 38 User image Dustin J. Mitchell [:dustin] 2016-07-20 08:32:48 PDT
Comment on attachment 8772623 [details]
Bug 1247168 - Use robustcheckout in decision image;

https://reviewboard.mozilla.org/r/65376/#review62564
Comment 39 User image Dustin J. Mitchell [:dustin] 2016-07-20 08:33:24 PDT
Comment on attachment 8772608 [details]
Bug 1247168 - Use robustcheckout in desktop-build task;

https://reviewboard.mozilla.org/r/65358/#review62560

::: testing/docker/desktop-build/Dockerfile:1
(Diff revision 4)
> -FROM          taskcluster/centos6-build-upd:0.1.6.20160329195300
> +FROM          taskcluster/centos6-build-upd:0.1.7.201607192200

Ah, I see you've added this in the next cset.  I would perfer to see all of these numbers change in the same cset (well, honestly, I'd like to not use version numbers..)

::: testing/docker/desktop-build/bin/checkout-sources.sh:29
(Diff revision 4)
>  
>  : WORKSPACE                     ${WORKSPACE:=/home/worker/workspace}
>  
>  set -v
>  
> +checkoutargs = "--purge --sharebase /home/worker/hg-shared"

bash: checkoutargs: command not found

(spaces aren't allowed around `=`)

::: testing/docker/desktop-build/bin/checkout-sources.sh:42
(Diff revision 4)
> -        ln -s $WORKSPACE/build/tools build/tools
>      fi
> +
> +    mkdir -p build ${WORKSPACE}/build
> +    hg robustcheckout ${args} ${TOOLS_HEAD_REPOSITORY} ${WORKSPACE}/build/tools
> +    hg robustcheckout ${args} ${TOOLS_HEAD_REPOSITORY} build/tools

why not use the symbolic link?

::: testing/docker/desktop-build/bin/checkout-sources.sh:52
(Diff revision 4)
>      HEAD_REPO="${extra_repo}_HEAD_REPOSITORY"
>      HEAD_REV="${extra_repo}_HEAD_REV"
>      HEAD_REF="${extra_repo}_HEAD_REF"
>      DEST_DIR="${extra_repo}_DEST_DIR"
>  
>      tc-vcs checkout ${!DEST_DIR} ${!BASE_REPO} ${!HEAD_REPO} ${!HEAD_REV} ${!HEAD_REF}

This is used for android partner builds, for which check with :jlund -- in `taskcluster/ci/legacy/tasks/builds/android_api_15_partner_sample1.yml` they are git repositories.  I'd hate to leave this one use of tc-vcs hanging around :(
Comment 40 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Created attachment 8772925 [details]
Bug 1247168 - Change decision image to Ubuntu 16.04;

We're currently running Ubuntu 14.04 in the decision image. While still
in LTS support, 16.04 ships with a modern, properly configured Python
2.7. So we upgrade to 16.04 and drop the install of Python from source
because it is no longer needed.

This is part 1 of a larger refactor to this image.

Review commit: https://reviewboard.mozilla.org/r/65604/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65604/
Comment 41 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Created attachment 8772926 [details]
Bug 1247168 - Don't install virtualenv in the decision image;

AFAICT this isn't needed.

Review commit: https://reviewboard.mozilla.org/r/65606/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65606/
Comment 42 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Created attachment 8772927 [details]
Bug 1247168 - Install Mercurial 3.8.4 from .deb files;

We change the installation of Mercurial from via peep to .deb files in
tooltool. The .deb files were produced by Mercurial's built-in make
targets to produce .deb packages.

As part of this, we upgrade to Mercurial 3.8.4. It should be a drop-in
replacement.

Since we no longer use peep, we stop installing it and pip/setuptools
since they were only needed to run peep.

Review commit: https://reviewboard.mozilla.org/r/65608/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65608/
Comment 43 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Created attachment 8772928 [details]
Bug 1247168 - Stop installing most system packages;

I'm not sure why the decision image has so many packages installed.
Most of them don't need to exist because the decision image only
needs to obtain a copy of the Firefox repo and run `mach`. This
doesn't require any build system per se. And all the Python
dependencies are vendored in the Firefox repo. All we need is a
Python 2.7 interpreter.

This change reduces the decision image size from ~700 MB to ~300 MB.

Review commit: https://reviewboard.mozilla.org/r/65610/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65610/
Comment 44 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Created attachment 8772929 [details]
Bug 1247168 - Vendor tooltool.py;

Previously, we were downloading tooltool.py from github.com. There
were 2 problems with this.

First, there is a dependency on a 3rd party service. While the Docker
image should be cached, as a matter of principle we don't like hitting
3rd party services in our automation. The file is small enough, so we
just vendor it.

Second - and more importantly - we weren't validating the integrity of
the downloaded file. This means that a MiTM could possibly alter the
content of the file without us knowing (they would need a valid CA but
since the Ubuntu trusted CA bundle contains a lot of CAs from e.g.
governments, this isn't out of the question). Vendoring the file removes
this risk.

Third, behavior wasn't deterministic over time. We were always
downloading the "master" revision of the file. I like determinism over
time. Vendoring makes things deterministic.

Since curl was only used to download tooltool.py, we remove that package
install, which also removes a bunch of dependency packages and cuts down
the image size by several megabytes.

Review commit: https://reviewboard.mozilla.org/r/65612/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65612/
Comment 45 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Created attachment 8772930 [details]
Bug 1247168 - Configure Mercurial and install global hgrc;

web.cacerts matches what the Ubuntu package does by default.

[progress] changes are to make output in TaskCluster logs less
spammy (only 1 update per second instead of up to 10).

The robustcheckout extension will be used in a subsequent commit to
handle repository checkouts.

Review commit: https://reviewboard.mozilla.org/r/65614/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65614/
Comment 46 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Created attachment 8772931 [details]
Bug 1247168 - Tag and use version 0.1.2 of the decision image;

Changes to the decision Docker image have been compelted. We're ready to
use the new image.

We tag the image, update version references, and change the task caches
so the new Mercurial pooled storage from the robustcheckout extension is
used.

Review commit: https://reviewboard.mozilla.org/r/65616/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65616/
Comment 47 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Comment on attachment 8772623 [details]
Bug 1247168 - Use robustcheckout in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/2-3/
Comment 48 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Comment on attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/1-2/
Comment 49 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/2-3/
Comment 50 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Comment on attachment 8772604 [details]
Bug 1247168 - Don't remove hg-shared scopes;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/2-3/
Comment 51 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/3-4/
Comment 52 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/3-4/
Comment 53 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Comment on attachment 8772607 [details]
Bug 1247168 - Update Mercurial version and add robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/3-4/
Comment 54 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 10:43:31 PDT
Comment on attachment 8772608 [details]
Bug 1247168 - Use robustcheckout in desktop-build task;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/4-5/
Comment 55 User image Dustin J. Mitchell [:dustin] 2016-07-20 11:17:51 PDT
Comment on attachment 8772931 [details]
Bug 1247168 - Tag and use version 0.1.2 of the decision image;

https://reviewboard.mozilla.org/r/65616/#review62606

Let me know when you want me to build this locally and push it.
Comment 56 User image Dustin J. Mitchell [:dustin] 2016-07-20 11:18:28 PDT
Comment on attachment 8772930 [details]
Bug 1247168 - Configure Mercurial and install global hgrc;

https://reviewboard.mozilla.org/r/65614/#review62608
Comment 57 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772926 [details]
Bug 1247168 - Don't install virtualenv in the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65606/diff/1-2/
Comment 58 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772927 [details]
Bug 1247168 - Install Mercurial 3.8.4 from .deb files;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65608/diff/1-2/
Comment 59 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772928 [details]
Bug 1247168 - Stop installing most system packages;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65610/diff/1-2/
Comment 60 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772929 [details]
Bug 1247168 - Vendor tooltool.py;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65612/diff/1-2/
Comment 61 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772930 [details]
Bug 1247168 - Configure Mercurial and install global hgrc;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65614/diff/1-2/
Comment 62 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772623 [details]
Bug 1247168 - Use robustcheckout in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/3-4/
Comment 63 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772931 [details]
Bug 1247168 - Tag and use version 0.1.2 of the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/1-2/
Comment 64 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/2-3/
Comment 65 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/3-4/
Comment 66 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772604 [details]
Bug 1247168 - Don't remove hg-shared scopes;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/3-4/
Comment 67 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/4-5/
Comment 68 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/4-5/
Comment 69 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772607 [details]
Bug 1247168 - Update Mercurial version and add robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/4-5/
Comment 70 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:18:54 PDT
Comment on attachment 8772608 [details]
Bug 1247168 - Use robustcheckout in desktop-build task;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/5-6/
Comment 71 User image Dustin J. Mitchell [:dustin] 2016-07-20 11:20:16 PDT
Comment on attachment 8772929 [details]
Bug 1247168 - Vendor tooltool.py;

https://reviewboard.mozilla.org/r/65612/#review62610

We should really vendor tooltool.py in-tree in general, rather than depending on it being in /builds or ~/build or whatever.
Comment 72 User image Dustin J. Mitchell [:dustin] 2016-07-20 11:21:36 PDT
Comment on attachment 8772928 [details]
Bug 1247168 - Stop installing most system packages;

https://reviewboard.mozilla.org/r/65610/#review62614
Comment 73 User image Dustin J. Mitchell [:dustin] 2016-07-20 11:22:43 PDT
Comment on attachment 8772927 [details]
Bug 1247168 - Install Mercurial 3.8.4 from .deb files;

https://reviewboard.mozilla.org/r/65608/#review62616

::: testing/docker/decision/system-setup.sh:78
(Diff revision 2)
> -    "unpack": false
>  }
>  ]
>  EOF
>  
> -cd $BUILD
> +dpkg -i mercurial-common_3.8.4_all.deb mercurial_3.8.4_amd64.deb

Why not install from PyPi?
Comment 74 User image Dustin J. Mitchell [:dustin] 2016-07-20 11:23:02 PDT
Comment on attachment 8772926 [details]
Bug 1247168 - Don't install virtualenv in the decision image;

https://reviewboard.mozilla.org/r/65606/#review62618
Comment 75 User image Dustin J. Mitchell [:dustin] 2016-07-20 11:24:35 PDT
Comment on attachment 8772925 [details]
Bug 1247168 - Change decision image to Ubuntu 16.04;

https://reviewboard.mozilla.org/r/65604/#review62620
Comment 76 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 11:28:27 PDT
https://reviewboard.mozilla.org/r/65608/#review62616

> Why not install from PyPi?

Because this requires having a compiler installed. I'd rather avoid the complexity (and bloat).
Comment 77 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772604 [details]
Bug 1247168 - Don't remove hg-shared scopes;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/4-5/
Comment 78 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772925 [details]
Bug 1247168 - Change decision image to Ubuntu 16.04;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65604/diff/1-2/
Comment 79 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772926 [details]
Bug 1247168 - Don't install virtualenv in the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65606/diff/2-3/
Comment 80 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772927 [details]
Bug 1247168 - Install Mercurial 3.8.4 from .deb files;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65608/diff/2-3/
Comment 81 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772928 [details]
Bug 1247168 - Stop installing most system packages;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65610/diff/2-3/
Comment 82 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772929 [details]
Bug 1247168 - Vendor tooltool.py;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65612/diff/2-3/
Comment 83 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772930 [details]
Bug 1247168 - Configure Mercurial and install global hgrc;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65614/diff/2-3/
Comment 84 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772623 [details]
Bug 1247168 - Use robustcheckout in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/4-5/
Comment 85 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772931 [details]
Bug 1247168 - Tag and use version 0.1.2 of the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/2-3/
Comment 86 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/3-4/
Comment 87 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/4-5/
Comment 88 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/5-6/
Comment 89 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/5-6/
Comment 90 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772607 [details]
Bug 1247168 - Update Mercurial version and add robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/5-6/
Comment 91 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 12:22:50 PDT
Comment on attachment 8772608 [details]
Bug 1247168 - Use robustcheckout in desktop-build task;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/6-7/
Comment 92 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:44:38 PDT
Configuring the lint tasks to use a cache for the gecko checkout, I've seen flake8 tasks complete in as little as 11.1s (https://tools.taskcluster.net/task-inspector/#AMLtSM1kTW6JjMf3tdMC1g/0).

You read that right: 11.1s.

They were taking ~4 minutes before (most of that spent creating a VCS checkout).
Comment 93 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Created attachment 8773094 [details]
Bug 1247168 - Actually use workspace cache in decision task;

The decision task configures a /home/worker/workspace cache. However,
the command we run in the container references a "workspace" relative
path. From logs in automation, it appears that PWD during execution
is "/" because "workspace" is being resolved to "/workspace." The
net result of this is we appear to be performing a VCS clone+checkout
on every single task.

This commit fixes the paths so our workspace cache is actually used.

Review commit: https://reviewboard.mozilla.org/r/65724/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65724/
Comment 94 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Created attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

Previously, every lint task would have to create its own checkout. This
was time consuming.

The robustcheckout extension purges the working copy of *all*
untracked and ignored files. It also restores modified files to
their original state. In other words, as long as you trust Mercurial
to go from revision X to revision Y, robustcheckout is as good as
a fresh checkout.

This commit adds a cache for the working directory checkout so lint
tasks only have to effectively perform incremental `hg update` between
task executions. This should make tasks spend a lot less time
doing version control foo.

On Try, time for flake8 tasks is currently hovering around 4 minutes.
After this change, I've seen tasks finish as quickly as 25s!

Review commit: https://reviewboard.mozilla.org/r/65726/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65726/
Comment 95 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Created attachment 8773096 [details]
Bug 1247168 - Bump centos6-build and centos6-build-upd image versions;

Our changes to these images are done. We bump the versions to signal the
creation of new images.

Review commit: https://reviewboard.mozilla.org/r/65728/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65728/
Comment 96 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Comment on attachment 8772623 [details]
Bug 1247168 - Use robustcheckout in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/5-6/
Comment 97 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Comment on attachment 8772931 [details]
Bug 1247168 - Tag and use version 0.1.2 of the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/3-4/
Comment 98 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Comment on attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/4-5/
Comment 99 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/5-6/
Comment 100 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/6-7/
Comment 101 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/6-7/
Comment 102 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Comment on attachment 8772607 [details]
Bug 1247168 - Update Mercurial version and add robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/6-7/
Comment 103 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 16:46:06 PDT
Comment on attachment 8772608 [details]
Bug 1247168 - Use robustcheckout in desktop-build task;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/7-8/
Comment 104 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 17:12:17 PDT
https://reviewboard.mozilla.org/r/65358/#review62560

> This is used for android partner builds, for which check with :jlund -- in `taskcluster/ci/legacy/tasks/builds/android_api_15_partner_sample1.yml` they are git repositories.  I'd hate to leave this one use of tc-vcs hanging around :(

I agree: I'd love to kill it.

It looks like this task has its origins in bug 1163084 in September/October 2015. It isn't obvious that this task runs at all. I'm tempted to kill it. Or at least break it by removing tc-vcs.
Comment 105 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-20 17:14:40 PDT
Jordan: can you please weigh in at https://reviewboard.mozilla.org/r/65358/#review62560
Comment 106 User image Dustin J. Mitchell [:dustin] 2016-07-21 05:25:17 PDT
Comment on attachment 8773094 [details]
Bug 1247168 - Actually use workspace cache in decision task;

https://reviewboard.mozilla.org/r/65724/#review62864
Comment 107 User image Dustin J. Mitchell [:dustin] 2016-07-21 05:28:15 PDT
https://reviewboard.mozilla.org/r/65726/#review62866

If `hg robustcheckout` is deleting everything, does that mean it's deleting the objdir too?

::: taskcluster/ci/legacy/tasks/lint.yml:35
(Diff revision 1)
>      # Thirty minutes should be enough for lint checks
>      maxRunTime: 1800
>  
>      cache:
>        level-{{level}}-hg-shared: '/home/worker/hg-shared'
> +      level-{{level}}-checkouts: '/home/worker/checkouts'

Why not workspace, to match other tasks?
Comment 108 User image Dustin J. Mitchell [:dustin] 2016-07-21 08:02:55 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

https://reviewboard.mozilla.org/r/65726/#review62920
Comment 109 User image Dustin J. Mitchell [:dustin] 2016-07-21 08:05:29 PDT
Comment on attachment 8773096 [details]
Bug 1247168 - Bump centos6-build and centos6-build-upd image versions;

https://reviewboard.mozilla.org/r/65728/#review62922
Comment 110 User image Dustin J. Mitchell [:dustin] 2016-07-21 08:06:45 PDT
Comment on attachment 8772608 [details]
Bug 1247168 - Use robustcheckout in desktop-build task;

syncing bz with mozreview..
Comment 111 User image Dustin J. Mitchell [:dustin] 2016-07-21 08:06:50 PDT
Comment on attachment 8772607 [details]
Bug 1247168 - Update Mercurial version and add robustcheckout;

syncing bz with mozreview..
Comment 112 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 10:14:12 PDT
*** Bug 1288377 has been marked as a duplicate of this bug. ***
Comment 113 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 10:29:50 PDT
I'm going to drop the build images/tasks from this bug and punt to a follow-up: they are a bit more work than decision and lint and I like landing things.
Comment 114 User image Jordan Lund (:jlund) 2016-07-21 11:34:15 PDT
https://reviewboard.mozilla.org/r/65358/#review62560

> I agree: I'd love to kill it.
> 
> It looks like this task has its origins in bug 1163084 in September/October 2015. It isn't obvious that this task runs at all. I'm tempted to kill it. Or at least break it by removing tc-vcs.

it's still used occasionally: https://bugzilla.mozilla.org/show_bug.cgi?id=1255119

I think #mobile were even using it for: https://bugzilla.mozilla.org/show_bug.cgi?id=1284582

It might not be needed for much longer but it would be great to still support this for now. Is there a work around? Can we just change this to a `git clone`. Does tc-vcs use git at all or does it just simply grab a tarball and our images don't even have git installed?
Comment 115 User image Dustin J. Mitchell [:dustin] 2016-07-21 12:01:22 PDT
git is installed -- tc-vcs is just a frontend for it (and does some tarball stuff too, yeah)

Is there a bug representing the "not..needed for much longer"?  If you can point to that, we can embed a pointer to it in a comment so the next person to wonder "why is this still here?" can either learn why, or learn that it can be removed.
Comment 116 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Created attachment 8773463 [details]
Bug 1247168 - Create worker user/group in decision image;

In preparation for running tasks as the worker user.

Review commit: https://reviewboard.mozilla.org/r/66210/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/66210/
Comment 117 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Created attachment 8773464 [details]
Bug 1247168 - Introduce scripts for running action and decision tasks;

When we switch to use robustcheckout for version control foo, we'll
also be taking the opportunity to have the decision and action tasks
execute as the "worker" user.

Since caches are mounted and owned by root and since tasks initially
run as root, this makes defining the container command in YAML a bit
difficult because we have to do some work as root then switch users
and continue executing. Rather than shoehorning all that complicated
logic into YAML, we introduce bash scripts that do it. These will
be plugged into the task YAML when we formally switch the tasks
to use the new Docker image.

We provide one script for running Gecko decision tasks. We provide
another for running action tasks. These are the two consumers of
the decision image we care about.

We also sneak in a change to add the executable bit to checkout-gecko.

Review commit: https://reviewboard.mozilla.org/r/66212/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/66212/
Comment 118 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Created attachment 8773465 [details]
Bug 1247168 - Add a script to perform a checkout then run a command;

The script will be used as the main command in task YAML files.
It changes ownership of caches. Then switches to the "worker" user.
Then performs a Gecko checkout. Then executes whatever command was
requested via its arguments.

Review commit: https://reviewboard.mozilla.org/r/66214/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/66214/
Comment 119 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772604 [details]
Bug 1247168 - Don't remove hg-shared scopes;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/5-6/
Comment 120 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772925 [details]
Bug 1247168 - Change decision image to Ubuntu 16.04;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65604/diff/2-3/
Comment 121 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772926 [details]
Bug 1247168 - Don't install virtualenv in the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65606/diff/3-4/
Comment 122 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772927 [details]
Bug 1247168 - Install Mercurial 3.8.4 from .deb files;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65608/diff/3-4/
Comment 123 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772928 [details]
Bug 1247168 - Stop installing most system packages;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65610/diff/3-4/
Comment 124 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772929 [details]
Bug 1247168 - Vendor tooltool.py;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65612/diff/3-4/
Comment 125 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772930 [details]
Bug 1247168 - Configure Mercurial and install global hgrc;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65614/diff/3-4/
Comment 126 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8773094 [details]
Bug 1247168 - Actually use workspace cache in decision task;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65724/diff/1-2/
Comment 127 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772623 [details]
Bug 1247168 - Use robustcheckout in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/6-7/
Comment 128 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772931 [details]
Bug 1247168 - Tag and use version 0.1.2 of the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/4-5/
Comment 129 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/5-6/
Comment 130 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/6-7/
Comment 131 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/7-8/
Comment 132 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/7-8/
Comment 133 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 13:38:47 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/1-2/
Comment 134 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:34:34 PDT
Comment on attachment 8773464 [details]
Bug 1247168 - Introduce scripts for running action and decision tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66212/diff/1-2/
Comment 135 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:34:34 PDT
Comment on attachment 8772931 [details]
Bug 1247168 - Tag and use version 0.1.2 of the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/5-6/
Comment 136 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:34:34 PDT
Comment on attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/6-7/
Comment 137 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:34:34 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/7-8/
Comment 138 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:34:34 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/8-9/
Comment 139 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:34:34 PDT
Comment on attachment 8773465 [details]
Bug 1247168 - Add a script to perform a checkout then run a command;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/1-2/
Comment 140 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:34:34 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/8-9/
Comment 141 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:34:34 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/2-3/
Comment 142 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:43:13 PDT
Comment on attachment 8773465 [details]
Bug 1247168 - Add a script to perform a checkout then run a command;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/2-3/
Comment 143 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:43:13 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/9-10/
Comment 144 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:43:13 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/3-4/
Comment 145 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:50:57 PDT
Comment on attachment 8773465 [details]
Bug 1247168 - Add a script to perform a checkout then run a command;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/3-4/
Comment 146 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:50:57 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/10-11/
Comment 147 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 14:50:57 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/4-5/
Comment 148 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 16:32:15 PDT
Created attachment 8773543 [details]
Bug 1247168 - Add special Dockerfile syntax to add arbitrary files to context;

A limitation of traditional `docker build` is that it only has access
to files in the same directory as the Dockerfile.

Typically, when you do `docker build`, Docker will create a tar archive
of all files in the same directory as the Dockerfile and upload that to
Docker and the image building process will have access to all files in
the archive.

Over a year ago, I realized you could write some code to create custom
context archives and talk to the Docker build API directly to use your
custom archive. I hacked some code into version-control-tools that
parsed Dockerfiles for special syntax denoting extra paths from the
source checkout to add to the context and proceed to add them to
context archives. This commit essentially copied that code for use
by taskgraph's built-in Docker image building.

Using the syntax "# %include <path>" you are able to include paths
or directories (relative from the top source directory root) in the
generated context archive. Files add this way are available under the
"topsrcdir/" path.

The "lint" image has been changed to use this syntax to add in
in-tree version of tooltool.py (instead of downloading from github.com).
This eliminates a dependency on a third party service and increases
security and determinism. Yay.

In order to write tests, I had to make archiving deterministic. That's
why we no longer use a single "tar.add()" for the Dockerfile directory.
Instead, we obtain the list of files up front, sort them, then add with
uid/gid set to 0, so uid/gid is consistent no matter what it is on the
filesystem performing context creation. More determinism, yay.

I would like to test this feature a bit more. However, the test
environment for custom Docker image building doesn't currently
facilitate custom source paths: it expects Docker files to be in
$topsrcdir/testing/docker. If we add more functionality to this, we
should definitely invest in writing better tests.

Review commit: https://reviewboard.mozilla.org/r/66240/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/66240/
Comment 149 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 16:32:15 PDT
Comment on attachment 8773465 [details]
Bug 1247168 - Add a script to perform a checkout then run a command;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/4-5/
Comment 150 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 16:32:15 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/11-12/
Comment 151 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-21 16:32:15 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/5-6/
Comment 152 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Created attachment 8773941 [details]
Bug 1247168 - Use vendored tooltool in lint image;

Using our special Dockerfile syntax to include arbitrary files, we
include the previously vendored tooltool.py file in the image build
context and add it directly from there. No github.com communication
needed.

Review commit: https://reviewboard.mozilla.org/r/66566/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/66566/
Comment 153 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8773463 [details]
Bug 1247168 - Create worker user/group in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66210/diff/1-2/
Comment 154 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772604 [details]
Bug 1247168 - Don't remove hg-shared scopes;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/6-7/
Comment 155 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772925 [details]
Bug 1247168 - Change decision image to Ubuntu 16.04;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65604/diff/3-4/
Comment 156 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772926 [details]
Bug 1247168 - Don't install virtualenv in the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65606/diff/4-5/
Comment 157 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772927 [details]
Bug 1247168 - Install Mercurial 3.8.4 from .deb files;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65608/diff/4-5/
Comment 158 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772928 [details]
Bug 1247168 - Stop installing most system packages;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65610/diff/4-5/
Comment 159 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772929 [details]
Bug 1247168 - Vendor tooltool.py;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65612/diff/4-5/
Comment 160 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772930 [details]
Bug 1247168 - Configure Mercurial and install global hgrc;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65614/diff/4-5/
Comment 161 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8773094 [details]
Bug 1247168 - Actually use workspace cache in decision task;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65724/diff/2-3/
Comment 162 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772623 [details]
Bug 1247168 - Use robustcheckout in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/7-8/
Comment 163 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8773464 [details]
Bug 1247168 - Introduce scripts for running action and decision tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66212/diff/2-3/
Comment 164 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772931 [details]
Bug 1247168 - Tag and use version 0.1.2 of the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/6-7/
Comment 165 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/7-8/
Comment 166 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/8-9/
Comment 167 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/9-10/
Comment 168 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8773465 [details]
Bug 1247168 - Add a script to perform a checkout then run a command;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/5-6/
Comment 169 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/12-13/
Comment 170 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 13:59:41 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/6-7/
Comment 171 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 14:07:56 PDT
Comment on attachment 8773465 [details]
Bug 1247168 - Add a script to perform a checkout then run a command;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/6-7/
Comment 172 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 14:07:56 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/13-14/
Comment 173 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 14:07:56 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/7-8/
Comment 174 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-22 17:19:19 PDT
I rebased this on top of bug 1288567. dustin: you may want to look at interdiffs, as some commits you already r+d have changed a bit.
Comment 175 User image Dustin J. Mitchell [:dustin] 2016-07-25 10:29:45 PDT
Comment on attachment 8773463 [details]
Bug 1247168 - Create worker user/group in decision image;

https://reviewboard.mozilla.org/r/66210/#review63646
Comment 176 User image Dustin J. Mitchell [:dustin] 2016-07-25 11:14:07 PDT
Comment on attachment 8773941 [details]
Bug 1247168 - Use vendored tooltool in lint image;

https://reviewboard.mozilla.org/r/66566/#review63672
Comment 177 User image Dustin J. Mitchell [:dustin] 2016-07-25 11:15:40 PDT
Comment on attachment 8773465 [details]
Bug 1247168 - Add a script to perform a checkout then run a command;

https://reviewboard.mozilla.org/r/66214/#review63676
Comment 178 User image Dustin J. Mitchell [:dustin] 2016-07-25 11:18:23 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

https://reviewboard.mozilla.org/r/65726/#review63678
Comment 179 User image Dustin J. Mitchell [:dustin] 2016-07-25 11:22:02 PDT
Comment on attachment 8773464 [details]
Bug 1247168 - Introduce scripts for running action and decision tasks;

https://reviewboard.mozilla.org/r/66212/#review63682
Comment 180 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8773463 [details]
Bug 1247168 - Create worker user/group in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66210/diff/2-3/
Comment 181 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772604 [details]
Bug 1247168 - Don't remove hg-shared scopes;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/7-8/
Comment 182 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772925 [details]
Bug 1247168 - Change decision image to Ubuntu 16.04;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65604/diff/4-5/
Comment 183 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772926 [details]
Bug 1247168 - Don't install virtualenv in the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65606/diff/5-6/
Comment 184 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772927 [details]
Bug 1247168 - Install Mercurial 3.8.4 from .deb files;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65608/diff/5-6/
Comment 185 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772928 [details]
Bug 1247168 - Stop installing most system packages;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65610/diff/5-6/
Comment 186 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772929 [details]
Bug 1247168 - Vendor tooltool.py;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65612/diff/5-6/
Comment 187 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772930 [details]
Bug 1247168 - Configure Mercurial and install global hgrc;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65614/diff/5-6/
Comment 188 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8773094 [details]
Bug 1247168 - Actually use workspace cache in decision task;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65724/diff/3-4/
Comment 189 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772623 [details]
Bug 1247168 - Use robustcheckout in decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/8-9/
Comment 190 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8773464 [details]
Bug 1247168 - Introduce scripts for running action and decision tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66212/diff/3-4/
Comment 191 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772931 [details]
Bug 1247168 - Tag and use version 0.1.2 of the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/7-8/
Comment 192 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/8-9/
Comment 193 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8773941 [details]
Bug 1247168 - Use vendored tooltool in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66566/diff/1-2/
Comment 194 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/9-10/
Comment 195 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/10-11/
Comment 196 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8773465 [details]
Bug 1247168 - Add a script to perform a checkout then run a command;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/7-8/
Comment 197 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/14-15/
Comment 198 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 12:52:03 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/8-9/
Comment 199 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 13:07:58 PDT
Comment on attachment 8772931 [details]
Bug 1247168 - Tag and use version 0.1.2 of the decision image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/8-9/
Comment 200 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 13:07:58 PDT
Comment on attachment 8772602 [details]
Bug 1247168 - Reformat requirements.txt;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/9-10/
Comment 201 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 13:07:58 PDT
Comment on attachment 8773941 [details]
Bug 1247168 - Use vendored tooltool in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66566/diff/2-3/
Comment 202 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 13:07:58 PDT
Comment on attachment 8772603 [details]
Bug 1247168 - Install Mercurial 3.8.4 in lint image;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/10-11/
Comment 203 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 13:07:58 PDT
Comment on attachment 8772605 [details]
Bug 1247168 - Make mozilla-unified the base repository for Firefox;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/11-12/
Comment 204 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 13:07:58 PDT
Comment on attachment 8773465 [details]
Bug 1247168 - Add a script to perform a checkout then run a command;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/8-9/
Comment 205 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 13:07:58 PDT
Comment on attachment 8772606 [details]
Bug 1247168 - Convert lint image and tasks to use robustcheckout;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/15-16/
Comment 206 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 13:07:58 PDT
Comment on attachment 8773095 [details]
Bug 1247168 - Use a cache for repo checkout in lint tasks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/9-10/
Comment 207 User image Pulsebot 2016-07-25 13:21:50 PDT
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/085b1857a36f
Create worker user/group in decision image; r=dustin
https://hg.mozilla.org/integration/autoland/rev/fcc1771bbe23
Don't remove hg-shared scopes; r=dustin
https://hg.mozilla.org/integration/autoland/rev/eb08ab6d2349
Change decision image to Ubuntu 16.04; r=dustin
https://hg.mozilla.org/integration/autoland/rev/418fc99dfe92
Don't install virtualenv in the decision image; r=dustin
https://hg.mozilla.org/integration/autoland/rev/4f1747b9ce4a
Install Mercurial 3.8.4 from .deb files; r=dustin
https://hg.mozilla.org/integration/autoland/rev/d3a1a822bbaf
Stop installing most system packages; r=dustin
https://hg.mozilla.org/integration/autoland/rev/846f7e94c1ff
Vendor tooltool.py; r=dustin
https://hg.mozilla.org/integration/autoland/rev/34a4e796fa4d
Configure Mercurial and install global hgrc; r=dustin
https://hg.mozilla.org/integration/autoland/rev/6c9ced9abf7e
Actually use workspace cache in decision task; r=dustin
https://hg.mozilla.org/integration/autoland/rev/7450495f5182
Use robustcheckout in decision image; r=dustin
https://hg.mozilla.org/integration/autoland/rev/8b43386ca744
Introduce scripts for running action and decision tasks; r=dustin
https://hg.mozilla.org/integration/autoland/rev/4bb063392974
Tag and use version 0.1.2 of the decision image; r=dustin
https://hg.mozilla.org/integration/autoland/rev/3ad658f9ddc5
Reformat requirements.txt; r=dustin
https://hg.mozilla.org/integration/autoland/rev/f80ee4875041
Use vendored tooltool in lint image; r=dustin
https://hg.mozilla.org/integration/autoland/rev/2a31541145b5
Install Mercurial 3.8.4 in lint image; r=dustin
https://hg.mozilla.org/integration/autoland/rev/cab3a2a82ac0
Make mozilla-unified the base repository for Firefox; r=dustin
https://hg.mozilla.org/integration/autoland/rev/423fef62913f
Add a script to perform a checkout then run a command; r=dustin
https://hg.mozilla.org/integration/autoland/rev/b4c3e613fee8
Convert lint image and tasks to use robustcheckout; r=dustin
https://hg.mozilla.org/integration/autoland/rev/722707a63810
Use a cache for repo checkout in lint tasks; r=dustin
Comment 208 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-25 15:04:27 PDT
https://public-artifacts.taskcluster.net/Lzzl8h9rSqOzYke2wNo3lA/0/public/logs/live_backing.log is a flake8 task running in 9.87s on the autoland repo.

That's ideal conditions: the worker already had a cached working copy of the revision we requested to run, so essentially no file writes were performed.

In contrast, before we weren't using cached working directories and had to obtain a fresh checkout on every run. The fastest flake8 task I saw before was in the ~3 minute range.

Note You need to log in before you can comment on or make changes to this bug.