Closed Bug 1247168 Opened 9 years ago Closed 8 years ago

Use pooled/shared storage and robustcheckout extension

Categories

(Taskcluster :: General, defect, P1)

Product:
Taskcluster
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: glandium, Assigned: gps)

References

Details

Attachments

(19 files, 5 obsolete files)

Bug 1247168 - Reformat requirements.txt;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Install Mercurial 3.8.4 in lint image;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Don't remove hg-shared scopes;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Make mozilla-unified the base repository for Firefox;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Convert lint image and tasks to use robustcheckout;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Use robustcheckout in decision image;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Change decision image to Ubuntu 16.04;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Don't install virtualenv in the decision image;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Install Mercurial 3.8.4 from .deb files;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Stop installing most system packages;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Vendor tooltool.py;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Configure Mercurial and install global hgrc;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Tag and use version 0.1.2 of the decision image;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Actually use workspace cache in decision task;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Use a cache for repo checkout in lint tasks;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Create worker user/group in decision image;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Introduce scripts for running action and decision tasks;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Add a script to perform a checkout then run a command;
58 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1247168 - Use vendored tooltool in lint image;
58 bytes, text/x-review-board-request
dustin
: review+
Details
tc-vcs is downloading tarballs of repositories as a first step. It should stop doing that for mercurial, because mercurial will do it much more efficiently with bundles (requires version 3.6, iirc).
Blocks: 1247169
No longer blocks: 1247169
tc-vcs is currently snapshotting the .hg and working copy directories into tar.gz files and shipping those around. .hg won't compress very well because most content therein is already zlib compressed. So there's overhead doing the decompress of those files. The working directory will compress well. However, I'm not convinced that `tar` is faster than hg when a streaming clone is involved. A streaming hg clone is effectively a tar file (no compression). When using S3, it will complete as fast as the network and I/O can send/write bytes. I've seen EC2 machines apply streaming clones at 90 MB/s. I've seen Try builds apply stream clone bundles at 25-35 MB/s. But that's with less beefy and shared EC2 instances. This appears to be on par or faster than the tarball downloads from https://queue.taskcluster.net/. For the working copy, `hg up` will use multiple processes for writing files. This theoretically has a leg up on `tar` which uses a single I/O thread. However, my experience is your filesystem will be the limiting factor here, so `hg up` and `tar` tie in terms of performance. On Windows, streaming clones with hg 3.7+ should *always* be faster than tar because Mercurial will use multiple I/O threads for writing files. It turns out writing thousands of files on NTFS is inherently slow. On my Windows 10 desktop at home, `hg clone` is 3-4x faster than 7-zip.exe because hg uses multiple threads. So, yes, I second your opinion that tc-vcs is being too smart for its own good. It feels like [now] unjustifiable complexity.
Since this bug was filed, we now have a "robustcheckout" Mercurial extension that does checkouts, well, robustly. It is already being used by mozharness and has made Mercurial interactions on buildbot infra more efficient. https://hg.mozilla.org/try/rev/0c8c9fd64fd167466d4d4312db8ed8691c8a414d contains hacks getting this to work in TC. I wanted to replace tc-vcs with robustcheckout. But speaking with Dustin in London, I believe his suggestion was to change tc-vcs to use robustcheckout. The benefit of that approach is tc-vcs could maintain version control agnosticism. This is a benefit because theoretically you could point TC at a Git repo of the Firefox repo and it would "just work." Apparently that was one of the initial goals of tc-vcs (although AFAICT nobody has done this with Firefox and there are no plans to do this so it is arguably cart coming before the horse). At this point, I'd be inclined to rip out tc-vcs and replace it with something simpler because tc-vcs is just adding complexity.
Blocks: 1286336
Summary: tc-vcs should not try to be smart with mercurial → tc-vcs should use pooled/shared storage and robustcheckout extension
I don't know if my feelings were that strong -- I like the agnosticism, but it's certainly cart-before-horse. You should feel free to rip out tc-vcs and replace it wholesale with appropriate hg invocations (noting, though, that mulet builds still use tc-vcs to do git checkouts.. and for some unknown reason a few things also check out tooltool from github on every run). The tc-vcs caches are a major PITA, as they must be primed manually every time someone adds a new repo. So I (and I suspect :garndt) am in favor of this project!
/me slyly removes "tc-vcs" from summary line
Summary: tc-vcs should use pooled/shared storage and robustcheckout extension → use pooled/shared storage and robustcheckout extension
Dustin: could you please enlighten me about the security model (read: scopes) of TaskCluster and how sharing VCS storage may impact things? (We had talked about this briefly on IRC a few weeks ago but I had just gotten back from Europe, was jet lagged, and didn't want to talk about it at the time.) Essentially, you point the robustcheckout extension at a cache base directory. When it clones, it looks at the SHA-1 of the root revision in the remote repo and uses the SHA-1 as the repo directory in the cache. e.g. /hg-shared/8ba995b74e18334ab3707f27e9eb8f4e37ba3d29 would be the repo directory for mozilla-central, fx-team, inbound, try, etc because they all share the same root commit. It uses `hg share` under the hood to link a checkout back to the cache directory. The theory is that each revision is only stored once because you only have 1 copy for each logical group of repos. I guess the question is whether we can share a cache across scopes or whether we'd need to create separate caches for separate scopes. The checkouts would be independent. So the only security risks I see from allowing a less-privileged scope to share a cache with a higher-privileged scope are: a) a less-privileged scope could contaminate the cache, leading to the higher-privilege scope executing untrusted code or some such b) a less-privileged scope could gain access to sensitive data that a higher-privilege scope added to the cache I'm not sure if "b" is a scenario we have in Firefox automation. Do we have "hidden" tasks that run on the same TC workers as less-privileged tasks? (I hope not because I don't trust Docker that much.) For "a," I trust Mercurial because it verifies SHA-1s on read (like when it does checkouts). However, if you are paranoid, then you don't trust Mercurial, even if I say we can trust it.
Flags: needinfo?(dustin)
Assignee: nobody → gps
Priority: -- → P1
Summary: use pooled/shared storage and robustcheckout extension → Use pooled/shared storage and robustcheckout extension
I am a little paranoid :) The basis of our separation is scm levels -- 1 being try through 3 being integration and release branches. Every level has a role, i.e. a set of scopes, which control what it can accesse. For level 1, that is https://tools.taskcluster.net/auth/roles/#moz-tree:level:1 the interesting scope in there is docker-worker:cache:level-1-* which means that level-1 tasks can access caches with the prefix "level-1-*". So that separation is enforced at the docker level (via volume mounts). We could define a cache named "level-1-hgshare" that would be shared among level-1 tasks, but not shared with level 2 or 3. In bug 1220686 there is work underway to use separate workerTypes, and thus separate instances, for each level, which will avoid depending entirely on docker/kernel security. Mercurial's own security model is a great additional layer here. Regarding the risks you see: the above should address both, but I can't fathom what secret data might be added to an hgshared repository so I might be missing something for (b). All of this rests on the decision task being created with a revision and level which match. That is, we want to be sure we never start a level-3 decision task with a changeset that was pushed by someone with only level-1 access. To date, we've determined the "level of the changeset" by the repository to which the push was made -- pushes to try trigger level-1 decision tasks, while pushes to mozilla-inbound trigger level-3 decision tasks. I think that's out of scope for this bug, but it's an important and related point so IMHO worth noting.
Flags: needinfo?(dustin)
Visual aligning makes diffs harder to read. Use line continuations to avoid this. Also make the package list alphabetical. Review commit: https://reviewboard.mozilla.org/r/65346/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65346/
Attachment #8772602 - Flags: review?(dustin)
Attachment #8772603 - Flags: review?(dustin)
Attachment #8772604 - Flags: review?(dustin)
Attachment #8772605 - Flags: review?(dustin)
Attachment #8772606 - Flags: review?(dustin)
Attachment #8772607 - Flags: review?(dustin)
Attachment #8772608 - Flags: review?(dustin)
More control and more deterministic than installing via apt. This also upgrades the Mercurial used by the image from 3.7.x. Since the OS won't provide a global hgrc, we create one ourselves. Review commit: https://reviewboard.mozilla.org/r/65348/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65348/
These scopes will be introduced in subsequent commits. We don't want them to get removed, just like we don't want the tc-vcs caches to get removed. Review commit: https://reviewboard.mozilla.org/r/65350/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65350/
https://hg.mozilla.org/mozilla-unified contains heads from all the major Firefox repos. In addition, it is encoded on the server in such a way that it is several hundred megabytes smaller despite containing 30,000+ more changesets. This means faster clones and faster operations. Cloning this repo does require a new version of Mercurial. So if any TC tasks are using an ancient Mercurial - one that is vulnerable to known CVE issues in fact - this will flush them out. Review commit: https://reviewboard.mozilla.org/r/65352/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65352/
The robustcheckout Mercurial extension does a clone+checkout optimally. Read the bug for more on it. robustcheckout is already used by mozharness automation. It has resulted in a significant reduction in I/O usage and utilization in automation. This commit replaces tc-vcs with the robustcheckout equivalent. We replace the existing tc-vcs scope and cache with a new one. Because Dustin and I are paranoid, we maintain separate caches per SCM level - even though we could arguably share the same cache. Defense in depth. Robustcheckout (when used with --sharebase) pools storage for related repos automatically. i.e. changesets from inbound and central will be in the same store. This means you likely only have one copy of each changeset per cache. This can result in significant space savings. And, since there are fewer copies floating around, hg.mozilla.org and various network appliances are working less too! Since tc-vcs is no longer used, we stop it from being installed. Review commit: https://reviewboard.mozilla.org/r/65354/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65354/
We should be using the latest stable release of Mercurial in automation to get performance gains and to squash bugs. This commit installs Mercurial 3.8.4 and the robustcheckout extension on the centos6-build image. It also installs a global hgrc similar to the one we used for the lint image. The centos6-build-upd image has been updated to bump image versions. Review commit: https://reviewboard.mozilla.org/r/65356/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65356/
Like we've done for the lint tasks, we convert desktop-build to use the robustcheckout extension instead of tc-vcs. The logic is slightly more complicated because of various environment variables. Some unused environment variables have been removed. Review commit: https://reviewboard.mozilla.org/r/65358/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65358/
Comment on attachment 8772608 [details] Bug 1247168 - Use robustcheckout in desktop-build task; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/1-2/
Before I update this image to change VCS interactions, I thought I'd bloat scope to update some Python packaging versions. I dropped the virtualenv package because AFAICT nobody is using it. In fact, there are a number of things in the decision image I don't think should belong. Curious. Review commit: https://reviewboard.mozilla.org/r/65374/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65374/
Attachment #8772622 - Flags: review?(dustin)
Attachment #8772623 - Flags: review?(dustin)
Similar to other commits. We uprade to Mercurial 3.8.4. We install robustcheckout extension and configure the /etc/mercurial/hgrc file. We change the task to use `hg robustcheckout` instead of tc-vcs. Review commit: https://reviewboard.mozilla.org/r/65376/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65376/
Comment on attachment 8772603 [details] Bug 1247168 - Install Mercurial 3.8.4 in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/1-2/
Comment on attachment 8772604 [details] Bug 1247168 - Don't remove hg-shared scopes; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/1-2/
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/1-2/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/1-2/
Comment on attachment 8772607 [details] Bug 1247168 - Update Mercurial version and add robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/1-2/
Comment on attachment 8772608 [details] Bug 1247168 - Use robustcheckout in desktop-build task; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/2-3/
https://reviewboard.mozilla.org/r/65352/#review62404 This change didn't do what I wanted everywhere: Try jobs are still using mozilla-central as the base repo. I guess I'll need to seek out and destroy more references...
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/2-3/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/2-3/
Comment on attachment 8772607 [details] Bug 1247168 - Update Mercurial version and add robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/2-3/
Comment on attachment 8772608 [details] Bug 1247168 - Use robustcheckout in desktop-build task; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/3-4/
Comment on attachment 8772622 [details] Bug 1247168 - Upgrade to pip 8.1.2 and setuptools 24.0.3 in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65374/diff/1-2/
Comment on attachment 8772623 [details] Bug 1247168 - Use robustcheckout in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/1-2/
Attachment #8772602 - Flags: review?(dustin) → review+
Attachment #8772603 - Flags: review?(dustin) → review+
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; https://reviewboard.mozilla.org/r/65352/#review62544 This is great!
Attachment #8772605 - Flags: review?(dustin) → review+
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; https://reviewboard.mozilla.org/r/65354/#review62546
Attachment #8772606 - Flags: review?(dustin) → review+
https://reviewboard.mozilla.org/r/65348/#review62548 ::: testing/docker/lint/system-setup.sh:73 (Diff revision 2) > npm install -g taskcluster-vcs-v2.3.12.tar.gz > > > ### > # Flake8 Setup > ### This comment is no longer accurate. Either change it to reflect that this is installing python packages in bulk, or use a separate pip invocation to install mercurial. (sorry I missed this on the first go-round)
Comment on attachment 8772607 [details] Bug 1247168 - Update Mercurial version and add robustcheckout; https://reviewboard.mozilla.org/r/65356/#review62552 ::: testing/docker/centos6-build-upd/VERSION:1 (Diff revision 3) > -0.1.6.20160329195300 > +0.1.7.201607192200 You'll also need to change desktop-build/Dockerfile to refer to this version ::: testing/docker/centos6-build/VERSION:1 (Diff revision 3) > -0.1.6 > +0.1.7 :egoktas just released 0.1.7 to install something else, so this will need to be 0.1.8 after rebase (bug 1272629)
Attachment #8772607 - Flags: review?(dustin) → review-
Comment on attachment 8772622 [details] Bug 1247168 - Upgrade to pip 8.1.2 and setuptools 24.0.3 in decision image; https://reviewboard.mozilla.org/r/65374/#review62562 This image hasn't gotten a lot of attention, so thanks for updating it!
Attachment #8772622 - Flags: review?(dustin) → review+
Attachment #8772623 - Flags: review?(dustin) → review+
Comment on attachment 8772608 [details] Bug 1247168 - Use robustcheckout in desktop-build task; https://reviewboard.mozilla.org/r/65358/#review62560 ::: testing/docker/desktop-build/Dockerfile:1 (Diff revision 4) > -FROM taskcluster/centos6-build-upd:0.1.6.20160329195300 > +FROM taskcluster/centos6-build-upd:0.1.7.201607192200 Ah, I see you've added this in the next cset. I would perfer to see all of these numbers change in the same cset (well, honestly, I'd like to not use version numbers..) ::: testing/docker/desktop-build/bin/checkout-sources.sh:29 (Diff revision 4) > > : WORKSPACE ${WORKSPACE:=/home/worker/workspace} > > set -v > > +checkoutargs = "--purge --sharebase /home/worker/hg-shared" bash: checkoutargs: command not found (spaces aren't allowed around `=`) ::: testing/docker/desktop-build/bin/checkout-sources.sh:42 (Diff revision 4) > - ln -s $WORKSPACE/build/tools build/tools > fi > + > + mkdir -p build ${WORKSPACE}/build > + hg robustcheckout ${args} ${TOOLS_HEAD_REPOSITORY} ${WORKSPACE}/build/tools > + hg robustcheckout ${args} ${TOOLS_HEAD_REPOSITORY} build/tools why not use the symbolic link? ::: testing/docker/desktop-build/bin/checkout-sources.sh:52 (Diff revision 4) > HEAD_REPO="${extra_repo}_HEAD_REPOSITORY" > HEAD_REV="${extra_repo}_HEAD_REV" > HEAD_REF="${extra_repo}_HEAD_REF" > DEST_DIR="${extra_repo}_DEST_DIR" > > tc-vcs checkout ${!DEST_DIR} ${!BASE_REPO} ${!HEAD_REPO} ${!HEAD_REV} ${!HEAD_REF} This is used for android partner builds, for which check with :jlund -- in `taskcluster/ci/legacy/tasks/builds/android_api_15_partner_sample1.yml` they are git repositories. I'd hate to leave this one use of tc-vcs hanging around :(
Attachment #8772608 - Flags: review?(dustin) → review-
We're currently running Ubuntu 14.04 in the decision image. While still in LTS support, 16.04 ships with a modern, properly configured Python 2.7. So we upgrade to 16.04 and drop the install of Python from source because it is no longer needed. This is part 1 of a larger refactor to this image. Review commit: https://reviewboard.mozilla.org/r/65604/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65604/
Attachment #8772623 - Attachment description: Bug 1247168 - Use robustcheckout in decision task; → Bug 1247168 - Use robustcheckout in decision image;
Attachment #8772925 - Flags: review?(dustin)
Attachment #8772926 - Flags: review?(dustin)
Attachment #8772927 - Flags: review?(dustin)
Attachment #8772928 - Flags: review?(dustin)
Attachment #8772929 - Flags: review?(dustin)
Attachment #8772930 - Flags: review?(dustin)
Attachment #8772931 - Flags: review?(dustin)
Attachment #8772607 - Flags: review- → review?(dustin)
Attachment #8772608 - Flags: review- → review?(dustin)
We change the installation of Mercurial from via peep to .deb files in tooltool. The .deb files were produced by Mercurial's built-in make targets to produce .deb packages. As part of this, we upgrade to Mercurial 3.8.4. It should be a drop-in replacement. Since we no longer use peep, we stop installing it and pip/setuptools since they were only needed to run peep. Review commit: https://reviewboard.mozilla.org/r/65608/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65608/
I'm not sure why the decision image has so many packages installed. Most of them don't need to exist because the decision image only needs to obtain a copy of the Firefox repo and run `mach`. This doesn't require any build system per se. And all the Python dependencies are vendored in the Firefox repo. All we need is a Python 2.7 interpreter. This change reduces the decision image size from ~700 MB to ~300 MB. Review commit: https://reviewboard.mozilla.org/r/65610/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65610/
Previously, we were downloading tooltool.py from github.com. There were 2 problems with this. First, there is a dependency on a 3rd party service. While the Docker image should be cached, as a matter of principle we don't like hitting 3rd party services in our automation. The file is small enough, so we just vendor it. Second - and more importantly - we weren't validating the integrity of the downloaded file. This means that a MiTM could possibly alter the content of the file without us knowing (they would need a valid CA but since the Ubuntu trusted CA bundle contains a lot of CAs from e.g. governments, this isn't out of the question). Vendoring the file removes this risk. Third, behavior wasn't deterministic over time. We were always downloading the "master" revision of the file. I like determinism over time. Vendoring makes things deterministic. Since curl was only used to download tooltool.py, we remove that package install, which also removes a bunch of dependency packages and cuts down the image size by several megabytes. Review commit: https://reviewboard.mozilla.org/r/65612/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65612/
web.cacerts matches what the Ubuntu package does by default. [progress] changes are to make output in TaskCluster logs less spammy (only 1 update per second instead of up to 10). The robustcheckout extension will be used in a subsequent commit to handle repository checkouts. Review commit: https://reviewboard.mozilla.org/r/65614/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65614/
Changes to the decision Docker image have been compelted. We're ready to use the new image. We tag the image, update version references, and change the task caches so the new Mercurial pooled storage from the robustcheckout extension is used. Review commit: https://reviewboard.mozilla.org/r/65616/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65616/
Comment on attachment 8772623 [details] Bug 1247168 - Use robustcheckout in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/2-3/
Comment on attachment 8772602 [details] Bug 1247168 - Reformat requirements.txt; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/1-2/
Comment on attachment 8772603 [details] Bug 1247168 - Install Mercurial 3.8.4 in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/2-3/
Comment on attachment 8772604 [details] Bug 1247168 - Don't remove hg-shared scopes; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/2-3/
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/3-4/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/3-4/
Comment on attachment 8772607 [details] Bug 1247168 - Update Mercurial version and add robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/3-4/
Comment on attachment 8772608 [details] Bug 1247168 - Use robustcheckout in desktop-build task; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/4-5/
Attachment #8772622 - Attachment is obsolete: true
Comment on attachment 8772931 [details] Bug 1247168 - Tag and use version 0.1.2 of the decision image; https://reviewboard.mozilla.org/r/65616/#review62606 Let me know when you want me to build this locally and push it.
Attachment #8772931 - Flags: review?(dustin) → review+
Comment on attachment 8772930 [details] Bug 1247168 - Configure Mercurial and install global hgrc; https://reviewboard.mozilla.org/r/65614/#review62608
Attachment #8772930 - Flags: review?(dustin) → review+
Comment on attachment 8772926 [details] Bug 1247168 - Don't install virtualenv in the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65606/diff/1-2/
Attachment #8772926 - Attachment description: Bug 1247168 - Don't install virtualenv in the decision task; → Bug 1247168 - Don't install virtualenv in the decision image;
Comment on attachment 8772927 [details] Bug 1247168 - Install Mercurial 3.8.4 from .deb files; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65608/diff/1-2/
Comment on attachment 8772928 [details] Bug 1247168 - Stop installing most system packages; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65610/diff/1-2/
Comment on attachment 8772929 [details] Bug 1247168 - Vendor tooltool.py; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65612/diff/1-2/
Comment on attachment 8772930 [details] Bug 1247168 - Configure Mercurial and install global hgrc; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65614/diff/1-2/
Comment on attachment 8772623 [details] Bug 1247168 - Use robustcheckout in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/3-4/
Comment on attachment 8772931 [details] Bug 1247168 - Tag and use version 0.1.2 of the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/1-2/
Comment on attachment 8772602 [details] Bug 1247168 - Reformat requirements.txt; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/2-3/
Comment on attachment 8772603 [details] Bug 1247168 - Install Mercurial 3.8.4 in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/3-4/
Comment on attachment 8772604 [details] Bug 1247168 - Don't remove hg-shared scopes; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/3-4/
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/4-5/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/4-5/
Comment on attachment 8772607 [details] Bug 1247168 - Update Mercurial version and add robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/4-5/
Comment on attachment 8772608 [details] Bug 1247168 - Use robustcheckout in desktop-build task; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/5-6/
Comment on attachment 8772929 [details] Bug 1247168 - Vendor tooltool.py; https://reviewboard.mozilla.org/r/65612/#review62610 We should really vendor tooltool.py in-tree in general, rather than depending on it being in /builds or ~/build or whatever.
Attachment #8772929 - Flags: review?(dustin) → review+
Attachment #8772928 - Flags: review?(dustin) → review+
Comment on attachment 8772927 [details] Bug 1247168 - Install Mercurial 3.8.4 from .deb files; https://reviewboard.mozilla.org/r/65608/#review62616 ::: testing/docker/decision/system-setup.sh:78 (Diff revision 2) > - "unpack": false > } > ] > EOF > > -cd $BUILD > +dpkg -i mercurial-common_3.8.4_all.deb mercurial_3.8.4_amd64.deb Why not install from PyPi?
Attachment #8772927 - Flags: review?(dustin) → review+
Attachment #8772926 - Flags: review?(dustin) → review+
Comment on attachment 8772925 [details] Bug 1247168 - Change decision image to Ubuntu 16.04; https://reviewboard.mozilla.org/r/65604/#review62620
Attachment #8772925 - Flags: review?(dustin) → review+
https://reviewboard.mozilla.org/r/65608/#review62616 > Why not install from PyPi? Because this requires having a compiler installed. I'd rather avoid the complexity (and bloat).
Comment on attachment 8772604 [details] Bug 1247168 - Don't remove hg-shared scopes; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/4-5/
Comment on attachment 8772925 [details] Bug 1247168 - Change decision image to Ubuntu 16.04; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65604/diff/1-2/
Comment on attachment 8772926 [details] Bug 1247168 - Don't install virtualenv in the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65606/diff/2-3/
Comment on attachment 8772927 [details] Bug 1247168 - Install Mercurial 3.8.4 from .deb files; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65608/diff/2-3/
Comment on attachment 8772928 [details] Bug 1247168 - Stop installing most system packages; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65610/diff/2-3/
Comment on attachment 8772929 [details] Bug 1247168 - Vendor tooltool.py; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65612/diff/2-3/
Comment on attachment 8772930 [details] Bug 1247168 - Configure Mercurial and install global hgrc; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65614/diff/2-3/
Comment on attachment 8772623 [details] Bug 1247168 - Use robustcheckout in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/4-5/
Comment on attachment 8772931 [details] Bug 1247168 - Tag and use version 0.1.2 of the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/2-3/
Comment on attachment 8772602 [details] Bug 1247168 - Reformat requirements.txt; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/3-4/
Comment on attachment 8772603 [details] Bug 1247168 - Install Mercurial 3.8.4 in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/4-5/
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/5-6/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/5-6/
Comment on attachment 8772607 [details] Bug 1247168 - Update Mercurial version and add robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/5-6/
Comment on attachment 8772608 [details] Bug 1247168 - Use robustcheckout in desktop-build task; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/6-7/
Depends on: 1288215
Configuring the lint tasks to use a cache for the gecko checkout, I've seen flake8 tasks complete in as little as 11.1s (https://tools.taskcluster.net/task-inspector/#AMLtSM1kTW6JjMf3tdMC1g/0). You read that right: 11.1s. They were taking ~4 minutes before (most of that spent creating a VCS checkout).
The decision task configures a /home/worker/workspace cache. However, the command we run in the container references a "workspace" relative path. From logs in automation, it appears that PWD during execution is "/" because "workspace" is being resolved to "/workspace." The net result of this is we appear to be performing a VCS clone+checkout on every single task. This commit fixes the paths so our workspace cache is actually used. Review commit: https://reviewboard.mozilla.org/r/65724/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65724/
Attachment #8773094 - Flags: review?(dustin)
Attachment #8773095 - Flags: review?(dustin)
Attachment #8773096 - Flags: review?(dustin)
Previously, every lint task would have to create its own checkout. This was time consuming. The robustcheckout extension purges the working copy of *all* untracked and ignored files. It also restores modified files to their original state. In other words, as long as you trust Mercurial to go from revision X to revision Y, robustcheckout is as good as a fresh checkout. This commit adds a cache for the working directory checkout so lint tasks only have to effectively perform incremental `hg update` between task executions. This should make tasks spend a lot less time doing version control foo. On Try, time for flake8 tasks is currently hovering around 4 minutes. After this change, I've seen tasks finish as quickly as 25s! Review commit: https://reviewboard.mozilla.org/r/65726/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65726/
Our changes to these images are done. We bump the versions to signal the creation of new images. Review commit: https://reviewboard.mozilla.org/r/65728/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/65728/
Comment on attachment 8772623 [details] Bug 1247168 - Use robustcheckout in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/5-6/
Comment on attachment 8772931 [details] Bug 1247168 - Tag and use version 0.1.2 of the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/3-4/
Comment on attachment 8772602 [details] Bug 1247168 - Reformat requirements.txt; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/4-5/
Comment on attachment 8772603 [details] Bug 1247168 - Install Mercurial 3.8.4 in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/5-6/
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/6-7/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/6-7/
Comment on attachment 8772607 [details] Bug 1247168 - Update Mercurial version and add robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65356/diff/6-7/
Comment on attachment 8772608 [details] Bug 1247168 - Use robustcheckout in desktop-build task; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65358/diff/7-8/
Status: NEW → ASSIGNED
https://reviewboard.mozilla.org/r/65358/#review62560 > This is used for android partner builds, for which check with :jlund -- in `taskcluster/ci/legacy/tasks/builds/android_api_15_partner_sample1.yml` they are git repositories. I'd hate to leave this one use of tc-vcs hanging around :( I agree: I'd love to kill it. It looks like this task has its origins in bug 1163084 in September/October 2015. It isn't obvious that this task runs at all. I'm tempted to kill it. Or at least break it by removing tc-vcs.
Jordan: can you please weigh in at https://reviewboard.mozilla.org/r/65358/#review62560
Flags: needinfo?(jlund)
Comment on attachment 8773094 [details] Bug 1247168 - Actually use workspace cache in decision task; https://reviewboard.mozilla.org/r/65724/#review62864
Attachment #8773094 - Flags: review?(dustin) → review+
https://reviewboard.mozilla.org/r/65726/#review62866 If `hg robustcheckout` is deleting everything, does that mean it's deleting the objdir too? ::: taskcluster/ci/legacy/tasks/lint.yml:35 (Diff revision 1) > # Thirty minutes should be enough for lint checks > maxRunTime: 1800 > > cache: > level-{{level}}-hg-shared: '/home/worker/hg-shared' > + level-{{level}}-checkouts: '/home/worker/checkouts' Why not workspace, to match other tasks?
Comment on attachment 8773095 [details] Bug 1247168 - Use a cache for repo checkout in lint tasks; https://reviewboard.mozilla.org/r/65726/#review62920
Attachment #8773095 - Flags: review?(dustin) → review-
Comment on attachment 8773096 [details] Bug 1247168 - Bump centos6-build and centos6-build-upd image versions; https://reviewboard.mozilla.org/r/65728/#review62922
Attachment #8773096 - Flags: review?(dustin) → review+
Comment on attachment 8772608 [details] Bug 1247168 - Use robustcheckout in desktop-build task; syncing bz with mozreview..
Attachment #8772608 - Flags: review?(dustin) → review-
Comment on attachment 8772607 [details] Bug 1247168 - Update Mercurial version and add robustcheckout; syncing bz with mozreview..
Attachment #8772607 - Flags: review?(dustin) → review-
Depends on: 1288377
I'm going to drop the build images/tasks from this bug and punt to a follow-up: they are a bit more work than decision and lint and I like landing things.
https://reviewboard.mozilla.org/r/65358/#review62560 > I agree: I'd love to kill it. > > It looks like this task has its origins in bug 1163084 in September/October 2015. It isn't obvious that this task runs at all. I'm tempted to kill it. Or at least break it by removing tc-vcs. it's still used occasionally: https://bugzilla.mozilla.org/show_bug.cgi?id=1255119 I think #mobile were even using it for: https://bugzilla.mozilla.org/show_bug.cgi?id=1284582 It might not be needed for much longer but it would be great to still support this for now. Is there a work around? Can we just change this to a `git clone`. Does tc-vcs use git at all or does it just simply grab a tarball and our images don't even have git installed?
Flags: needinfo?(jlund)
git is installed -- tc-vcs is just a frontend for it (and does some tarball stuff too, yeah) Is there a bug representing the "not..needed for much longer"? If you can point to that, we can embed a pointer to it in a comment so the next person to wonder "why is this still here?" can either learn why, or learn that it can be removed.
In preparation for running tasks as the worker user. Review commit: https://reviewboard.mozilla.org/r/66210/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/66210/
Attachment #8772931 - Attachment description: Bug 1247168 - Tag and use version 0.1.1 of the decision image; → Bug 1247168 - Tag and use version 0.1.2 of the decision image;
Attachment #8773463 - Flags: review?(dustin)
Attachment #8773464 - Flags: review?(dustin)
Attachment #8773465 - Flags: review?(dustin)
Attachment #8773095 - Flags: review- → review?(dustin)
When we switch to use robustcheckout for version control foo, we'll also be taking the opportunity to have the decision and action tasks execute as the "worker" user. Since caches are mounted and owned by root and since tasks initially run as root, this makes defining the container command in YAML a bit difficult because we have to do some work as root then switch users and continue executing. Rather than shoehorning all that complicated logic into YAML, we introduce bash scripts that do it. These will be plugged into the task YAML when we formally switch the tasks to use the new Docker image. We provide one script for running Gecko decision tasks. We provide another for running action tasks. These are the two consumers of the decision image we care about. We also sneak in a change to add the executable bit to checkout-gecko. Review commit: https://reviewboard.mozilla.org/r/66212/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/66212/
The script will be used as the main command in task YAML files. It changes ownership of caches. Then switches to the "worker" user. Then performs a Gecko checkout. Then executes whatever command was requested via its arguments. Review commit: https://reviewboard.mozilla.org/r/66214/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/66214/
Comment on attachment 8772604 [details] Bug 1247168 - Don't remove hg-shared scopes; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/5-6/
Comment on attachment 8772925 [details] Bug 1247168 - Change decision image to Ubuntu 16.04; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65604/diff/2-3/
Comment on attachment 8772926 [details] Bug 1247168 - Don't install virtualenv in the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65606/diff/3-4/
Comment on attachment 8772927 [details] Bug 1247168 - Install Mercurial 3.8.4 from .deb files; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65608/diff/3-4/
Comment on attachment 8772928 [details] Bug 1247168 - Stop installing most system packages; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65610/diff/3-4/
Comment on attachment 8772929 [details] Bug 1247168 - Vendor tooltool.py; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65612/diff/3-4/
Comment on attachment 8772930 [details] Bug 1247168 - Configure Mercurial and install global hgrc; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65614/diff/3-4/
Comment on attachment 8773094 [details] Bug 1247168 - Actually use workspace cache in decision task; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65724/diff/1-2/
Comment on attachment 8772623 [details] Bug 1247168 - Use robustcheckout in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/6-7/
Comment on attachment 8772931 [details] Bug 1247168 - Tag and use version 0.1.2 of the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/4-5/
Comment on attachment 8772602 [details] Bug 1247168 - Reformat requirements.txt; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/5-6/
Comment on attachment 8772603 [details] Bug 1247168 - Install Mercurial 3.8.4 in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/6-7/
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/7-8/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/7-8/
Comment on attachment 8773095 [details] Bug 1247168 - Use a cache for repo checkout in lint tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/1-2/
Attachment #8772607 - Attachment is obsolete: true
Attachment #8773096 - Attachment is obsolete: true
Attachment #8772608 - Attachment is obsolete: true
Comment on attachment 8773464 [details] Bug 1247168 - Introduce scripts for running action and decision tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66212/diff/1-2/
Comment on attachment 8772931 [details] Bug 1247168 - Tag and use version 0.1.2 of the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/5-6/
Comment on attachment 8772602 [details] Bug 1247168 - Reformat requirements.txt; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/6-7/
Comment on attachment 8772603 [details] Bug 1247168 - Install Mercurial 3.8.4 in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/7-8/
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/8-9/
Comment on attachment 8773465 [details] Bug 1247168 - Add a script to perform a checkout then run a command; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/1-2/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/8-9/
Comment on attachment 8773095 [details] Bug 1247168 - Use a cache for repo checkout in lint tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/2-3/
Comment on attachment 8773465 [details] Bug 1247168 - Add a script to perform a checkout then run a command; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/2-3/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/9-10/
Comment on attachment 8773095 [details] Bug 1247168 - Use a cache for repo checkout in lint tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/3-4/
Comment on attachment 8773465 [details] Bug 1247168 - Add a script to perform a checkout then run a command; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/3-4/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/10-11/
Comment on attachment 8773095 [details] Bug 1247168 - Use a cache for repo checkout in lint tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/4-5/
A limitation of traditional `docker build` is that it only has access to files in the same directory as the Dockerfile. Typically, when you do `docker build`, Docker will create a tar archive of all files in the same directory as the Dockerfile and upload that to Docker and the image building process will have access to all files in the archive. Over a year ago, I realized you could write some code to create custom context archives and talk to the Docker build API directly to use your custom archive. I hacked some code into version-control-tools that parsed Dockerfiles for special syntax denoting extra paths from the source checkout to add to the context and proceed to add them to context archives. This commit essentially copied that code for use by taskgraph's built-in Docker image building. Using the syntax "# %include <path>" you are able to include paths or directories (relative from the top source directory root) in the generated context archive. Files add this way are available under the "topsrcdir/" path. The "lint" image has been changed to use this syntax to add in in-tree version of tooltool.py (instead of downloading from github.com). This eliminates a dependency on a third party service and increases security and determinism. Yay. In order to write tests, I had to make archiving deterministic. That's why we no longer use a single "tar.add()" for the Dockerfile directory. Instead, we obtain the list of files up front, sort them, then add with uid/gid set to 0, so uid/gid is consistent no matter what it is on the filesystem performing context creation. More determinism, yay. I would like to test this feature a bit more. However, the test environment for custom Docker image building doesn't currently facilitate custom source paths: it expects Docker files to be in $topsrcdir/testing/docker. If we add more functionality to this, we should definitely invest in writing better tests. Review commit: https://reviewboard.mozilla.org/r/66240/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/66240/
Attachment #8773543 - Flags: review?(dustin)
Comment on attachment 8773465 [details] Bug 1247168 - Add a script to perform a checkout then run a command; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/4-5/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/11-12/
Comment on attachment 8773095 [details] Bug 1247168 - Use a cache for repo checkout in lint tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/5-6/
Attachment #8773543 - Attachment is obsolete: true
Attachment #8773543 - Flags: review?(dustin)
Depends on: 1288567
Using our special Dockerfile syntax to include arbitrary files, we include the previously vendored tooltool.py file in the image build context and add it directly from there. No github.com communication needed. Review commit: https://reviewboard.mozilla.org/r/66566/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/66566/
Attachment #8773941 - Flags: review?(dustin)
Comment on attachment 8773463 [details] Bug 1247168 - Create worker user/group in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66210/diff/1-2/
Comment on attachment 8772604 [details] Bug 1247168 - Don't remove hg-shared scopes; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/6-7/
Comment on attachment 8772925 [details] Bug 1247168 - Change decision image to Ubuntu 16.04; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65604/diff/3-4/
Comment on attachment 8772926 [details] Bug 1247168 - Don't install virtualenv in the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65606/diff/4-5/
Comment on attachment 8772927 [details] Bug 1247168 - Install Mercurial 3.8.4 from .deb files; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65608/diff/4-5/
Comment on attachment 8772928 [details] Bug 1247168 - Stop installing most system packages; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65610/diff/4-5/
Comment on attachment 8772929 [details] Bug 1247168 - Vendor tooltool.py; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65612/diff/4-5/
Comment on attachment 8772930 [details] Bug 1247168 - Configure Mercurial and install global hgrc; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65614/diff/4-5/
Comment on attachment 8773094 [details] Bug 1247168 - Actually use workspace cache in decision task; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65724/diff/2-3/
Comment on attachment 8772623 [details] Bug 1247168 - Use robustcheckout in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/7-8/
Comment on attachment 8773464 [details] Bug 1247168 - Introduce scripts for running action and decision tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66212/diff/2-3/
Comment on attachment 8772931 [details] Bug 1247168 - Tag and use version 0.1.2 of the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/6-7/
Comment on attachment 8772602 [details] Bug 1247168 - Reformat requirements.txt; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/7-8/
Comment on attachment 8772603 [details] Bug 1247168 - Install Mercurial 3.8.4 in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/8-9/
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/9-10/
Comment on attachment 8773465 [details] Bug 1247168 - Add a script to perform a checkout then run a command; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/5-6/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/12-13/
Comment on attachment 8773095 [details] Bug 1247168 - Use a cache for repo checkout in lint tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/6-7/
Comment on attachment 8773465 [details] Bug 1247168 - Add a script to perform a checkout then run a command; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/6-7/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/13-14/
Comment on attachment 8773095 [details] Bug 1247168 - Use a cache for repo checkout in lint tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/7-8/
I rebased this on top of bug 1288567. dustin: you may want to look at interdiffs, as some commits you already r+d have changed a bit.
Comment on attachment 8773463 [details] Bug 1247168 - Create worker user/group in decision image; https://reviewboard.mozilla.org/r/66210/#review63646
Attachment #8773463 - Flags: review?(dustin) → review+
Blocks: 1259456
Blocks: 1255017
Comment on attachment 8773941 [details] Bug 1247168 - Use vendored tooltool in lint image; https://reviewboard.mozilla.org/r/66566/#review63672
Attachment #8773941 - Flags: review?(dustin) → review+
Comment on attachment 8773465 [details] Bug 1247168 - Add a script to perform a checkout then run a command; https://reviewboard.mozilla.org/r/66214/#review63676
Attachment #8773465 - Flags: review?(dustin) → review+
Attachment #8773095 - Flags: review?(dustin) → review+
Comment on attachment 8773464 [details] Bug 1247168 - Introduce scripts for running action and decision tasks; https://reviewboard.mozilla.org/r/66212/#review63682
Attachment #8773464 - Flags: review?(dustin) → review+
Comment on attachment 8773463 [details] Bug 1247168 - Create worker user/group in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66210/diff/2-3/
Comment on attachment 8772604 [details] Bug 1247168 - Don't remove hg-shared scopes; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65350/diff/7-8/
Comment on attachment 8772925 [details] Bug 1247168 - Change decision image to Ubuntu 16.04; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65604/diff/4-5/
Comment on attachment 8772926 [details] Bug 1247168 - Don't install virtualenv in the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65606/diff/5-6/
Comment on attachment 8772927 [details] Bug 1247168 - Install Mercurial 3.8.4 from .deb files; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65608/diff/5-6/
Comment on attachment 8772928 [details] Bug 1247168 - Stop installing most system packages; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65610/diff/5-6/
Comment on attachment 8772929 [details] Bug 1247168 - Vendor tooltool.py; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65612/diff/5-6/
Comment on attachment 8772930 [details] Bug 1247168 - Configure Mercurial and install global hgrc; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65614/diff/5-6/
Comment on attachment 8773094 [details] Bug 1247168 - Actually use workspace cache in decision task; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65724/diff/3-4/
Comment on attachment 8772623 [details] Bug 1247168 - Use robustcheckout in decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65376/diff/8-9/
Comment on attachment 8773464 [details] Bug 1247168 - Introduce scripts for running action and decision tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66212/diff/3-4/
Comment on attachment 8772931 [details] Bug 1247168 - Tag and use version 0.1.2 of the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/7-8/
Comment on attachment 8772602 [details] Bug 1247168 - Reformat requirements.txt; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/8-9/
Comment on attachment 8773941 [details] Bug 1247168 - Use vendored tooltool in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66566/diff/1-2/
Comment on attachment 8772603 [details] Bug 1247168 - Install Mercurial 3.8.4 in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/9-10/
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/10-11/
Comment on attachment 8773465 [details] Bug 1247168 - Add a script to perform a checkout then run a command; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/7-8/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/14-15/
Comment on attachment 8773095 [details] Bug 1247168 - Use a cache for repo checkout in lint tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/8-9/
Comment on attachment 8772931 [details] Bug 1247168 - Tag and use version 0.1.2 of the decision image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65616/diff/8-9/
Comment on attachment 8772602 [details] Bug 1247168 - Reformat requirements.txt; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65346/diff/9-10/
Comment on attachment 8773941 [details] Bug 1247168 - Use vendored tooltool in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66566/diff/2-3/
Comment on attachment 8772603 [details] Bug 1247168 - Install Mercurial 3.8.4 in lint image; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65348/diff/10-11/
Comment on attachment 8772605 [details] Bug 1247168 - Make mozilla-unified the base repository for Firefox; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65352/diff/11-12/
Comment on attachment 8773465 [details] Bug 1247168 - Add a script to perform a checkout then run a command; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/66214/diff/8-9/
Comment on attachment 8772606 [details] Bug 1247168 - Convert lint image and tasks to use robustcheckout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65354/diff/15-16/
Comment on attachment 8773095 [details] Bug 1247168 - Use a cache for repo checkout in lint tasks; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/65726/diff/9-10/
Pushed by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/085b1857a36f Create worker user/group in decision image; r=dustin https://hg.mozilla.org/integration/autoland/rev/fcc1771bbe23 Don't remove hg-shared scopes; r=dustin https://hg.mozilla.org/integration/autoland/rev/eb08ab6d2349 Change decision image to Ubuntu 16.04; r=dustin https://hg.mozilla.org/integration/autoland/rev/418fc99dfe92 Don't install virtualenv in the decision image; r=dustin https://hg.mozilla.org/integration/autoland/rev/4f1747b9ce4a Install Mercurial 3.8.4 from .deb files; r=dustin https://hg.mozilla.org/integration/autoland/rev/d3a1a822bbaf Stop installing most system packages; r=dustin https://hg.mozilla.org/integration/autoland/rev/846f7e94c1ff Vendor tooltool.py; r=dustin https://hg.mozilla.org/integration/autoland/rev/34a4e796fa4d Configure Mercurial and install global hgrc; r=dustin https://hg.mozilla.org/integration/autoland/rev/6c9ced9abf7e Actually use workspace cache in decision task; r=dustin https://hg.mozilla.org/integration/autoland/rev/7450495f5182 Use robustcheckout in decision image; r=dustin https://hg.mozilla.org/integration/autoland/rev/8b43386ca744 Introduce scripts for running action and decision tasks; r=dustin https://hg.mozilla.org/integration/autoland/rev/4bb063392974 Tag and use version 0.1.2 of the decision image; r=dustin https://hg.mozilla.org/integration/autoland/rev/3ad658f9ddc5 Reformat requirements.txt; r=dustin https://hg.mozilla.org/integration/autoland/rev/f80ee4875041 Use vendored tooltool in lint image; r=dustin https://hg.mozilla.org/integration/autoland/rev/2a31541145b5 Install Mercurial 3.8.4 in lint image; r=dustin https://hg.mozilla.org/integration/autoland/rev/cab3a2a82ac0 Make mozilla-unified the base repository for Firefox; r=dustin https://hg.mozilla.org/integration/autoland/rev/423fef62913f Add a script to perform a checkout then run a command; r=dustin https://hg.mozilla.org/integration/autoland/rev/b4c3e613fee8 Convert lint image and tasks to use robustcheckout; r=dustin https://hg.mozilla.org/integration/autoland/rev/722707a63810 Use a cache for repo checkout in lint tasks; r=dustin
https://public-artifacts.taskcluster.net/Lzzl8h9rSqOzYke2wNo3lA/0/public/logs/live_backing.log is a flake8 task running in 9.87s on the autoland repo. That's ideal conditions: the worker already had a cached working copy of the revision we requested to run, so essentially no file writes were performed. In contrast, before we weren't using cached working directories and had to obtain a fresh checkout on every run. The fastest flake8 task I saw before was in the ~3 minute range.
Blocks: 1289249
Blocks: 1289643
Blocks: 1298947
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: