1137278 - Speed up the Travis run

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P3)

Description

[Ed Morley [:emorley]]

The treeherder-service Travis run takes up to 15 mins to run, which is a pain to wait for. We're also going to add flake8 linting & if we do combine the UI and service repos, we'll also be running other tests too.

There are some suggestions of things we can do on:
http://docs.travis-ci.com/user/speeding-up-the-build/

Mainly:
* Caching dependencies: http://docs.travis-ci.com/user/caching/ . To do this without a paid account, apparently we need to use Travis' new container based infrastructure: http://docs.travis-ci.com/user/workers/container-based-infrastructure/
* Splitting the run up into multiple sub-jobs: http://docs.travis-ci.com/user/speeding-up-the-build/#Parallelizing-your-builds-across-virtual-machines
* Using "fast_finish" to make the job fail as soon as one of the sub-jobs fail: http://docs.travis-ci.com/user/build-configuration/#Fast-finishing

There are also things we can do:
* Bug 1133268 - Turn off code coverage checks for Travis test runs
* Remove the |pip install -r requirements/pure.txt| per bug 1070470 comment 10.

Comment 1

[Ed Morley [:emorley]]

Attachment: Speed up the Travis run

aiui, we won't see the speed win from caching until this hits master, since PRs use master's cache, which is currently non-existent.

Comment 2

[Cameron Dawson [:camd]]

sorry for the delay.  I'll review tomorrow.

Comment 3

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/533a656b494558b593a124475f2dc9c6cf71c1d7
Bug 1137278 - Defer import of thclient until vendor/ is in sys.path

The treeherder client is in the vendor directory, however that doesn't
get added to the sys.path until settings/base.py is loaded, so defer the
import until we need it.

https://github.com/mozilla/treeherder-service/commit/d3bd03a5f479fd9a0ea09e1d97f577cb8bf3dd05
Bug 1137278 - Travis: Don't pip install packages already in the repo

Dependencies in pure.txt are used directly from the in-repo vendor
directory, so we should not waste time installing them in the virtualenv
during Travis runs.

https://github.com/mozilla/treeherder-service/commit/c2bcb2182e556d4fac63e8cbea8381dd3cb442a8
Bug 1137278 - Travis: Allow jobs to run on the new container based infra

Travis have a newer container based stack that runs on EC2, that is both
faster & allows the use of extra features (such as dependency caching):
http://docs.travis-ci.com/user/workers/container-based-infrastructure/

https://github.com/mozilla/treeherder-service/commit/cb5600711f556b3ae2f3a9db9742bb4b5b606d7e
Bug 1137278 - Travis: Remove deprecated --use-mirrors pip option

https://github.com/mozilla/treeherder-service/commit/c896e9401ba758f4b519a23d28aa22af0af37cb6
Bug 1137278 - Travis: Switch on caching of pip installed dependencies

From:
http://docs.travis-ci.com/user/caching/#pip-cache

https://github.com/mozilla/treeherder-service/commit/37f998db4c134183f072007ad2579c4d9d863648
Bug 1137278 - Travis: Move the Cython build step to before_script

The install step is usually only used for dependency installation.
(The Travis build cycle is install -> before_script -> script)

Comment 4

[Ed Morley [:emorley]]

Comment on attachment 8570039 [details] [review]
Speed up the Travis run

This is checked in & has helped to a certain extent, but the main win will be from caching the pip install - however I'm seeing:

Setting up build cache
$ export CASHER_DIR=$HOME/.casher
$ Installing caching utilities
attempting to download cache archive
could not download cache

The docs imply this should just work, but there may be some additional paths that need setting up.

Comment 5

[Ed Morley [:emorley]]

Attachment: Cache the .cache/pip/ directory explicitly

I'm just going to land this without review, since it's fairly specific & the Travis run itself (once on master) should make it clear whether it works or not.

Comment 6

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/da55cc3a1daf8fe3a33464b46a4241c8d5a4d1db
Bug 1137278 - Travis: Cache the .cache/pip/ directory explicitly

Travis |cache: pip| should do this for us, except it only works if we
don't set our own install step, due to:
https://github.com/travis-ci/travis-ci/issues/3239

Instead, let's manually specify the directory to cache.

Comment 7

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-ui

https://github.com/mozilla/treeherder-ui/commit/30a08093b502096e49ef6b85d648a2cf276ba4c5
Bug 1137278 - Travis: Allow jobs to run on the new container based infra

Travis have a newer container based stack that runs on EC2, that is both
faster & allows the use of extra features (such as dependency caching):
http://docs.travis-ci.com/user/workers/container-based-infrastructure/

Comment 8

[Ed Morley [:emorley]]

Attachment: Cache the virtualenv rather than the pip cache

Comment 9

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/3e3bd7c90894dd59622dd3bc81bf5ed4ec8634ef
Bug 1137278 - Travis: Re-use the virtualenv rather than the pip cache

Whilst making Travis store the pip cache saves us from having to
download the packages from PyPI each time, the vast majority of time
spent during Travis run setup is compiling the packages that contain
binary components. As such, let's try caching the virtualenv itself so
we don't have to repeat this each time. We no longer get Travis to
store the pip cache since it will only be used when the virtualenv is
first created & will bloat the Travis cache archive that has to be
downloaded from AWS at the start of each run.

Comment 10

[Ed Morley [:emorley]]

Travis runtimes...

Before this bug: 10-15 mins (of which ~3.75-4.75 mins installing packages)
After first PR: 7-9 mins (~2.25-2.5 mins installing packages)
After virtualenv caching (if cache populated): 5 mins (2-3 seconds installing packages \o/)

I'd call that done :-)

If the cached virtualenv ever gets in a pickle, we can clear it via the webui at:
https://travis-ci.org/mozilla/treeherder-service/caches
(Available under the settings cog menu)

Comment 11

[Ed Morley [:emorley]]

Also worth noting that once we have bug 1143033 the total overhead (repo clone, virtualenv creation, cache download from AWS, package installation check, checking to see if the cache has changed and so if it needs uploading to AWS) is only 30 seconds, so it's actually now viable to chunk the run into smaller parts (per http://docs.travis-ci.com/user/speeding-up-the-build/#Parallelizing-your-builds-across-virtual-machines).

Comment 12

[Ed Morley [:emorley]]

The Travis runtime is currently being lengthened by 60s (~20%) since peep re-downloaded ZIP archives each time, and in the case of datasource, for some reason causes Travis to think a file has changed in the cache, so causes it to re-upload to S3 at the end of the run. I've filed an issue against peep to see if the UX can be improved for URL-specified dependencies:
https://github.com/erikrose/peep/issues/91

Comment 13

[Ed Morley [:emorley]]

(By ZIP archives I mean the ones specified by a URL in the requirements file, rather than a Pypi version number)

Comment 14

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/9e4bf710411c569acbdabda22452f3b473723d13
Bug 1137278 - Travis: Allow jobs to run on the new container based infra

Travis have a newer container based stack that runs on EC2, that is both
faster & allows the use of extra features (such as dependency caching):
http://docs.travis-ci.com/user/workers/container-based-infrastructure/