1219889 - Speed up the Travis run by splitting it up

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P3)

Description

[Ed Morley [:emorley]]

Travis currently runs all tests as a single job. Since we need mysql 5.6 for the python unit tests, this job has to run on non-container infra which doesn't support dependency caching (costing us 2.5 mins) & also takes 30-60s longer to start up (since they're whole VMs rather than just containers - this bootup time isn't included in the durations shown in the dashboard).

By splitting the job up into several sub-jobs, we can both run tests in parallel & also only have to use the non-container infra for one of the pieces, not all of them.

Comment 1

[Ed Morley [:emorley]]

Attachment: Speed up the Travis run by splitting it up

Time until result... (includes "hidden" non-container boot time)

Before:
* linters/JS tests: ~3.5 mins
* python tests: ~7.5 mins
* grunt build: ~8 mins

After:
* linters: 23s
* JS tests/grunt build: 55s
* python tests: 5 min 40s

As an added bonus, failing the linters tests will now not stop all of the other tests from running.

It's also now easy to split the python job into two and run a subset of directories in each, should we wish.

Comment 2

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #1)
> * python tests: 5 min 40s

This should read:

* python tests: ~6.25 mins

Comment 3

[William Lachance (:wlach)]

Comment on attachment 8680853 [details] [review]
Speed up the Travis run by splitting it up

Looks good to me, looking forward to the improvements.

Comment 4

[Ed Morley [:emorley]]

I've added another commit that splits out the E2E and log parser tests, into another job. This makes the before/after:

Before:
* linters/JS tests: ~3.5 mins
* python tests: ~7.5 mins
* grunt build: ~8 mins

After:
* linters: 24s
* JS tests/grunt build: 55s
* python tests (main): ~4.75 mins
* python tests (e2e + log parser): ~3.5 mins

Comment 5

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/19a87884b322566258ce1a21d5b777f75579e9b6
Bug 1219889 - Travis: Move the job config under matrix.include

So that in a later commit we can split it into several jobs. See:
http://docs.travis-ci.com/user/customizing-the-build/#Explicity-Including-Builds

This is a no-op, done to make later commit diffs clearer.

https://github.com/mozilla/treeherder/commit/cfdd90efeb59fc73c420d593d62fc316fc5a386b
Bug 1219889 - Travis: Break out the linter steps to their own job

Since this job sets `sudo: false`, it will run on the container infra,
so be able to use caching. The `env: linters` is so that a nice label
appears in the Travis sub-job UI, eg see:
https://travis-ci.org/mozilla/treeherder/builds/87763293

We install the full set of node and python dependencies, since
hardcoding just the packages we need here could lead to the versions
getting out of sync with package.json/the pip requirements files. In
addition isort requires a populated virtualenv, and npm sucks if
passing package names directly (it reinstalls them even if installed).
It's worth noting that with the cache, full dependency installation
only takes 2.5s, so this is all moot.

This job (once it has populated the cache) now only takes 24s
end-to-end, whereas previously it would have been ~3.5 mins before the
linters steps had completed.

https://github.com/mozilla/treeherder/commit/99fd1617eced678d2a29ca6f69e42eea0150737d
Bug 1219889 - Travis: Break out the JS tests to their own job

Move the JS karma tests and the grunt build smoketest to their own job,
that runs on the container infra. The results are now available after
~50s rather than ~4mins previously. Ad added advantage is that since
we can now use `language: node_js`, we can pin to a specific version,
reducing the chance of breakage.

In addition, the python test run now no longer needs to npm install
(which isn't cacheable) and doesn't have the JS test/grunt build
runtime, saving ~120s.

https://github.com/mozilla/treeherder/commit/2bde0993dbfc55a0b4bace7526ff5da49ee888e4
Bug 1219889 - Travis: Use pip rather than peep for the python-tests job

The python-tests job is run on non-container infra so can't use caching.
Peep install takes 60s compared to 30s for pip (part hashing cost, part
peep's unavoidable design inefficiencies due to not being built into
pip), which is painful given that we do a fully install every time due
to lack of caching.

The linters step (which can use caching) is already testing the peep
install works, so we can fall back to plain pip for this job part to
save an extra 30s from the runtime.

This also adds the `language: python` key which was the missing
ingredient to getting a virtualenv set up for us. Now that we're not
using peep, it doesn't matter that the provided virtualenv is using
pip v7.x (which is incompatible with peep).

https://github.com/mozilla/treeherder/commit/7eb602e41a506f6f36104a8b167f51710c1b2d78
Bug 1219889 - Travis: Misc command syntax cleanup

https://github.com/mozilla/treeherder/commit/6c5721cee09dfbb497cedd2cbc3bc074051cdfed
Bug 1219889 - Travis: Split the log parser/E2E tests into their own job

Since they take approx 35% of the py.test run time. Ideally we'd move
the ETL tests instead of the E2E ones to this separate job (since they
are more related to the log parsing tests than some of the others, and
would make the split more even), but they currently break when run
standalone, so we'll do that once bug 1219922 is fixed.

Comment 6

[Ed Morley [:emorley]]

On the plus side, this means Heroku deploys will be faster, since they have to wait for the CI on master to turn green after the merge.