1389572 - Reduce the time taken between pushing to the production branch and the Heroku deploy completing

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, enhancement, P2)

Description

[Ed Morley [:emorley]]

Currently the time between a push being made to an auto-deploy branch, and the Heroku IRC bot reporting the deployment is at least 13 minutes, with occasional deploys (eg the last one this morning) taking up to 20 minutes. 

The breakdown is roughly as follows:
* Time for Travis run to complete: 8-9 minutes (15 minutes in the chronic 20 min deploy case today)
* Time for buildpack compile: ~5 minutes

The long pole for the Travis run is the JS chunk (the webpack parts).

Similarly, the long pole for the buildpack compile is surprise surprise the webpack parts.

It seems like reducing the time spent during the webpack/neutrino build would give an easy win, since we'll see twice the benefit.

Comment 1

[Ed Morley [:emorley]]

I found another easy win in the buildpack:
https://github.com/heroku/heroku-buildpack-python/pull/465

Comment 2

[Ed Morley [:emorley]]

Approx 40s (25-30%) of the Heroku build is spent generating the gzipped and Brotli versions of the static assets using WhiteNoise (via the CLI tool plus for the Django collectstatic assets, CompressedStaticFilesMixin).

This is due to:
1) WhiteNoise being slow at compression (eg not using multi-{thread,process}ing)
  -> Filed: https://github.com/evansd/whitenoise/issues/148
2) Django generating duplicate/intermediate versions of hashed files during collectstatic, meaning there are 2-3x as many files under `treeherder/static/` to compress as there should be.
  -> Filed for possible WN workaround: https://github.com/evansd/whitenoise/issues/147
  -> Started Django mailing list discussion: https://groups.google.com/forum/#!topic/django-developers/fmfQvuHBStk

Comment 3

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #2)
> 2) Django generating duplicate/intermediate versions of hashed files during collectstatic

Now that I've tracked down the cause more, I've filed some Django tickets:
https://code.djangoproject.com/ticket/28604
https://code.djangoproject.com/ticket/28607
https://code.djangoproject.com/ticket/28606

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/2980

Comment 5

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/4b4a7341c368f23699fd9e00601f115ce3554ea0
Bug 1389572 - Travis: Don't install custom yarn

Since I've adjusted the Travis images to always use the latest yarn
at the time the images are created, so it's not worth installing the
latest version at every runtime:
https://github.com/travis-ci/travis-cookbooks/pull/935

https://github.com/mozilla/treeherder/commit/e9696485927445dcc44a0137ce81f89e9a7dd9d5
Bug 1389572 - Travis: Skip Elasticsearch install if version matches

Since after my update to the Travis images, the pre-installed version
of Elasticsearch matches the one we're looking for, meaning the
download/install can be skipped, saving time.

Comment 6

[Ed Morley [:emorley]]

I've opened a Heroku support ticket with some feature requests for making tracking down the cause of slow builds easier (eg prefixing logs with timestamps, adding the overall duration to the app activity page entry etc):
https://help.heroku.com/tickets/556127

Comment 7

[Ed Morley [:emorley]]

Nice tool for plotting Travis build times over time:
https://cmdcolin.github.io/travigraphjs/?repo=mozilla/treeherder&start=17000&end=20000

Comment 8

[Ed Morley [:emorley]]

Note to self:

When https://github.com/heroku/heroku-buildpack-nodejs/pull/519 gets deployed, we'll need to set `YARN_PRODUCTION=true` to prevent it from regressing our compile times.

Comment 9

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #8)
> Note to self:
> 
> When https://github.com/heroku/heroku-buildpack-nodejs/pull/519 gets
> deployed, we'll need to set `YARN_PRODUCTION=true` to prevent it from
> regressing our compile times.

Heroku rolled out that change today, so I set `YARN_PRODUCTION=true` on prototype/stage, however it hasn't prevented compile times from regressing.

I've filed:
https://github.com/heroku/heroku-buildpack-nodejs/issues/526

Comment 10

[Ed Morley [:emorley]]

Heroku now bundle a Node.js metrics plugin in the slug when using the nodejs buildpack, which we have no need for, since we have no runtime node.js parts. I've set `HEROKU_SKIP_NODE_PLUGIN=true` on {dev,stage,prod} to skip the installation of this plugin.

See:
https://github.com/heroku/heroku-buildpack-nodejs/blob/6ea576cf5a1d31031b4790391f181a0cd68126fb/lib/plugin.sh#L20

Comment 11

[Ed Morley [:emorley]]

The one remaining piece that would help here is adding support to WhiteNoise for parallel Brotli/gzip compression. I have a WIP for it but haven't had time to finish it up. Either way it's filed upstream here:
https://github.com/evansd/whitenoise/issues/148

As such I'm going to close this out for now, given that the WhiteNoise part and bug 1504996 are more longer term.