1365567 - Update Heroku to the Heroku-16 stack based on Ubuntu 16.04

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, enhancement, P3)

Description

[Ed Morley [:emorley]]

On Heroku we're currently using the cedar-14 stack, based on Ubuntu 14.04:
https://devcenter.heroku.com/articles/cedar-14-stack

Heroku now have an Ubuntu 16.04 based stack (called Heroku-16), which was in private alpha in December, beta in March and GA 11th April. They've even already made it the default stack for new apps, as of 8th May.

This new stack has a number of advantages, such as:
* smaller image (so faster dyno start times)
* proper docker image that we can use for a docker based development environment with fewer hacks required
* newer system packages

See:
https://blog.heroku.com/heroku-16-is-generally-available

Bug 1363444 switched the Vagrant environment from 14.04 to 16.04, so the next step is to do so on prototype (then stage, then prod).

Travis still doesn't yet have an Ubuntu 16.04 environment (https://github.com/travis-ci/travis-ci/issues/5821), however I don't think it's worth blocking, since there are much more significant differences between Travis and production other than this, and by doing this bug and bug 1169263 we will actually increase parity between local development and production significantly.

Comment 1

[Ed Morley [:emorley]]

One issue I found with the Heroku-16 stack was that the Python archives used by the buildpack contain superfluous files, for which I filed:
https://github.com/heroku/heroku-buildpack-python/issues/379

Whilst debugging that, I encountered some issues with the upstream release tool they use called bob-builder, for which I've opened a PR:
https://github.com/kennethreitz/bob-builder/issues?q=is%3Aissue%20author%3Aedmorley%20
https://github.com/kennethreitz/bob-builder/pull/28

Comment 2

[Ed Morley [:emorley]]

I've updated the Python buildpack to use the new version of bob-builder:
https://github.com/heroku/heroku-buildpack-python/pull/387
https://github.com/heroku/heroku-buildpack-python/pull/399

...so once Kenneth re-does the Python binaries using it, we should be good to go.

Comment 3

[Ed Morley [:emorley]]

I've scanned the Python archives and found the ones that need to be regenerated, and asked Kenneth to update them:
https://github.com/heroku/heroku-buildpack-python/issues/379#issuecomment-307834594

Comment 4

[Ed Morley [:emorley]]

I believe the last blocker is Heroku-16 support for memcachier-tls-buildpack, for which I've filed:
https://github.com/memcachier/memcachier-tls-buildpack/issues/13

Comment 5

[Ed Morley [:emorley]]

I found some other issues in the Heroku images which I've fixed upstream and added Travis testing:
https://github.com/heroku/stack-images/pull/61
https://github.com/heroku/stack-images/pull/59
https://github.com/heroku/stack-images/pull/63

Comment 6

[Ed Morley [:emorley]]

I have a PR in bug 1381151 to add Heroku-16 support to the memcachier buildpack. Until that's merged we can point the Heroku instances at my fork instead, like so:

$ thd buildpacks:set -i 1 https://github.com/edmorley/memcachier-tls-buildpack.git#heroku16-support
Buildpack set. Next release on treeherder-prototype will use:
  1. https://github.com/edmorley/memcachier-tls-buildpack.git#heroku16-support
  2. heroku/nodejs
  3. heroku/python

Initial test of Heroku-16 on prototype:

$ thd stack:set heroku-16
Stack set. Next release on treeherder-prototype will use heroku-16.

I've then performed a manual GitHub deploy using https://dashboard.heroku.com/apps/treeherder-prototype/deploy/github, since not possible via the CLI.

Build log:
https://dashboard.heroku.com/apps/treeherder-prototype/activity/builds/38778006-630f-44e4-b105-f918b78c2f44

Release log:
https://dashboard.heroku.com/apps/treeherder-prototype/activity/releases/ef1b0139-6480-4380-a8c8-49cacb9dc52a

Whilst everything succeeded, I did notice that the nodejs buildpack doesn't clear the cache after changing stacks (unlike the python buildpack), for which I've filed:
https://github.com/heroku/heroku-buildpack-nodejs/issues/445

For now, just in case I've cleared the cache manually using:
https://github.com/heroku/heroku-repo#purge-cache

And rebuilt:
https://dashboard.heroku.com/apps/treeherder-prototype/activity/builds/19c9f181-64ce-4108-9a60-24b2f3256b60

Everything looks fine, but I'll check again after a few days.

Comment 7

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #6)
> Whilst everything succeeded, I did notice that the nodejs buildpack doesn't
> clear the cache after changing stacks (unlike the python buildpack)

I've opened a PR to fix this:
https://github.com/heroku/heroku-buildpack-nodejs/pull/446

Comment 8

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:heroku-16 > mozilla:master

Comment 9

[James Graham [:jgraham]]

FWIW if there are docker images, we could use Docker on travis for testing. I guess it would add significant overhead though, so might not be worthwhile.

Comment 10

[Ed Morley [:emorley]]

That's the plan :-)

(The overhead is not as bad as you'd think, since we already have to do so much custom installation in the Travis config, that downloading a Docker image is actually pretty comparable - particularly if we use the the new `cache-from` combined with Docker Hub automated build. ie: both Travis and the local development environment can just download the bits from Docker Hub and have to do minimal work locally).

Comment 11

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/beb4d6740e5adad82ce1b38c744d6665b8df7b49
Bug 1365567 - Heroku: Don't vendor libmysqlclient on Heroku-16

Since the OS package present in the Heroku-16 image is now new enough
(v5.7.x) that it's not affected by the MITM issue. See:
https://devcenter.heroku.com/articles/stack-packages

Comment 12

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #7)
> I've opened a PR to fix this:
> https://github.com/heroku/heroku-buildpack-nodejs/pull/446

And also fixed:
https://github.com/heroku/heroku-buildpack-nodejs/pull/448
https://github.com/heroku/heroku-buildpack-nodejs/pull/447

Comment 13

[Ed Morley [:emorley]]

I'm still waiting on:
https://github.com/heroku/heroku-buildpack-python/issues/379#issuecomment-316784849

...however I can just purge the cache again once that's done, so it needn't block.

I've switched stage now:

$ ths repo:purge_cache
...

$ ths stack:set heroku-16
Stack set. Next release on treeherder-stage will use heroku-16.

+ manual github deploy via :
https://dashboard.heroku.com/apps/treeherder-stage/deploy/github

Build log:
https://dashboard.heroku.com/apps/treeherder-stage/activity/builds/7427dfb3-72f3-409d-86db-d533d9c5e404

Release log:
https://dashboard.heroku.com/apps/treeherder-stage/activity/releases/97d11794-7199-4031-ac16-9ee289284033

Comment 14

[Ed Morley [:emorley]]

Prod:

$ thp repo:purge_cache
...
$ thp stack:set heroku-16
Stack set. Next release on treeherder-prod will use heroku-16.

Build log:
https://dashboard.heroku.com/apps/treeherder-prod/activity/builds/028a04f0-5e06-4bc7-9ab7-82ef765509da

Release log:
https://dashboard.heroku.com/apps/treeherder-prod/activity/releases/12f06183-1dc5-4bf3-b861-28f046bf25d3

Comment 15

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:rm-vendor-libmysqlclient > mozilla:master

Comment 16

[Ed Morley [:emorley]]

I'll hold off merging that PR until I'm back.

At the moment (ie without the PR removing things), rolling back to cedar-14 is just a case of:
$ heroku stack:set cedar-14 --app <APP-NAME>

...then press deploy on the web dashboard.

Comment 17

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/2ae5c579599817d1d1a295bbfd51826bf1a04dfc
Bug 1365567 - Remove unused libmysqlclient vendoring support

All environments are now using the native MySQL 5.7 libmysqlclient
library, so the custom vendoring script is unused. The tests checking
that the library isn't vulnerable aren't needed any more, since
heroku-16 doesn't even have that package installed (it's not available
on Xenial).