1174967 - Add a BMO Docker project with a more contributor-friendly workflow

Status: RESOLVED INCOMPLETE

(bugzilla.mozilla.org :: Developer Box, defect)

Description

[Ed Morley [:emorley]]

(This is a fresh start of bug 1171057, since that bug morphed a bit along the way)

The config in contrib/docker/ is a great start, however for new contributors wanting to test local changes, it can be confusing and time-consuming to get up and running.

Some of the current workflow & build-speed limitations are due to:

* Having to support TaskCluster, which for now only works with single containers. Therefore, many packages/services have to be crammed into one image, which is not the 'typical docker way' of doing things and makes everything more complicated, slows the build etc - since off the shelf base images cannot be used.

* Supporting experienced developer work-flows (eg multi-BMO installations), that are unnecessary for a new/occasional contributors. Whilst the productivity of core developers is critical - we're optimising for them at the expense of new contributors. An experienced dev would have no problems switching to a non-default config or knowing they need to run XYZ, whereas a new contributor very heavily relies on the out-of-box experience "just working".

As such, I think it makes sense to add an entirely new Docker config to the BMO repo, that is targeted specifically at new/occasional contributors and doesn't have to meet the above requirements. Ideally in the future (particularly when TaskCluster supports multi-containers) we can then consolidate the docker configs / reduce duplication by using Docker Compose's 'extend' feature or Dockerfiles that inherit from each other.

Issues this new Docker project should solve:

1) Initial builds are very slow / sometimes hang (at least here in the UK) due to problems with EPEL mirrors.
2) It's not obvious that out-of-the-box, the local BMO source has no effect on the build (it clones from Github master). Once you do realise, it's then not clear how to remedy this (there are no docs for this).
3) The Docker cache is often invalidated (eg due to changing environment variables at the top of the Dockerfile), causing full rebuilds. This plus #1, makes the experimentation necessary due to #2 very time consuming.
4) checksetup.pl/generate_bmo_data.pl are run during the image build, rather than on initial container startup. This means if you try to mount your local source into the container (to work around #2) you end up with a non-functioning BMO instance, until they are re-run - which for a new contributor is non-obvious.
5) The documentation misses a out a few steps and should ideally document common tasks (particularly since people may be unfamiliar with Docker).
6) The user has to perform some tasks manually during initial setup that could be simplified or avoided entirely. eg: making them manually edit checksetup_answers.txt if using boot2docker (which can then get committed to the repo accidentally), or using '<host>:8080/bmo/' rather than '<host>/' (which is not only more typing, but caused me confusion since the document root apache page is still present, making it look like the install failed when I forgot to add the '/bmo/' suffix).
7) The built bugzilla image is too large at a whopping 3.2GB, given the base centos image is only 215MB (need to check again after the recent Dockerfile changes that were landed at the same time as the file moves).

In addition, we can take advantage of the features new to Docker/Compose since the docker config was first created, and should also heed the advice given in:
https://docs.docker.com/articles/dockerfile_best-practices/

We'll need to update the instructions in docker/README.md, as well as on https://wiki.mozilla.org/BMO/DeveloperBox

Once complete, we should also advertise the simplified docker project on the newsgroups, with the aim of making devs realise it's now pretty easy to contribute and "scratch their own itch".

Comment 2

[Ed Morley [:emorley]]

WIP available at:
https://github.com/edmorley/webtools-bmo-bugzilla/commits/docker-new

Comment 3

[David Lawrence [:dkl]]

(In reply to Ed Morley [:emorley] from comment #2)
> WIP available at:
> https://github.com/edmorley/webtools-bmo-bugzilla/commits/docker-new

Looks nice from first glance. I will take a deeper look at it over the work week and we can do some more testing.

dkl

Comment 4

[Ed Morley [:emorley]]

Yeah I'm fairly pleased with it so far, still a few things left to do. Have just pushed an updated version.

Startup time once first-run is complete is pretty nice (that was one of the concerns with the multi-container approach iirc):

docker@boot2docker:/c/Users/Ed/src/bmo$ time docker-compose start
Starting bmo_db_1...
Starting bmo_memcached_1...
Starting bmo_web_1...
real    0m 1.37s
user    0m 0.26s
sys     0m 0.02s

Comment 5

[Ed Morley [:emorley]]

Docker compose supports Windows in the soon to be released 1.5.0 version (https://github.com/docker/compose/blob/1.5.0rc1/CHANGELOG.md#150-2015-10-13) which should make the story here much nicer.

I'll need to find time to double back to this, but Windows support plus some of the other recent Docker/Compose changes should make for a really user friendly docker-compose setup.