1248751 - Update balrog to fit Dockerflow

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

Description

[Benson Wong [:mostlygeek]]

The balrog application needs to be updated to fit the Dockerflow[1] specification.

[1] https://github.com/mozilla-services/Dockerflow

Comment 1

[[e-mail] (:bhearsum)]

Looks like there's a few specific things that to happen:
1) Update the main Dockerfile to build a production-ready image. This means pulling out things like npm, and making sure all the necessary data is installed (instead of relying on bind mounts). We also need to make sure that changes to Python dependencies (currently specified in setup.py) will cause the image to be rebuilt.
2) Add __version__, __heartbeat__, and __lbheartbeat__ endpoints per the docs.
3) Tweak uwsgi configs to listen on $PORT instead of a hardcoded port.
4) Get rid of uwsgi .ini files or bake them into the image.
5) Ensure app logs go to stdout and stderr, not a file, and use the mozlog format (https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=42895640)

Other things will probably come up, this is a start though.

Comment 2

[[e-mail] (:bhearsum)]

Quick update here:
(In reply to Ben Hearsum (:bhearsum) from comment #1)
> Looks like there's a few specific things that to happen:
> 1) Update the main Dockerfile to build a production-ready image. This means
> pulling out things like npm, and making sure all the necessary data is
> installed (instead of relying on bind mounts). We also need to make sure
> that changes to Python dependencies (currently specified in setup.py) will
> cause the image to be rebuilt.

I've got a patch in progress for this: https://github.com/mozilla/balrog/compare/master...bhearsum:production-dockerfile

I think the current state of the Dockerfile in that branch will build production-ready Docker images. I'm still trying to sort out whether or not we can push them to dockerhub from Taskcluster. If I can't get that figured out next week I'll probably punt on it and switch to CircleCI.

> 2) Add __version__, __heartbeat__, and __lbheartbeat__ endpoints per the
> docs.

This is also started in https://github.com/mozilla/balrog/compare/master...bhearsum:dockerflow-endpoints

Nothing difficult here AFAICT, just need to do it.

> 4) Get rid of uwsgi .ini files or bake them into the image.

This probably fixed with #1, as I'm copying the entire repository into the image, which includes these files.

Comment 3

[Benson Wong [:mostlygeek]]

Took a look at the patch: https://github.com/mozilla/balrog/compare/master...bhearsum:production-dockerfile

- are you planning on turning this into a PR? 
- why requirements.txt and then use setup.py to install it?
  - the convention is usually to install with pip. pip is already on the base container 
    and in Dockerfile we can copy in requirements.txt, pip install ... and get 
    all the requirements in and cached as a layer. This way new builds where
    requirements.txt don't change will use the cached layer.

Comment 4

[Benson Wong [:mostlygeek]]

Here are some examples you can work from. I've been working with the TestPilot guys and I think they have the best/most advanced Dockerflow implementation so far: 

* Dockerfile: https://github.com/mozilla/testpilot/blob/master/Dockerfile
* CircleCI: https://github.com/mozilla/testpilot/blob/master/circle.yml

With circle.yml: 

* Static frontend assets are built by the CI and tested using a headless browser
* Copied into the container as static assets
* Container is built and tested
* Container is shipped to DockerHub

It sounds pretty similar to what balrog's needs are.

Comment 6

[[e-mail] (:bhearsum)]

Sorry, missed these comments until just now.

(In reply to Benson Wong [:mostlygeek] from comment #3)
> Took a look at the patch:
> https://github.com/mozilla/balrog/compare/master...bhearsum:production-
> dockerfile
> 
> - are you planning on turning this into a PR? 

Should have one ready tomorrow. Any progress on getting the "mozillabalrog" dockerhub account access to push images? I was hoping to test that out first.

> - why requirements.txt and then use setup.py to install it?
>
>   - the convention is usually to install with pip. pip is already on the
> base container 
>     and in Dockerfile we can copy in requirements.txt, pip install ... and
> get 
>     all the requirements in and cached as a layer. This way new builds where
>     requirements.txt don't change will use the cached layer.

I originally did this because it's what shavar does (https://github.com/mozilla-services/shavar/blob/master/setup.py). I think it's proper to include the requirements in setup.py, because the normal way to install Python modules is "python setup.py install", which should pull in required dependencies. However, there's no reason we can't run pip before setup.py in the Dockerfile to have the benefit of caching without breaking setup.py, I think?

Comment 7

[Benson Wong [:mostlygeek]]

- mozillabalrog should have write access to https://hub.docker.com/r/mozilla/balrog now. 

re: setup.py

- could we copy in setup.py first and run `python setup.py install` without the rest of the app?
- if we can, then I'm +1 on doing it this way, otherwise the requirements.txt way is preferred
  since it will make use of the docker cache

Comment 8

[[e-mail] (:bhearsum)]

Attachment: add support for dockerflow requirements, minus mozlog

More details in the PR.

Comment 9

[Benson Wong [:mostlygeek]]

Comment on attachment 8725887 [details] [review]
add support for dockerflow requirements, minus mozlog

commented in the PR

Comment 10

[Benson Wong [:mostlygeek]]

Balrog's Dockerflow PR: https://github.com/mozilla/balrog/pull/55

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8725887 [details] [review]
add support for dockerflow requirements, minus mozlog

f+ with some comments on github where we can improve the patch.

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 8725887 [details] [review]
add support for dockerflow requirements, minus mozlog

I think this is ready for review now, details in https://github.com/mozilla/balrog/pull/55#issuecomment-194936702.

Comment 13

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/efe00da440b91f15957e7d9ecb67acde67734bf3
Merge pull request #55 from bhearsum/dockerflow-endpoints

bug 1248751: Add support for Dockerflow requirements (minus mozlog). r=nthomas,mostlygeek

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 8725887 [details] [review]
add support for dockerflow requirements, minus mozlog

It required a few follow-ups to deal with some issues with the commands in the Docker script, but this is now landed and pushing images to https://hub.docker.com/r/mozilla/balrog/tags/

Comment 15

[[e-mail] (:bhearsum)]

I think we're done with this bug. bug 1251335 will deal with mozlog.