1075894 - Abstract Tupperware buildbot-app provisioning logic from its Dockerfile

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Morgan Phillips [:mrrrgn]]

By moving the app's provisioning logic out of the Dockerfile we gain the ability to deploy it outside of a docker container. This may be useful for things like setting up development environments or running experiments.

The simplest approach would be to move the steps currently in the Dockerfile to a bash script; but a provisioning system like ansible or puppet would be preferred.

Comment 1

[Morgan Phillips [:mrrrgn]]

Attachment: buildbot-app-ansible-01.diff

I've used this to set up the docker container in tupperware, and also to setup a local development environment for myself to good effect.

Comment 2

[Morgan Phillips [:mrrrgn]]

Attachment: buildbot-app-ansible-02.diff

This just moves all of the provisioning stuff to its own directory. I made it a separate commit so for the sake of reviewers. buildbot-app.yml and Dockerfile were the only changes that could have breaking effects (though I've tested the changes).

Comment 3

[Morgan Phillips [:mrrrgn]]

Attachment: buildbot-app-ansible-03.diff

Finally, a short readme explaining how to use the app as a standalone project.

Comment 4

[Morgan Phillips [:mrrrgn]]

Attachment: buildbot-app-ansible-03.diff

I had entered buildapi instead of buildbot in the readme *doh*

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 8498495 [details] [diff] [review]
buildbot-app-ansible-01.diff

Review of attachment 8498495 [details] [diff] [review]:
-----------------------------------------------------------------

overall, looks great!

nit: there's some extraneous whitespace in the yaml files that should be cleaned up.

::: buildbot-app/Dockerfile
@@ +2,5 @@
>  MAINTAINER John Zeller <[e-mail]>
>  
> +RUN apt-get update
> +RUN apt-get -y -q install python-pip gcc build-essential python-dev python-yaml
> +RUN pip install ansible

is ansible in apt? maybe that's a better place to get it from? then you can just install ansible and its deps, and have the playbook manage the rest.

::: buildbot-app/buildbot-app.yml
@@ +29,5 @@
> +         
> +    - name: Clone Mozilla projects
> +      hg: repo=https://hg.mozilla.org/build/{{ item }}
> +          dest={{ project_root }}/{{ item }}
> +      with_items: mozilla_projects

nice! you're changing which branch is checked out ("default" vs. one of the production branches). this is probably ok, just wanted to flag it.

@@ +46,5 @@
> +
> +    - name: Link buildbot custom to site packages
> +      file: src={{ project_root }}/buildbotcustom
> +            dest={{ project_venv }}/lib/python2.7/site-packages/buildbotcustom
> +            state=link

hm, I don't think this is necessary if we're adding it to buildbotcustom.pth above

::: buildbot-app/templates/install_masters.sh.j2
@@ +37,5 @@
> +done
> +
> +# Adjust configs in the /Buildbot/ and /Buildbot/build-master/
> +sed -i 's/\[\"mozilla-1.9.2\", \"mozilla-beta\"\]/\[\]/' master.json
> +sed -i ':a;N;$!ba;s/\[\n    \"mozilla-1.9.2\", \n    \"mozilla-beta\"\n  \]/[]/' build-master/master_config.json

hm, I wonder why we need to do this. zeller, any ideas?

::: buildbot-app/vars.yml
@@ +33,5 @@
> +  - six
> +
> +mozilla_projects:
> +  - buildbot
> +  - twisted

we don't actually use twisted from this repository any more, we use the upstream package from pypi

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 8498496 [details] [diff] [review]
buildbot-app-ansible-02.diff

Review of attachment 8498496 [details] [diff] [review]:
-----------------------------------------------------------------

what did you use to generate the diff? normally file renames can be represented nicely rather than a complete deletion and addition of the file. that would probably clean up the patch a bunch.

Comment 7

[Morgan Phillips [:mrrrgn]]

(In reply to Chris AtLee [:catlee] from comment #6)
> Comment on attachment 8498496 [details] [diff] [review]
> buildbot-app-ansible-02.diff
> 
> Review of attachment 8498496 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> what did you use to generate the diff? normally file renames can be
> represented nicely rather than a complete deletion and addition of the file.
> that would probably clean up the patch a bunch.

I've been using hg export, I'm on a mission today to nail down some of these workflow issues I noticed (this one included).

Comment 8

[Morgan Phillips [:mrrrgn]]

(In reply to Chris AtLee [:catlee] from comment #5)
> Comment on attachment 8498495 [details] [diff] [review]
> buildbot-app-ansible-01.diff
> 
> Review of attachment 8498495 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> overall, looks great!
> 
> nit: there's some extraneous whitespace in the yaml files that should be
> cleaned up.
> 
> ::: buildbot-app/Dockerfile
> @@ +2,5 @@
> >  MAINTAINER John Zeller <[e-mail]>
> >  
> > +RUN apt-get update
> > +RUN apt-get -y -q install python-pip gcc build-essential python-dev python-yaml
> > +RUN pip install ansible
> 
> is ansible in apt? maybe that's a better place to get it from? then you can
> just install ansible and its deps, and have the playbook manage the rest.
> 
> ::: buildbot-app/buildbot-app.yml
> @@ +29,5 @@
> > +         
> > +    - name: Clone Mozilla projects
> > +      hg: repo=https://hg.mozilla.org/build/{{ item }}
> > +          dest={{ project_root }}/{{ item }}
> > +      with_items: mozilla_projects
> 
> nice! you're changing which branch is checked out ("default" vs. one of the
> production branches). this is probably ok, just wanted to flag it.
> 
> @@ +46,5 @@
> > +
> > +    - name: Link buildbot custom to site packages
> > +      file: src={{ project_root }}/buildbotcustom
> > +            dest={{ project_venv }}/lib/python2.7/site-packages/buildbotcustom
> > +            state=link
> 
> hm, I don't think this is necessary if we're adding it to buildbotcustom.pth
> above
> 
> ::: buildbot-app/templates/install_masters.sh.j2
> @@ +37,5 @@
> > +done
> > +
> > +# Adjust configs in the /Buildbot/ and /Buildbot/build-master/
> > +sed -i 's/\[\"mozilla-1.9.2\", \"mozilla-beta\"\]/\[\]/' master.json
> > +sed -i ':a;N;$!ba;s/\[\n    \"mozilla-1.9.2\", \n    \"mozilla-beta\"\n  \]/[]/' build-master/master_config.json
> 
> hm, I wonder why we need to do this. zeller, any ideas?
> 
> ::: buildbot-app/vars.yml
> @@ +33,5 @@
> > +  - six
> > +
> > +mozilla_projects:
> > +  - buildbot
> > +  - twisted
> 
> we don't actually use twisted from this repository any more, we use the
> upstream package from pypi

Cool, thanks for the suggestions! I can use apt to get ansible, though, I'd like to ask zeller about changing the docker image from precise to trusty (since I'd need to add the ppa to this image which leads to as many extra steps as using pip).

I'm also going to make the repository branches configurable for people using this to set up a dev environment. Thanks for the suggestions! New patches coming once I get my diff generation sussed out. :)

Comment 9

[Morgan Phillips [:mrrrgn]]

Attachment: buildbot-app-ansible.diff

I removed some unnecessary packages, install steps, and cleaned up the source files a bit. I also finally figured out why hg export was acting so crazy with regard to renames (I needed to set my configuration to diff.git = 1).
This patch also introduces the ability to pass in a specific buildbot branch to be checked out. 

With regard to the bug in general, using Ansible here does have a trade off in that we slow down image builds considerably. Based on my attempts to set up a development environment using Tupperware however, I felt like having the ability to quickly provision buildbot within the vagrant vm and/or my local machine made it well worth the slower docker builds. 

The reason for this is that I've built a hybrid environment where I run whatever app I want to experiment with on the vagrant vm directly and then run its dependencies (mysql, rabbitmq) as docker containers. Another approach to development within Tupperware could be to have the Dockerfile pull in source code from a directory on the vm and rebuild the image after each change.

Both approaches have merits; but I'm favouring the hybrid development environment approach because developing outside of docker makes it less hassle to do things like ad-hoc debugging with pdb. That said, I'm very curious to learn about any disadvantages I haven't thought of.

Lastly:

Unfortunately, using the latest version of twisted caused this error so I left in the one from the repo for now. I'll play around with this and try to see what happened.

lt-py2.7.egg/buildbot/master.py", line 586, in startService
2014-10-03 03:17:18+0000 [-]     buildbot.monkey.monkeypatch_twisted_cbLogin()
2014-10-03 03:17:18+0000 [-]   File "/var/projects/buildbot/venv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_b8b419061e7c_default-py2.7.egg/buildbot/monkey.py", line 17, in monkeypatch_twisted_cbLogin
2014-10-03 03:17:18+0000 [-]     "monkeypatch failed version check; see note in monkey.py"
2014-10-03 03:17:18+0000 [-] AssertionError: monkeypatch failed version check; see note in monkey.py

Comment 10

[Rail Aliiev [:rail]]

Tupperware is dead \o/