1077510 - Check a Vagrant/Docker project file (excl DB content) into the Bugzilla repo

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Developer Box, defect)

Description

[Ed Morley [:emorley]]

In an ideal world, developing on Bugzilla would be as simple as:
1) git clone foo/bugzilla.git && cd bugzilla && vagrant up
2) Open a browser to <IP>/localhost to see the Bugzilla instance

After setup, one would simply:
1) Edit a file in the host (the folder then syncs to the VM & the VM auto-reloads the HTTP process/cache/whatever it needs to)
2) Refresh the page in your browser to see the change

...this is basically what we're able to do with projects like TBPL, Orangefactor, Treeherder - and it makes a huge difference to ease of access of the project.

As an added bonus (though secondary to this bug), for those who do not wish to use a VM, they can still use the puppet scripts that Vagrant uses, to bootstrap a local environment on their own Linux box. (See TBPL for an example of this: https://hg.mozilla.org/webtools/tbpl/file/default/puppet/ubuntu-bootstrap.sh)

What would be even more amazing (read: we can defer this to another bug), is if there were an easy way to pull in a sanitized DB dump of BMO (or at least the settings/product names/flags/custom fields etc). This could either be a step during the puppet provision, or a manual command run after |vagrant ssh|ing into the box after the first |vagrant up|.

Now I know there was a pre-made Vagrant box in existence previously (https://wiki.mozilla.org/BMO/DeveloperBox) however it's no longer available after the DB dumps were pulled.

Checking a Vagrant project into the tree has several advantages over the standalone box:
1) People don't have to find a random wiki page to know about Vagrant support (we just mention it in the README & people can see the Vagrantfile in the root of the repo)
2) It separates the DB dump debate from the rest, meaning we can at least get _something_ running easily with a |vagrant up|
3) Much smaller download (particularly should someone not need the full DB).
4) We can backport it to Bugzilla upstream (should the maintainers not mind), which means it can be used for both, whereas the box on https://wiki.mozilla.org/BMO/DeveloperBox was just for BMO.

Comment 1

[Ed Morley [:emorley]]

Could you update this with info/docs about the new version-control-tools docker instance, and also the pre-filled DB stuff?

Comment 2

[Ed Morley [:emorley]]

Happy for this to be about Docker instead, if that is preferred.

Comment 3

[Mark Côté [:mcote]]

I defer to dkl here.

Comment 4

[David Lawrence [:dkl]]

I have the beginnings of this now available at https://github.com/dklawren/docker-bugzilla/tree/bmo which could be moved to a directory in the BMO git repo. One could then update the fig.yml file to map a local directory over
the webroot in the Docker container to use local Bugzilla files. This would give similar result to what is described in comment 0 using Vagrant.

If I were to add the files to contrib/docker and add patch here, Ed, would you mind testing and reviewing it?

Note, it will not have a full sanitized dump of BMO obviously, but it will have a script to create some starter data that should help in BMO development.

The master branch of the github repo is a working upstream Bugzilla instance as well which I would like to add to the upstream repo once we get the kinks ironed out here.

dkl

Comment 5

[David Lawrence [:dkl]]

Attachment: 1077510_1.patch

Here is a patch that creates a new contrib/docker directory in the BMO code base which contains the needed files to get a small BMO installation up and running.

Please give it a try and let me know where I can do better and if it works at all for you.

Basic instructions:

1. Install Docker, Fig, and boot2docker (if needed for OSX) as per their own 
   instructions (some in the README.md).
2. http://git.mozilla.org/?p=webtools/bmo/bugzilla.git bmo
3. cd bmo
4. patch -p1 < /path/to/1077510_1.patch
5. cd contrib/docker
6. fig build
7. fig up
8. In browser, go to http://localhost:8080/bmo and login as admin.

In an ideal world, we could figure out a way to mount the current git BMO checkout inside the docker image so the changes you make show up in realtime. But right now the docker container contains it's own fresh git clone.

I have done the initial work in github at https://github.com/dklawren/docker-bugzilla. We could actual make the contrib directory a submodule pointing to my github repo. I do not really want to make changes to two places when things are fixed/improved. The github repo also has a master branch which is useful for upstream work, and a dkl branch which will be my work environment going forward.

Thanks
dkl

Comment 6

[Ed Morley [:emorley]]

Thank you - will take a look! :-)

Comment 7

[Ed Morley [:emorley]]

Comment on attachment 8537466 [details] [diff] [review]
1077510_1.patch

Review of attachment 8537466 [details] [diff] [review]:
-----------------------------------------------------------------

Unfortunately I got the following failure:

docker@boot2docker:/c/Users/Ed/src/bmo/contrib/docker$ fig build
...
Step 11 : RUN yum -y -q install supervisor mod_perl mod_perl-devel openssh-server openssh     passwd mysql-community-server mysql-community-devel git sudo perl-App-cpanminus     tar gcc gcc-c++ make unzip mpfr-devel vim-enhanced openssl-devel gmp-devel     gd-devel postfix graphviz ImageMagick-devel patch aspell-devel perl-CPAN     && yum clean all
 ---> Running in 17cddc42384e
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7/epel/packages/mod_perl-2.0.8-10.20140624svn1602105.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Public key for mod_perl-2.0.8-10.20140624svn1602105.el7.x86_64.rpm is not installed
warning: /var/cache/yum/x86_64/7/mysql56-community/packages/mysql-community-common-5.6.22-2.el7.x86_64.rpm: V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Public key for mysql-community-common-5.6.22-2.el7.x86_64.rpm is not installed
Importing GPG key 0x5072E1F5:
 Userid     : "MySQL Release Engineering <[e-mail]>"
 Fingerprint: a4a9 4068 76fc bd3c 4567 70c8 8c71 8d3b 5072 e1f5
 Package    : mysql-community-release-el7-5.noarch (@/mysql-community-release-el7-5.noarch)
 From       : file:/etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
Importing GPG key 0x352C64E5:
 Userid     : "Fedora EPEL (7) <[e-mail]>"
 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
 Package    : epel-release-7-5.noarch (@extras)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
install-info: No such file or directory for /usr/share/info/gdbm.info.gz
error: unpacking of archive failed on file /usr/sbin/suexec: cpio: cap_set_file failed - Operation not supported
Error unpacking rpm package httpd-2.4.6-18.el7.centos.x86_64
error: httpd-2.4.6-18.el7.centos.x86_64: install failed
sh: /bin/systemctl: No such file or directory
Service 'bugzilla' failed to build: The command [/bin/sh -c yum -y -q install supervisor mod_perl mod_perl-devel openssh-server openssh     passwd mysql-community-server mysql-community-devel git sudo perl-App-cpanminus     tar gcc gcc-c++ make unzip mpfr-devel vim-enhanced openssl-devel gmp-devel     gd-devel postfix graphviz ImageMagick-devel patch aspell-devel perl-CPAN     && yum clean all] returned a non-zero code: 1

::: contrib/docker/README.md
@@ +13,5 @@
> +  diffed, and branched using standard git commands
> +
> +## How to install Docker and Fig
> +
> +### Linux Workstation

I think perhaps either s/Workstation// or s/Workstation/Client/ for this and the others further down?

@@ +38,5 @@
> +
> +Windows based developers will be best served by installing [Vagrant][vagrant] and
> +relying on a shim VM to run Docker. Follow the instructions in the installer until
> +you reach the ``vagrant init`` section. Instead of doing ``vagrant init hashicorp/precise32`` do:
> +

I actually found the following works (and is likely faster than using a fully Vagrant image):
1) Install the Windows boot2docker installer
2) Run the "Boot2Docker Start" shortcut on the startmenu (this inits the VM, starts it and connects to it)
3) Run the following in the boot2docker VM as a one-off:
echo 'alias fig='"'"'docker run --rm -it \
        -v $(pwd):/app \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -e FIG_PROJECT_NAME=$(basename $(pwd)) \
        dduportal/fig'"'" >> /home/docker/.ashrc
4) Logout from the VM (ctrl+D)

Then all you need to do on later occasions is:
1) Re-run "Boot2Docker Start" (or else from the msys console just |boot2docker start && boot2docker ssh|)
2) cd /c/Users/Username/src/bmo/contrib/docker  (paths under c:\Users are automatically mapped by boot2docker from the client into the VM)
3) fig build (and so on)

Comment 9

[David Lawrence [:dkl]]

(In reply to Ed Morley [:edmorley] from comment #7)
> error: unpacking of archive failed on file /usr/sbin/suexec: cpio:
> cap_set_file failed - Operation not supported
> Error unpacking rpm package httpd-2.4.6-18.el7.centos.x86_64
> error: httpd-2.4.6-18.el7.centos.x86_64: install failed

This is the only error to worry about and is the reason the build fails entirely. It is a known upstream issue with the AUFS storage driver used by default by boot2docker.

https://github.com/docker/docker/issues/8966

The upstream docker community is reluctant to allow --privileged flags for 'docker build' as it would break compatibility which images on all platforms. it is best to fix the issues with package installers themselves directly.

A workaround for boot2docker itself to allow this to work is to change the storage driver to devicemapper. devicemapper is the default storage driver used in Fedora installs which is why I do not encounter this error when building locally. 

http://three1415.wordpress.com/2014/12/19/switching-storage-driver-of-boot2docker/

Before trying to 'fig build' but after starting the boot2docker VM:

$ boot2docker ssh
Boot2Docker version 1.4.1, build master : 86f7ec8 - Tue Dec 16 23:11:29 UTC 2014
Docker version 1.4.1, build 5bc2ff8
docker@boot2docker:~$ echo 'EXTRA_ARGS="--storage-driver=devicemapper"' | sudo tee -a /var/lib/boot2docker/profile
docker@boot2docker:~$ sudo /etc/init.d/docker restart

Please give that a try and I will update the README.md to relect until a proper upstream fix is found.
 
> ::: contrib/docker/README.md
> @@ +13,5 @@
> > +  diffed, and branched using standard git commands
> > +
> > +## How to install Docker and Fig
> > +
> > +### Linux Workstation
> 
> I think perhaps either s/Workstation// or s/Workstation/Client/ for this and
> the others further down?

Good idea.

> I actually found the following works (and is likely faster than using a
> fully Vagrant image):
> 1) Install the Windows boot2docker installer
> 2) Run the "Boot2Docker Start" shortcut on the startmenu (this inits the VM,
> starts it and connects to it)
> 3) Run the following in the boot2docker VM as a one-off:
> echo 'alias fig='"'"'docker run --rm -it \
>         -v $(pwd):/app \
>         -v /var/run/docker.sock:/var/run/docker.sock \
>         -e FIG_PROJECT_NAME=$(basename $(pwd)) \
>         dduportal/fig'"'" >> /home/docker/.ashrc
> 4) Logout from the VM (ctrl+D)
> 
> Then all you need to do on later occasions is:
> 1) Re-run "Boot2Docker Start" (or else from the msys console just
> |boot2docker start && boot2docker ssh|)
> 2) cd /c/Users/Username/src/bmo/contrib/docker  (paths under c:\Users are
> automatically mapped by boot2docker from the client into the VM)
> 3) fig build (and so on)

I will attach a new patch with revised Windows docs and you can run through it and make sure they still work as instructed.

dkl

Comment 10

[David Lawrence [:dkl]]

Attachment: 1077510_2.patch

Comment 11

[Ed Morley [:emorley]]

Comment on attachment 8544649 [details] [diff] [review]
1077510_2.patch

Review of attachment 8544649 [details] [diff] [review]:
-----------------------------------------------------------------

Overall works really well :-)

On fig up, I got some postfix errors (will attach log).

::: contrib/docker/README.md
@@ +109,5 @@
> +```
> +
> +## How to access the Bugzilla container
> +
> +You can point your browser to `http://localhost:8080/bmo` to see the the

This isn't true on Windows, it's http://192.168.59.103:8080/bmo/ (or whatever |boot2docker ip| outputs).

::: contrib/docker/checksetup_answers.txt
@@ +9,5 @@
> + $answer{'db_mysql_ssl_ca_file'}     = '';
> + $answer{'db_mysql_ssl_ca_path'}     = '';
> + $answer{'db_mysql_ssl_client_cert'} = '';
> + $answer{'db_mysql_ssl_client_key'}  = '';
> + $answer{'urlbase'} = 'http://localhost/bmo/';

This isn't true on Windows, it's http://192.168.59.103:8080/bmo/ (or whatever |boot2docker ip| outputs) does this have to be an absolute path?

Comment 12

[Ed Morley [:emorley]]

Attachment: errors during fig up

Comment 13

[Ed Morley [:emorley]]

Meant to say: even with the errors, is still much much easier than anything else at present, so might be good to just check it in and iterate from here :-)

Comment 14

[David Lawrence [:dkl]]

Thanks Ed. Updated the docs to reflect the IP address changes. Please file the postfix bug separately and we can improve incrementally as you suggested.

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   ee940fd..7bbb242  master -> master

dkl