1296047 - Clean up the Vagrant puppet scripts

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P3)

Description

[Ed Morley [:emorley]]

Whilst working on bug 1289156 I spotted a few things that could do with fixing:
* there are a number of tasks performed that really aren't necessary and just add to the provision time / puppet script complexity unnecessarily
* the environment variables / bashrc additions are hardcoded in the configs as numerous puppet steps, when having them in an external file would be clearer
* the environment variables / profile scripts are copied in rather than symlinked, so don't get updated until someone re-runs provision manually (this would be a good start towards fixing bug 1196725)
* the entire default mysql config is overwritten rather than just adding a separate config that adjusts the settings we use that are non-default

...and so on.

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:puppet-cleanup > mozilla:master

Comment 2

[Cameron Dawson [:camd]]

Comment on attachment 8782133 [details] [review]
[treeherder] mozilla:puppet-cleanup > mozilla:master

Whew!  That was a lot of work.  thanks for doing this.  :)

Comment 3

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/a5a6d5bb3cc64c9d1bb50eb96b9f63d4550a5cd9
Bug 1296047 - Vagrant: Stop creating a 'puppet' group during provision

Since it's unused.

https://github.com/mozilla/treeherder/commit/1fc05da62c0cea1183760bf8e478a9afa42067db
Bug 1296047 - Vagrant: Use the default rabbitmq guest user

A default user exists with username 'guest' and password 'guest', which
we might as well use to save having to create another and set up grants.

See:
https://www.rabbitmq.com/access-control.html

https://github.com/mozilla/treeherder/commit/fd554ba8c2a93180290fcebebae8621dc74f9dc2
Bug 1296047 - Vagrant: Avoid repetition in DATABASE_URL_RO

By using django-environ's proxied environment variable feature, like is
already being used in Heroku's value for `DATABASE_URL_RO`. See:
https://github.com/joke2k/django-environ/blob/v0.4/environ/environ.py#L257-L260

https://github.com/mozilla/treeherder/commit/55b29baf5d9dca6e5fb65095d45aa257a3f9c592
Bug 1296047 - Vagrant: Use the default root mysql user

A default user exists with username 'root' and blank password, which we
might as well use to save having to create another. We still have to add
a grant to allow root to connect from outside the VM, since the default
grant of `root@localhost` only allows connections via the loopback
interface.

The dependency on the `create-db` task has also been removed, since the
grant uses a wildcard, so doesn't refer to the `treeherder` DB directly.

https://github.com/mozilla/treeherder/commit/ec7845ff5de94f3c8b157bc830b73e147768c536
Bug 1296047 - Vagrant: Move environment variables to their own file

Having them in a separate file is cleaner, makes the discovering where
the environment variables are set easier, plus means we can symlink the
file, so future variable changes will take effect immediately, rather
than needing a re-provision after pulling latest master.

https://github.com/mozilla/treeherder/commit/3bb0ee5d30c31d7faef08c470fe008aedafd21cf
Bug 1296047 - Vagrant: Move helper functions to a symlinked .profile

This renames the existing `.bash_aliases` file to `.profile`, since
we're soon going to use it for more than just aliases. It overwrites the
default `.profile` file in the VM, so we need to source `.bashrc` as the
original did.

In addition, rather than copying the file we now symlink it, so that
future changes don't require a re-provision after pulling latest master
to take effect.

https://github.com/mozilla/treeherder/commit/7ce025af86338866b2a83b4247c450497d3a07fa
Bug 1296047 - Vagrant: Move .bashrc additions to .profile

Previously provision would append entries to .bashrc in multiple steps,
whereas now these are just included in the `.profile` that is symlinked
from the environment. As such, future changes will no longer need a
re-provision after pulling latest master, to take effect.

https://github.com/mozilla/treeherder/commit/1ef96e8366c432d7927e3ea2f137eacca80d2a5a
Bug 1296047 - Vagrant: Merge the two sample django settings files

Previously there were two sample Django config files, and confusingly
the one that would be used in the Vagrant environment wasn't the one
that was the most visible.

In addition, we're not performing any kind of variable substitution, so
don't need to use `content => template()`.

https://github.com/mozilla/treeherder/commit/033af82d778e4f1dc9f31d8cadcef8f69300ae0d
Bug 1296047 - Vagrant: Combine the directory creation steps

https://github.com/mozilla/treeherder/commit/19dbfe4c7dc8e7691dfd539b7fec1817e519bf24
Bug 1296047 - Vagrant: Combine the two pip install steps

Since the dev.txt packages no longer depend on system packages installed
in mysql.pp, so don't need to be installed separately. The working
directory has also been adjusted to avoid the need to specify the full
path to the requirements files.

https://github.com/mozilla/treeherder/commit/4504ccb5f7730290576e0cd5c2bab80f4f12c061
Bug 1296047 - Vagrant: Remove redundant fully qualified paths

The path used for `exec{}` commands (defined in `vagrant.pp`) includes
`${VENV_DIR}/bin` so we don't need to specify the full path for pip
and python invocations that are meant to use the virtualenv binaries.

https://github.com/mozilla/treeherder/commit/cc5b8558641c878f1d5ab0ce3d3dd2dc980d5742
Bug 1296047 - Vagrant: Only override specific MySQL server preferences

Rather than overwriting the default MySQL 5.6 config file, the specific
changes we wish to make are now made via a file in the `mysql/conf.d/`
include directory. This makes it easier to see where we differ from the
defaults, as well as preventing us from inadvertently overriding any
new defaults in `/etc/mysql/my.cnf` when we update to new MySQL major
versions in the future.

The contents of this new file were determined by diffing against the
untouched `/etc/mysql/my.cnf` file, and removing anything that was still
set to defaults.

https://github.com/mozilla/treeherder/commit/863925c59d759b3b58c3a859d27174153dd2763e
Bug 1296047 - Travis: Use the same MySQL config file as Vagrant

To reduce duplication and ensure the configurations remain in sync.

Note: The Vagrant config does set `bind-address` to allow non-localhost
connections, which isn't necessary on Travis, however this is fine since
(a) it's only Travis, (b) the user grants on Travis won't actually allow
non-localhost connections anyway, even if Travis' network settings
allowed connections between test nodes.