1137022 - [12fa] switch to environment-based config for instance settings

Status: RESOLVED FIXED

(Input Graveyard :: Backend, defect)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

The settings in the fjord/settings/*.py files fall into two categories:

1. project settings
2. instance settings

For instance settings, we want to change the code so that we pull the value from the environment. For values that can have a sensible default, we can default to that when it's not in the environment. For values that do not have a sensible default, the code should fail fast. Values should be over-rideable via the local.py file.

There are two caveats here.

First, all values are strings and we need them in the appropriate data type. We should look at using the smart_* utility functions we currently use for querystring params--are they helpful here?

Second, we need to do this work in such a way that our server environments are unaffected. If we have to make any changes to our server environments, that work should be done before this work lands.

It's probably worth taking some inspiration from python-decouple, but I think that project does more than we want, so I don't think we want to go with that.
https://github.com/henriquebastos/python-decouple

This bug involves the following:

1. make a list of instance settings

2. implement the code for pulling values from the environment, decoding to the correct type and failing when required values are missing

3. convert the instance settings to that

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Also of note, the database configuration is covered in bug #1137018.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Worth looking at:

https://github.com/MechanisM/django-confy

It combines dj-database-url, dj-email-url, dj-search-url, django-cache-url and some other things together.

Having said that, the README and LICENSE and code suggests the maintainer copy/pasted a bunch of code from those and other projects and is now asserting copyright for and changed the license of it. That's legal shenanigans. Not sure we want to use that library. I don't think it's hard to redo what he did without creating the legal problems.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Spent a while prototyping how settings could work with Harold (it's a much smaller project than Fjord) and then talked with Ricky about it, too.

That resulted in a nice set of good practices that covers the various scenarios I wanted covered.

Next step is to take that design and then figure out how to map it to Fjord taking into account we have a pre-existing setup for our server environments that we don't want to mess with.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this and making it a 3-pointer. I've put a bunch of work into it so far (mostly in researching things) and it still requires some implementation and testing work.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think the plan will be something like this:

Step 1. Spend some time removing settings that we don't need to have. What can get computed? What is deprecated?

Step 2. Figure out exactly what stage/prod are setting so we know what settings need to be there. The thinking is that stage/prod will continue to use ``fjord/settings/local.py`` and we won't touch that path at all until we switch to AWS.

Step 3. Change ``fjord/settings/base.py`` to pull secret settings and resource settings from the environment with sane defaults in appropriate places. This will use dj_database_url, dj_caches_url,
something for seach and possibly some other things.

Step 4. Change ``fjord/settings/__init__.py`` to skip importing things if ``DJANGO_SETTINGS_MODULE`` is set.

Step 5. Create a new ``fjord/settings/vagrant.py``. It will pull in everything from ``fjord.settings.base`` and then override some settings and set things up according to the vagrant environment. We'll check this into version control.

Environment variables will be set in the ``.bashrc`` file including ``DJANGO_SETTINGS_MODULE``. Maybe have it source an vagrant_settings.sh file which we keep in version control? That'd make it easier to update everyone's vagrant environment over time.

Step 6. Rework ``fjord/settings/test.py`` so that it does the same thing as ``vagrant.py``. Switch all testing to use this by setting ``DJANGO_SETTINGS_MODULE``.


After that, I think that's as far as we can go with this bug and we'll have to wait to do the rest when we migrate to AWS.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

I've got most of the work I'm going to do done now. I'm breaking it up into a few smaller PRs to make it easier to review.

First PR is mostly cosmetic cleanup changes: https://github.com/mozilla/fjord/pull/534

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed PR 534 in https://github.com/mozilla/fjord/commit/99c84d32b13e2ffe3dbc0a08eb9ddb92e59d305f

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

I built some settings infrastructure and also rewrote most of the startup code. In a PR: https://github.com/mozilla/fjord/pull/532

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in master:

https://github.com/mozilla/fjord/commit/84b11d866765d5a32fc1b33e889a77dbee9fe773
https://github.com/mozilla/fjord/commit/e4e615d74ee5df4e9690cd98de89e729e180b2ed
https://github.com/mozilla/fjord/commit/4bb85f8c946b4b99bbfe42064deac729e871d92f

Pushing to stage now. I'll let it sit there and then push it to prod tomorrow.

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

This PR fixes some issues with the test environment: https://github.com/mozilla/fjord/pull/536

Otherwise everything looks fine on stage, so once that lands I'll push this to prod.

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

PR 536 landed in master in https://github.com/mozilla/fjord/commit/9a95852ca1bbaa51e37546b6975cf5587b07b4d4

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pushed everything to production just now. Yay!

Marking as FIXED.