1296238 - refactor, slim down, and otherwise clean up configuration

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[[DEACTIVATED] Adrian Gaudebert]

We have a lot of configuration keys in consulate, and that is hard to parse and hard to maintain. We also have a few configuration files in our socorro-infra repo [0], and those are loaded in boxes when we build them. 

So, we want to clean that up a bit, doing the following:

* remove useless options or options that are the same as the application defaults
* move options with the same values between stage and prod into the socorro-infra config files

[0] https://github.com/mozilla/socorro-infra/tree/master/socorro-config

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro-infra/pull/251

First step: the webapp namespace. This pull request reorganizes the webapp-django.conf file, removes unneeded values and adds a few options that have the same values in stage and prod.

Comment 2

[[DEACTIVATED] Adrian Gaudebert]

Attachment: changes-stage-webapp.md

Changes to apply through consulate on stage after previous PR is merged.

Comment 3

[[DEACTIVATED] Adrian Gaudebert]

Attachment: changes-prod-webapp.md

Changes to apply through consulate on prod after previous PR is merged and deployed.

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro-infra

https://github.com/mozilla/socorro-infra/commit/4f37041fa2153d9feb0db2060d00d02cfb04880d
Bug 1296238 - Refactored basic webapp configuration.

https://github.com/mozilla/socorro-infra/commit/6d19a7cf4016bba3d7fc3ca95d4c3705ab1834fb
Merge pull request #251 from adngdb/1296238-refactor-webapp-config

Bug 1296238 - Refactored basic webapp configuration.

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro-infra

https://github.com/mozilla/socorro-infra/commit/9dc16f5f42c2847c811f467e4494475c89d15b28
Revert "Bug 1296238 - Refactored basic webapp configuration."

https://github.com/mozilla/socorro-infra/commit/c421b8856f510dba5782a4c846761bed5888d5df
Merge pull request #253 from mozilla/revert-251-1296238-refactor-webapp-config

Revert "Bug 1296238 - Refactored basic webapp configuration."

Comment 6

[Peter Bengtsson [:peterbe]]

Comment on attachment 8782361 [details]
changes-stage-webapp.md

Yes. I checked the "# Useless keys" part. They'are all useless. 

I don't know why I'm really struggling to understand what it means "# Already defined in config files"

I thought we had entirely done away with config file?! Are they still lurking? If so, we should delete them all and not use them. The only sane places for config is consul or defaults in .py files. 

Where are these config files?

Comment 7

[Peter Bengtsson [:peterbe]]

Comment on attachment 8782362 [details]
changes-prod-webapp.md

Same point I made on the changes-stage-webapp.md file. 
1) The useless keys definitely are useless. 
2) I don't understand why these values are in config files (and not in consul or source code)

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think I do want to remove keys that are no longer used, but I want to know what the process is for determining what is a key that's no longer used. Some of them are obvious. Others not so much.

I don't think I want to move configuration between consul and the config files in socorro-infra. I'd much rather have all the configuration in one place since it makes it easier to reason about what's going on without having to look in multiple places.

As a side note, I discovered some differences between -stage and -prod. Those differences would not have been alleviated by this work. We should do an audit between the two environments and fix the differences.

Comment 9

[Lonnen :lonnen]

I agree with Will, at least temporarily. Let's make a decision on this at or after the work week.

Comment 10

[Miles Crabill [:miles]]

What is the status of this bug? Can all secrets be set via environment variables?

Comment 11

[Lonnen :lonnen]

Consul is loosely coupled through environment variables via envconsul, everywhere

Comment 12

[Lonnen :lonnen]

@miles -- this has a confusing title. We have conflicting definitions of configuration within the project.

This bug is not 'remove consul entirely' but instead to remove any config values from consul that are in agreement with application defaults, and then move common configuration between stage+prod out of consul. Consul would still provide attributes about the environment like the address and creds for accessing data stores.

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

Two things:

1. Consul is going away with the new docker-based environment.

2. The docker-based dev environment is a -prod like environment. We can use that to test the kinds of changes in this bug in the comfort and safety of our throw-away local dev environments.

Given that, I'm going to grab this and work on it after I have the dev environment more functional.

Comment 14

[Will Kahn-Greene [:willkg] ET needinfo? me]

I did a pass on docker/config/local_dev.env . This file holds all the security and environment-specific configuration for all the Socorro components running in a local dev environment. This file will be used to figure out the equivalent file for the -stage and -prod server environments. Thus it behooves us to have a minimal file in which we know what all the things are.

We should do passes on the other configuration files as well.

Comment 15

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4005 - clean up docker/config/local_dev.env

Comment 16

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/f44d09d7385a756459d4d8e1e7ed768b1109d754
bug 1296238 - reduce vars in local_dev.env (#4005)

This removes a bunch of unneeded vars from local_dev.env and moves a few others
to better files.

Comment 17

[Will Kahn-Greene [:willkg] ET needinfo? me]

Mike's PR moves behavior configuration into Python code: https://github.com/mozilla-services/socorro/pull/4307

I think that'll radically reduce the amount of configuration we have floating around. We should wait until that lands and then see if there's anything left to do here.

Unassigning myself.

Comment 18

[Will Kahn-Greene [:willkg] ET needinfo? me]

Mike's PR has landed.

We also have bug #1439546 which removes a ton of dead configuration. Once that lands, I think we can call this bug done, too.

Comment 19

[Will Kahn-Greene [:willkg] ET needinfo? me]

I marked the last outstanding bug as FIXED.

Given that, I'm going to mark this FIXED, too. Yay!