598734 - Refactor configuration files

Status: RESOLVED DUPLICATE of bug 653197

(Socorro :: General, task)

Description

[Laura Thomson :laura]

In the postmortem, IT mentioned that it would be useful if things-that-don't-often-or-ever-change were separate from things-that-have-to-change in our config files.  

Example of the former: jsonFileSuffix 

Example of the latter: databaseHost

Comment 1

[Aravind Gottipati [:aravind]]

What I was proposing was more along the lines of, ship a config-dist file and take the values in the dist file as the defaults.

The ops folks will override values in the config file, but only those values that need to be different from the dist file.  This cuts down on the stuff we need to configure and override.

If you want to get rid of a bunch of options from the dist file entirely, that's fine as well, but it might be a good idea to leave those in just in case.

The deployment instructions should then say something like, you will probably need to set the following. "dbhostname, dbport, etc..", and if we want to override others, we know where to look (the -dist file).  Let me know if you need me to clarify any of that stuff.

Comment 2

[K Lars Lohn [:lars] [:klohn]]

I don't know if this gets you in the direction that you want to go, but just as a reminder of existing but unused features of the config file system:

you can use commandline parameters or environment variables to override the defaults in the config file.  If, for example, you want to just use the dist files without modification, you could set up your shell script to invoke apps like this:

#...
export hbaseHost=somehost.somewhere
export hbaseTimeout=5000
export databaseHost=postgres.somewhere
python ./startWhateverService --databasePassword=myPassword

You could even make your own files that have just the relevant exports in them and source them:

#...
. hbaseEnvironment
. postgresEnvironment
python ./startWhateverService

You can essentially make your own configuration files consisting of only the relevant parameters without ever having to touch the actual python language config files.

One caveat, however, some config parameters are required to have values that are python data structures ('throttleConditions' from collectorconfig.py is an example).  These cannot be expressed as environment variable strings (though it would be possible to correct that).