Closed Bug 1311468 Opened 4 years ago Closed 4 years ago

Allow external tools partial access to a read-only replica of the prod Treeherder DB

Categories

(Tree Management :: Treeherder: Infrastructure, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wlach, Assigned: wlach)

References

Details

Attachments

(2 files, 2 obsolete files)

For redash (bug XXX) and maybe other applications (ActiveData?), it would be nice to have a read-only copy of the treeherder database with any sensitive data (passwords, hawk credentials, session tokens) sanitized and removed. I understand this should be pretty easy with RDS?

NI'ing :emorley on the details of what should be sanitized. After that we can ping someone from operations to actually set this up.
Flags: needinfo?(emorley)
To be clear, I think we only want to mirror the "treeherder" database. The various per-project databases we can skip (since they will be going away soon).
I hope this extract is just a diff-since-last-extract[1], and in some popular format[2] so we do not need to spin up a database to access the records.

[1] I do not mean a fine grained diff; just a list of all records that have had some value change since last time; hopefully it will include all the related records (details, notes and stars) for completness.
[2] like JSON!!  :)
(In reply to Kyle Lahnakoski [:ekyle] from comment #2)
> I hope this extract is just a diff-since-last-extract[1], and in some
> popular format[2] so we do not need to spin up a database to access the
> records.
> 
> [1] I do not mean a fine grained diff; just a list of all records that have
> had some value change since last time; hopefully it will include all the
> related records (details, notes and stars) for completness.
> [2] like JSON!!  :)

The proposal is to expose a full database that you can access via the standard mysql client. Everything should be included except for security-sensitive stuff.
Depends on: treeherder-heroku
For completeness, the various options for exporting MySQL data from Treeherder are:
1) Periodic/one-off .sql dump to an archive on S3
2) Periodic/one-off creation of a standalone RDS instance based on a recent prod RDS snapshot
3) A read-only RDS replica that's always up to date with production (ie this bug)

For 1-2 just dropping/excluding sensitive tables makes the most sense to me.

For 3, I think the best option is just to limit access using MySQL user GRANTs:
https://dev.mysql.com/doc/refman/5.6/en/grant.html
https://dev.mysql.com/doc/refman/5.6/en/privileges-provided.html

Permissions can be given either per-DB, per-table or even per-column.

At a minimum, the sensitive tables are (someone will want to double-check):

treeherder.auth_user (due to the password field, though we don't use it)
treeherder.credentials (the `secret` field)
treeherder.django_admin_log (the `object_repr` field, if it were ever to contain eg a full representation of any secret, which it doesn't so far for the Credentials model)
treeherder.django_session (session_key and session_data fields)
mysql.<some of, probably best to not grant any>
Flags: needinfo?(emorley)
For example:

GRANT SELECT ON treeherder.foo TO 'someuser'@'%' REQUIRE SSL;
GRANT SELECT (column_bar) ON treeherder.baz TO 'someuser'@'%' REQUIRE SSL;

The REQUIRE SSL is very important.

MySQL doesn't support exclusion rules, so everything has to be whitelisted explicitly (though probably a good thing, since it prevent us accidentally allowing a new table to slip through). However to speed things up, the grant list can be generated using something like:
http://stackoverflow.com/a/16818441

We could check the resultant SQL into the repository for reference, or even run as part of deploy.
Also, if you wanted to help with the Terraform parts, it should be mostly a copy-paste of the existing prod entry:
https://github.com/mozilla-platform-ops/devservices-aws/blob/abd9ffe24e0a8ad6fb928eb981537dd17d776bdb/treeherder/rds.tf#L268-L293
...except using `replicate_source_db` set to `treeherder-prod` and removing `username`, `password` etc. See:
https://www.terraform.io/docs/providers/aws/r/db_instance.html
(In reply to Ed Morley [:emorley] from comment #4)
> For completeness, the various options for exporting MySQL data from
> Treeherder are:
> 1) Periodic/one-off .sql dump to an archive on S3
> 2) Periodic/one-off creation of a standalone RDS instance based on a recent
> prod RDS snapshot
> 3) A read-only RDS replica that's always up to date with production (ie this
> bug)
> 
> For 1-2 just dropping/excluding sensitive tables makes the most sense to me.
> 
> For 3, I think the best option is just to limit access using MySQL user
> GRANTs:
> https://dev.mysql.com/doc/refman/5.6/en/grant.html
> https://dev.mysql.com/doc/refman/5.6/en/privileges-provided.html
> 
> Permissions can be given either per-DB, per-table or even per-column.
> 
> At a minimum, the sensitive tables are (someone will want to double-check):
> ...

I think it makes sense to just leave out any tables which deal with administrativia or bookkeeping. However, we do want the auth_user table (or at least the 'id' and 'email' columns), as that's how we identify who made the various classifications. I guess that leaves this list of tables to mirror:

auth_user  # only 'id' and 'email' columns
bug_job_map
bugscache
build_platform
classified_failure
datasource
exclusion_profile
exclusion_profile_exclusions
failure_classification
failure_line
failure_match
job
job_detail
job_duration
job_exclusion
job_group
job_log
job_note
job_type
machine
machine_platform
matcher
option
option_collection
performance_alert
performance_alert_summary
performance_bug_template
performance_datum
performance_framework
performance_signature
product
reference_data_signatures
repository
repository_group
runnable_job
text_log_error
text_log_step
text_log_summary
text_log_summary_line
user_exclusion_profile

I think for now we can just create one user ('th_user') and give the password out selectively to internal staff who might want to use this db (for now I think that would just be myself, ekyle and whoever would set up the redash instance)

:fubar, is this + emorley's comments enough information for you to set this up in the next week or so?
Flags: needinfo?(klibby)
Whilst we were on SCL3 we didn't have the ability to do some of the tasks that were really within the app developer / dev-ops scope, so had to ask Kendall to do them. Now we're on Heroku we have the power to them ourselves, so we both can (and should) move that dividing line between roles.

Apart from the default user on RDS, all other users are not configured via AWS/Terraform, but standard MySQL commands.

As such, I think the ask from fubar needs to be constrained to:
* create a terraform config for a MySQL read replica of the prod DB
* apply that config

And then it's up to us to:
* figure out SQL for creating a user with those permissions (this is pretty much done with the above)
* get review on that SQL
* apply that SQL to the MySQL master using the existing th_admin account
* check that the read only user cannot access or write to things they shoudn't 
* give the credentials to whomever needs them
What type of replication is used to make the mirror?  Forgive if I show my age, but there used to be SQL statement replication (common) and binary replication (uncommon, or hard).  If it is the former, please be sure there are no SQL updates to master that contain dynamic values; the SQL replicated to the mirror will result in different data. No auto-increment columns, and definitely no mirror-corrupting CURDATE() (oh the stories!!).

https://dev.mysql.com/doc/refman/5.7/en/replication-features-auto-increment.html
By default RDS uses MIXED format for binlogs, which handles this fine:
http://dev.mysql.com/doc/refman/5.6/en/binary-log-mixed.html

The old SCL3 DB nodes also used MIXED, and we ran a master-slave setup fine there too.
There are enough mentions of 'sensitive data' in this bug that I've inclined ask for an RRA with infrasec (or however our actual sec rep is these days) before moving ahead. They're quite quick (~30min), and I'd like to know what data we're worried about and implications of leaks. Thoughts?  
https://wiki.mozilla.org/Security/Risk_management/Rapid_Risk_Assessment

Setting up a read replica RDS instance should be pretty easy, though, especially as I'm poking around working on 1309874 and 1309874.
Flags: needinfo?(klibby)
Yeah that seems fair. Adjusting bug summary to reflect using grants not sanitisation (comment 4). 

Hmm though I've been wondering how we're going to handle breaking schema changes in the future? ie I'd prefer to not slow down development velocity, so no more than notifying people prior to changing, unlike our API where we can use versioning. Is this going to be fine for people using this? It seems like we need confirmation that that's the case first?
Component: Treeherder → Treeherder: Infrastructure
Summary: Create a sanitized, read-only mirror of the treeherder production database → Allow external tools partial access to a read-only replica of the prod Treeherder DB
I think it is fair to demand clients of this mirror all know the schema can change without notice. I am building a client that reviews the `information.schema` before it pulls data: 

* information_schema.tables 
* information_schema.table_constraints
* information_schema.key_column_usage 
* information_schema.columns

Is there a version number in the TH database that tracks the migrations?
(In reply to Kyle Lahnakoski [:ekyle] from comment #13)
> I think it is fair to demand clients of this mirror all know the schema can
> change without notice. I am building a client that reviews the
> `information.schema` before it pulls data: 
> 
> * information_schema.tables 
> * information_schema.table_constraints
> * information_schema.key_column_usage 
> * information_schema.columns
> 
> Is there a version number in the TH database that tracks the migrations?

Migrations are tracked in treeherder.django_migrations, though that doesn't tell you what changed, only *that* something changed.

And yeah, I agree that we should just provide this data on an "as-is" basis. That said, I don't anticipate a huge amount of churn for the mature parts of the database (at least once the datasource migration is complete).
I believe the following SQL should be sufficient to create the user with the right permissions. Obviously 'mysecurepassword' should be replaced with something else. ;)
Comment on attachment 8806121 [details]
SQL statements for creating user + setting permissions

I do not know if the sensitive tables will be scrubbed from the mirror, if so it may be simpler to allow all reads:

> GRANT SELECT ON treeherder.* TO 'thuser'

I would also like to read the metadata, so I may understand how the data relates:

> GRANT SELECT ON information_schema.tables TO 'thuser'
> GRANT SELECT ON information_schema.table_constraints TO 'thuser'
> GRANT SELECT ON information_schema.key_column_usage TO 'thuser'
> GRANT SELECT ON information_schema.columns TO 'thuser'

Thanks!
(In reply to Kyle Lahnakoski [:ekyle] from comment #16)
> Comment on attachment 8806121 [details]
> SQL statements for creating user + setting permissions
> 
> I do not know if the sensitive tables will be scrubbed from the mirror, if
> so it may be simpler to allow all reads:
> 
> > GRANT SELECT ON treeherder.* TO 'thuser'

The way things currently are, no, they will not be scrubbed. So I think we should be selective about what we make viewable. Modifying these rules later should not be a big deal (any treeherder dev should be able to do it)

> I would also like to read the metadata, so I may understand how the data
> relates:
> 
> > GRANT SELECT ON information_schema.tables TO 'thuser'
> > GRANT SELECT ON information_schema.table_constraints TO 'thuser'
> > GRANT SELECT ON information_schema.key_column_usage TO 'thuser'
> > GRANT SELECT ON information_schema.columns TO 'thuser'

The tables in information schema seem to be readable by default in my testing.
Attached file Updated SQL with REQUIRE SSL (obsolete) —
Forgot to require ssl for the various grants
Attachment #8806121 - Attachment is obsolete: true
(In reply to Kendall Libby [:fubar] from comment #11)
> There are enough mentions of 'sensitive data' in this bug that I've inclined
> ask for an RRA with infrasec (or however our actual sec rep is these days)
> before moving ahead. They're quite quick (~30min), and I'd like to know what
> data we're worried about and implications of leaks. Thoughts?  
> https://wiki.mozilla.org/Security/Risk_management/Rapid_Risk_Assessment
> 
> Setting up a read replica RDS instance should be pretty easy, though,
> especially as I'm poking around working on 1309874 and 1309874.

I've created an RRA template and shared it with you, emorley, and jgriffin.

I've also talked with jgriffin, and I don't think we need to wait on going through the full process to proceed with a prototype. It's not like we're exposing a SQL server to the whole world: we're just configuring an internal, authenticated service to a very small number of trusted users @ Mozilla.
(In reply to William Lachance (:wlach) from comment #19)
> 
> I've created an RRA template and shared it with you, emorley, and jgriffin.

Awesome, thank you!

> I've also talked with jgriffin, and I don't think we need to wait on going
> through the full process to proceed with a prototype. It's not like we're
> exposing a SQL server to the whole world: we're just configuring an
> internal, authenticated service to a very small number of trusted users @
> Mozilla.

Agreed; today's a loss to meetings, but I'll pick up this and the other terraform bits for treeherder tomorrow.
Do we want the read replica to be multi-az like the source? If we're going to have other random consumers it seems likely, but want to verify as it does make the db instance more expensive.
(In reply to Kendall Libby [:fubar] from comment #21)
> Do we want the read replica to be multi-az like the source? If we're going
> to have other random consumers it seems likely, but want to verify as it
> does make the db instance more expensive.

So my understanding of what we're using to back the RDS database is a bit sketchy, but my understanding is that we're using a beefier instance type on production vs. stage/prototype. I honestly have noticed any problems with the performance of the db for stage/prototype, so let's just start with that level at first. We can always upgrade later if there's a need.
(In reply to Kendall Libby [:fubar] from comment #21)
> Do we want the read replica to be multi-az like the source? If we're going
> to have other random consumers it seems likely, but want to verify as it
> does make the db instance more expensive.

I don't think any need for multi-AZ, it can also be a db.m4.xlarge rather than the db.m4.2xlarge of prod. We can always change this later if needed (destroying and re-creating is fine for that, given it's just a replica).
Ok, I've spun up treeherder-prod-ro as single-AZ, m4.xl. but otherwise the same.
I updated the script for two changes:

* Changed the name of the readonly user to th_user
* Granted access to the new "push" and "commit" tables

I ran this on treeherder production (with a different password, obv) and can login into the mirror as th_user just fine from here. I think we're done here. Thanks all!
Attachment #8806350 - Attachment is obsolete: true
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Thank you for generating the SQL.

However in the RRA we said:
1) We'd code review any permissions SQL
2) We'd possibly check it into the repo, so we could tell what was the most recently applied permissions
3) We'd use one set of credentials per consumer (so need different usernames, and need to come up with a naming scheme; I think we should move away from the th_blah, since the DB is all ours)

In addition, I see there are now two duplicate users on prod - `thuser` and `th_user`.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Also we need to come up with some documentation that will ensure that anyone who receives these credentials:
1) Doesn't accidentally leak them by not using SSL (by missing CLI options or using the wrong version of the C API backend to their client)
2) Uses the correct read-only replica hostname, and not the master, since the credentials will work on either
I've disabled thuser and th_user for now, until we've completed the above.
Sorry, I didn't recall those takeaways from the RRA, and they weren't in the final document. They seem pretty reasonable though, so let's just do them.

(In reply to Ed Morley [:emorley] from comment #26)
> Thank you for generating the SQL.
> 
> However in the RRA we said:
> 1) We'd code review any permissions SQL

Ok, well, can you tell me whether it's ok? 

> 2) We'd possibly check it into the repo, so we could tell what was the most
> recently applied permissions
> 3) We'd use one set of credentials per consumer (so need different
> usernames, and need to come up with a naming scheme; I think we should move
> away from the th_blah, since the DB is all ours)

Should I create a script under bin/ to generate the sql (with username and password as input)?

For documentation I feel like the mana's the best place, though it looks like the treeherder page is pretty out of date. Thoughts?
Flags: needinfo?(emorley)
(In reply to Ed Morley [:emorley] from comment #27)
> Also we need to come up with some documentation that will ensure that anyone
> who receives these credentials:
> 1) Doesn't accidentally leak them by not using SSL (by missing CLI options
> or using the wrong version of the C API backend to their client)
> 2) Uses the correct read-only replica hostname, and not the master, since
> the credentials will work on either

This sounds dangerous.  I suggest we break the mirrorness... 

a) I hope the mysql credentials on the read-only image can be separate from those on production; after all, they are part of different schema (the mysql schema), so should not be mirrored.
b) The master TH users, and their authentication data, should be removed from the read-only mirror even though no user has permission to access them
c) It would be nice to have the RO usernames be email addresses, if only to have a contact name.

I imagine the master TH application would be periodically removing the sensitive data from the mirror, using it own secured credentials.
(In reply to Kyle Lahnakoski [:ekyle] from comment #30)
> (In reply to Ed Morley [:emorley] from comment #27)
> > Also we need to come up with some documentation that will ensure that anyone
> > who receives these credentials:
> > 1) Doesn't accidentally leak them by not using SSL (by missing CLI options
> > or using the wrong version of the C API backend to their client)
> > 2) Uses the correct read-only replica hostname, and not the master, since
> > the credentials will work on either
> 
> This sounds dangerous.  I suggest we break the mirrorness... 
> 
> a) I hope the mysql credentials on the read-only image can be separate from
> those on production; after all, they are part of different schema (the mysql
> schema), so should not be mirrored.

You can't write to the read-only miror because it's read-only.

> b) The master TH users, and their authentication data, should be removed
> from the read-only mirror even though no user has permission to access them

We already discussed this in the RRA. This is an internal system to be used by a very small number of trusted users (activedata and redash, basically). Given that, and the fact that Treeherder data is not particularly sensitive, it was determined that w don't need to spend a ton of effort on additional mitigations.

> c) It would be nice to have the RO usernames be email addresses, if only to
> have a contact name.

There are only two users, 'redash' and 'activedata'. I think that's enough. I don't want to user email addresses because the owner of a service might change.
(In reply to William Lachance (:wlach) from comment #31)
> (In reply to Kyle Lahnakoski [:ekyle] from comment #30)
> > This sounds dangerous.  I suggest we break the mirrorness... 
> > 
> > a) I hope the mysql credentials on the read-only image can be separate from
> > those on production; after all, they are part of different schema (the mysql
> > schema), so should not be mirrored.
> 
> You can't write to the read-only miror because it's read-only.

We call it a read-only mirror, but something is writing to it. Furthermore, I am referring to the `mysql` schema, not the `treeherder` schema.  The `mysql` schema holds the mysql users; the master and mirror can have different users.

> > b) The master TH users, and their authentication data, should be removed
> > from the read-only mirror even though no user has permission to access them
> 
> We already discussed this in the RRA. This is an internal system to be used
> by a very small number of trusted users (activedata and redash, basically).
> Given that, and the fact that Treeherder data is not particularly sensitive,
> it was determined that w don't need to spend a ton of effort on additional
> mitigations.

In that case, we should not be worried about "[credentials] doesn't accidentally leak them by not using SSL"; we can restrict the inbound IPs to the known locations of the few users. Even if the RO password leaks the attack surface is small.

> > c) It would be nice to have the RO usernames be email addresses, if only to
> > have a contact name.
> 
> There are only two users, 'redash' and 'activedata'. I think that's enough.
> I don't want to user email addresses because the owner of a service might
> change.

Good point. ActiveData is easy enough to find, but the "redash" user seems too generic. May I suggest inventing a project name, using it as the username, and making a wiki page for contact info (under https://wiki.mozilla.org/EngineeringProductivity/Projects)?  Or maybe name the user "bug1279580"?
(In reply to William Lachance (:wlach) from comment #29)
> Sorry, I didn't recall those takeaways from the RRA, and they weren't in the
> final document. They seem pretty reasonable though, so let's just do them.

No problem :-)
(We'd mentioned some in bug comments prior, and then very briefly in the last few minutes of the meeting, but I don't think they were written down by Tristan)

> Should I create a script under bin/ to generate the sql (with username and
> password as input)?

The main use cases for this script are to simplify/reduce margin for error when:
1) adding additional read-only users
2) adding GRANTs in response to new Django ORM tables
3) recreating the RO mirror from scratch (only relevant if we were to go via the route of only adding these users to the RO mirror)

I think it may be useful to commit the final SQL to the repo (rather than only the script to generate it), since it's easier to review. But we could always add a comment (or script) saying how to generate it using CONCAT() etc (which can be run whenever we add new Django tables). Open to ideas though.

I'll add some comments to the current attachment.


> For documentation I feel like the mana's the best place, though it looks
> like the treeherder page is pretty out of date. Thoughts?

When I finally get around to finishing the Heroku infra docs, I'll be adding them to read the docs, so I think this should be in RTD too. (At most the Mana page will be a copy-paste of parts of RTD just for MOC.)


--


(In reply to Kyle Lahnakoski [:ekyle] from comment #30)
> I imagine the master TH application would be periodically removing the
> sensitive data from the mirror, using it own secured credentials.

MySQL replication cannot be made to work like that (especially not on RDS). If you're interested in reading more, these are prety useful reads:
http://dev.mysql.com/doc/refman/5.6/en/replication.html
https://www.percona.com/blog/2013/01/09/how-does-mysql-replication-really-work/
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html

> We call it a read-only mirror, but something is writing to it. Furthermore,
> I am referring to the `mysql` schema, not the `treeherder` schema.  The
> `mysql` schema holds the mysql users; the master and mirror can have
> different users.

Yes the MySQL replication process requires that writes be made. When we say read only, we mean the `read_only` variable is set, however this doesn't affect the replication process (or any user who has the SUPER permissions, however we don't get that on RDS).

By default on RDS, the `read_only` variable is set to `{TrueIfReplica}`. We could override that and set it to `0` regardless though, via the existing `treeherder` parameter group.

Pros:
* Can add users just to the RO mirror

Cons:
* Means anyone who uses the standard RW user could accidentally make changes to the RO mirror and mean it's no longer in sync
* Means any time we delete+recreate the RO mirror, we have to manually re-add the RO users again (since they don't exist on the master)

I'm open to thoughts either way.

> In that case, we should not be worried about "[credentials] doesn't
> accidentally leak them by not using SSL"; we can restrict the inbound IPs to
> the known locations of the few users. Even if the RO password leaks the
> attack surface is small.

We can't limit access to certain IPs on the master, due to the way Heroku works (we discussed this in the RRA), so if the RO user is on both it doesn't help.

Even if we do add the RO users to only the RO mirror (and then limit IP access to that, if it's even possible to do a split VPC across master and replication slave using RDS), I disagree that it's ok to plan to leak credentials due to incorrect configuration. It takes very little effort to connect securely, so we may as well get people to do so. Defence in depth doesn't mean we can just ignore half of the defences that were agreed upon both before and in the RRA.
Flags: needinfo?(emorley)
Comment on attachment 8811907 [details]
Updated name of readonly user, grant access to push table

>CREATE USER 'th_user' IDENTIFIED BY 'mysecurepassword';

Not much help for now, but once we're on MySQL 5.7 we can move the `REQUIRE SSL` to the CREATE USER command to avoid duplication:
http://mysqlserverteam.com/building-a-better-create-user-command/

For the password we generate for them, I think we should make it reasonably long (just no longer than 32 characters, since that doesn't work with replication according to MySQL docs).

>GRANT SELECT (id, email) ON treeherder.auth_user to 'th_user' REQUIRE SSL;

I'm happy to also add `username` here.

>GRANT SELECT ON treeherder.bug_job_map to 'th_user' REQUIRE SSL;
...
>GRANT SELECT ON treeherder.user_exclusion_profile to 'th_user' REQUIRE SSL;

These look good to me :-)

To generate up to date lists of the table based GRANTs, we can use something like:

SELECT CONCAT('GRANT SELECT ON ', table_schema, '.', table_name, ' to ''user'' REQUIRE SSL;') AS grant_stmt
FROM information_schema.TABLES
WHERE table_schema = 'treeherder'
AND table_name NOT REGEXP 'django_|auth_|credentials|corsheaders_|task_set_meta';

>FLUSH PRIVILEGES;

Flush privileges is only required if the `mysql.user` table was modified directly rather than by using GRANT statements.
(In reply to Ed Morley [:emorley] from comment #33)
> I disagree that it's ok to plan to leak
> credentials due to incorrect configuration. It takes very little effort to
> connect securely, so we may as well get people to do so. Defence in depth
> doesn't mean we can just ignore half of the defences that were agreed upon
> both before and in the RRA.

I do not recall suggesting "a plan to leak" nor "ignoring half the defenses". If you got that impression from my use of "should not be worried", it is unfortunate you interpreted as "do not care and take no action"; I would have hoped you would have interpreted it as "relatively unlikely to be exploited while passwords were reissued, and the system re-secured".
(In reply to Kyle Lahnakoski [:ekyle] from comment #35)
> I would have hoped you would have interpreted it as "relatively
> unlikely to be exploited while passwords were reissued, and the system
> re-secured".

Sadly it's unlikely to happen like that though.

Either in automation, or when people are making one-off connections using the CLI/MySQLWorkbench, if the SSL options are omitted, the user may just enable them and try again, not realising they've leaked the credentials.

Worse, if the user if using an insecure version of libmysqlclient (this includes even the latest 5.5.x and 5.6.x releases), their connection can be silently MITMed without them knowing (this has been wontfixed by Oracle).

In both of these cases we wouldn't know to rotate the credentials.
Comment on attachment 8813386 [details] [review]
[treeherder] wlach:1311468-docs > mozilla:master

This only covers the administration portion.

I wasn't sure how to talk about the specifics of how a user might use these credentials (and the SSL certificates correctly). I figure we can probably just do that on a case-by-case basis for now, given that we only have two consumers of this service.
Attachment #8813386 - Flags: review?(emorley)
I think the wrong file was git added to the PR?
Comment on attachment 8813386 [details] [review]
[treeherder] wlach:1311468-docs > mozilla:master

(pending the git add of admin.rst :-))
Attachment #8813386 - Flags: review?(emorley)
Comment on attachment 8813386 [details] [review]
[treeherder] wlach:1311468-docs > mozilla:master

Oops, fixed. :)
Attachment #8813386 - Flags: review?(emorley)
Attachment #8813386 - Flags: review?(emorley) → review+
Can we resolve this bug? I'm personally fine with things the way they are for now, but if others believe we should do other stuff, they can always file followups...
Assignee: nobody → wlachance
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.