1378352 - Create bugzilla and Kinto accounts for OneCRL / Preload automations

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Mark Goodwin [:mgoodwin]]

The tooling for populating OneCRL with CCADB entries is working well enough for deployment - HPKP and HSTS preload tooling will follow soon after. We will need accounts for these automations to use for read / write access to the kinto collections as well as use of the bugzilla API (for creating bugs and making attachments to them).

Please advise on how to proceed.

Comment 1

[Julien Vehent]

There are various ways to go about this. We could create an LDAP account for this automation tool, but integration might be difficult due to VPN and planned moved to Auth0. Another option, which might be preferable, would be to run the automation alongside the Kinto service, maybe as a lambda function, and grant it an API token to the API.

needinfo :natim who may have some thoughts on the best way to implement this.

Mark, a few questions:
1. Is the automation lambda-ifiable?
2. Does it require special credentials to talk to CCADB?
3. Should we have two people in review of the changes created by the automation, to keep the two-man rule? (and can Kinto accommodate that?)

Comment 2

[Rémy Hubscher (:natim)]

From your messages, it seems to me that knowing the person who asked for changes in CCADB should be reflected somehow in Kinto's history of changes.

- Are users authenticated with Auth0 in CCADB? And if yes, should we consider authenticating them with their OAuth credentials in Kinto too?
- What triggers a change to Bugzilla and Kinto in CCADB? (Is it a user action or a periodic task?)
- Is the review process still happening in Kinto or is now in CCADB or through Bugzilla?

Technically speaking, we have a LDAP user configured for Kinto cloud-operations that we can configure if you write a lambda (we can also create a new one if necessary).

However if we do use them we won't be able to tell if the person who asked for a review in CCADB is the same one accepting the changes in Kinto.

It is possible to trigger the lambda function from an HTTP web hook or a periodic task.

Comment 3

[Julien Vehent]

I was chatting with the SHIELD folks, who have a similar problem, and we decided to take a different route that I think could apply to OneCRL as well: instead of having a lambda job that writes into the OneCRL Kinto DB, how about having an external service that prepares records the Kinto admin can pull in for review?

                                  KINTO SECURITY BOUNDARY
                               +-------------------------+
                               |                         |
                 +------+      | +------+      +------+  |
+-----+          |OneCRL|      | |OneCRL|      |OneCRL|  |
|CCADB|<---------+record|<-------+Kinto +----->|Kinto |  |
+-----+          |prep. |      | |Admin |      |  DB  |  |
                 +------+      | +------+      +------+  |
                               |                         |
                               +-------------------------+

This approach would allow mark to implement his record preparation service without having to worry about the security impact to OneCRL. It also wouldn't break the two-man rule, as one authorized user would pull in the pending records, and another one would approve the change. We also wouldn't need a kinto accounts, though maybe a bugzilla one?.

Comment 4

[Rémy Hubscher (:natim)]

What puzzles me in this scheme is the fact that we add business logic to Kinto.

Kinto is a database, we could create a plugin to handle that but if we could keep it as a simple database (that handle writes and reads requests only) it would be better.

Comment 5

[Mark Goodwin [:mgoodwin]]

I'd prefer a solution where we keep the logic in the tool (rather than Kinto).

(In reply to Julien Vehent [:ulfr] from comment #1)
> Mark, a few questions:
> 1. Is the automation lambda-ifiable?

Yes. I've modified your autograph monitor to wrap the salesforce2OneCRL tool and had it successfully creating entries.

> 2. Does it require special credentials to talk to CCADB?

No. The data is public. The required credentials are for Kinto (stage and production) and bugzilla (stage and production).

> 3. Should we have two people in review of the changes created by the
> automation, to keep the two-man rule? (and can Kinto accommodate that?)

Yes. We already have tools and processes in place to ensure this: See https://docs.google.com/a/mozilla.com/document/d/1JEMNBBpIuBFofyZtZritnCSL8LAt04CV-isYQ_hOcWY/edit?usp=sharing

Comment 6

[Mark Goodwin [:mgoodwin]]

I've spoken to bobm and ulfr about this; we're going to go with the original design and have the ccadb import tooling write directly to the staging bucket - but we still need some credentials. Bobm seemed to think we could create special credentials for system accounts; can you arrange the creation of these? If not, who do I need to speak to?

Comment 7

[Mark Goodwin [:mgoodwin]]

Also, we need a bugzilla API key for an account that can create new bugs and add attachments. How do we do this?

Comment 8

[Rémy Hubscher (:natim)]

Yes we have created LDAP accounts that are used by our LAMBDA functions. I guess we could also ask for a LDAP account for your use-case.

Jason do you know who we should ask for those credentials?

Mark do you need one for editing and one for reviewing?

Comment 9

[Mark Goodwin [:mgoodwin]]

(In reply to Rémy Hubscher (:natim) from comment #8)
> Mark do you need one for editing and one for reviewing?

We just need a single account; it needs to create and request review on the certificates collection of the staging bucket and the pins collection of the pinning-staging bucket.

We'd need an equivalent account (preferably separate) for the same operations in the staging environment.

Comment 10

[Dylan Hardison [:dylan] (he/him)]

(In reply to Mark Goodwin [:mgoodwin] from comment #7)
> Also, we need a bugzilla API key for an account that can create new bugs and
> add attachments. How do we do this?

1. Create an account using an email you control
2. Generate an API key,
3. file a bmo :: admin ticket asking an admin to give it access to whatever groups required.
   We will rename it something (make a suggestion, but something like [e-mail])
   We can also reset the password so that nobody can login as that user (but the API key will still work)
4. Add an entry for the bot here: https://wiki.mozilla.org/BMO/Bot_Registry so that if it misbehaves we know who to contact.

Comment 11

[Jason Thomas [:jason]]

(In reply to Rémy Hubscher (:natim) from comment #8)
> Yes we have created LDAP accounts that are used by our LAMBDA functions. I
> guess we could also ask for a LDAP account for your use-case.
> 
> Jason do you know who we should ask for those credentials?
> 
> Mark do you need one for editing and one for reviewing?

We should ask :jabba if we need new accounts. Or we could probably reuse the ldap accounts we use today for lambda.

Comment 12

[Mark Goodwin [:mgoodwin]]

(In reply to Jason Thomas [:jason] from comment #11)
> We should ask :jabba if we need new accounts. Or we could probably reuse the
> ldap accounts we use today for lambda.

If those already need the permission to write to staging/collections/certificates and request review, fine.

Comment 13

[Bob Micheletto [:bobm]]

(In reply to Jason Thomas [:jason] from comment #11)

> We should ask :jabba if we need new accounts. Or we could probably reuse the
> ldap accounts we use today for lambda.

:jabba could you provide us with some guidance here?

Comment 14

[Justin Dow [:jabba]]

There are currently two LDAP accounts that I see that look like what you are referring to: cloudservices_kinto_prod and cloudservices_kinto_reviewer_prod. 

From the LDAP perspective, therse are just simple system accounts with no special permissions or group memberships. I'm not sure what Kinto or OneCRL or CCADB is or how they are related to each other. It'd be up to you whether it's appropriate to re-use these accounts for another thing, or if you'd like me to provision new accounts for this new thing.

Comment 15

[Bob Micheletto [:bobm]]

(In reply to Justin Dow [:jabba] from comment #14)

> From the LDAP perspective, therse are just simple system accounts with no
> special permissions or group memberships. I'm not sure what Kinto or OneCRL
> or CCADB is or how they are related to each other. It'd be up to you whether
> it's appropriate to re-use these accounts for another thing, or if you'd
> like me to provision new accounts for this new thing.

:jason sounds like it's up to you.  What would you like to do?

Comment 16

[Rémy Hubscher (:natim)]

I guess we could use the account cloudservices_kinto_prod for that purpose it is used by lambda functions related to the blocklist form which OneCRL is part.

Comment 17

[Jason Thomas [:jason]]

+1 for using cloudservices_kinto_prod

Comment 18

[Mathieu Leplatre [:leplatrem]]

We'd like to switch the remote settings server to OpenID Connect. There is a conversation going on about enabling the internal accounts feature of Kinto to have dedicated user/token for lambdas, CLI scripts and other out-of-browser use-cases.

Comment 19

[Julien Vehent]

> We'd like to switch the remote settings server to OpenID Connect.

Let's not block mark's deployment on this, we can always move his credentials afterwards.

Comment 20

[J.C. Jones [:jcj] (he/they)]

This task really needs to move forward for CryptoEng.

Comment 21

[Julien Vehent]

We last discussed this with :mgoodwin on March 22. The deployment is assigned to :wezhou (needinfo for status update), and Mark has a few action items:

* Allowing the exceptions.json and configuration file to be specified via AWS environment, and
* Modifying the makefile to allow us to specify tagged releases of salesforce2OneCRL to allow services to make release combinations explicit (and to allow changes to be reverted more easily).


Mark: has that work been completed?

Comment 22

[:wezhou]

The work of setting up the lambda is tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=1397882

Comment 23

[Mathieu Leplatre [:leplatrem]]

>> We'd like to switch the remote settings server to OpenID Connect.
>
> Let's not block mark's deployment on this, we can always move his credentials afterwards.
>

FYI, we started the very first step of the migration to OpenID on STAGE, which consists in enabling Kinto local accounts for scripts and lambdas. See Bug 1467641

Comment 24

[Mathieu Leplatre [:leplatrem]]

We migrated STAGE and PROD to OpenID Connect. 

If you need your lambda that interacts with the servers, you'll have to get a Kinto internal account. Wei knows how to create it, manage secrets etc.

Comment 25

[:wezhou]

As discussed above, we'll be using the existing `cloudservices_kinto_prod` account for this purpose.

Comment 26

[Mathieu Leplatre [:leplatrem]]

> As discussed above, we'll be using the existing `cloudservices_kinto_prod` account for this purpose.

We're good then.