1255034 - Set up admin back-end for Kinto-Writer (VPN)

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Daniel Maher [:phrawzty]]

The OneCRL stack is driven by Kinto and is meant to provide restricted access to an administration panel (provided by Kinto).  At a minimum, this will involve modifications to the associated Cloudformation templates (et al.), as well as associated manipulations to the LDAP database (new group, add users, etc)[0].


[0] Or modifications made to an existing group; this will need to be verified.

Comment 1

[Tarek Ziadé (:tarek)]

What is the ETA for this change ?

Comment 2

[Daniel Maher [:phrawzty]]

> What is the ETA for this change ?

I've never done this in the SVCOPS context before, so I'm not sure exactly how much work is required. Also, we have to rely on IT for some portions of the work, and I absolutely don't want to make predictions or statements about their workload or timelines.

Due to my PTO and French Holidays over the next seven work days, I'd rather play it safe and aim for the week of 18 July.  I'll try to narrow down that window as work progresses forward.

Comment 3

[Tarek Ziadé (:tarek)]

So we're in August now, when do you think we can have it ? thank you

Comment 4

[Daniel Maher [:phrawzty]]

(In reply to Tarek Ziadé (:tarek) from comment #3)
> So we're in August now, when do you think we can have it ? thank you

Hi Tarek,

I've prepared PRs[0][1] and will be reviewing them with Relud when he comes online today. Once these are r+ and tested, we'll be able to spin up the new stacks and verify the resulting Elastic IP. Once that EIP is known, we can file the appropriate bugs with IT in order to have the VPN modifications made.

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1255034
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1284411

Comment 5

[Daniel Maher [:phrawzty]]

We've devised a methodology and management process that will allow newly-deployed "Admin" stacks to be spun up and promoted without incurring downtime.  It was necessary to develop this technique in order to ensure that the Kinto-Writer node remained functional even during a new stack deployment, since various bits of automation depend on it being available.

We've got a POC in place, and while it still requires some refinement (notably, the DNS migration is still manual), it has allowed us to set up a pool of reserved EIPs ahead of time, so we can move forward with the IT side of the VPN setup.

Comment 6

[Daniel Maher [:phrawzty]]

After some further testing and refinement, I'm happy to report that the new Kinto-Writer infra code has been merged into our main branch[0].  Along with the improvements to the management cycle noted above, the DNS promotion works automatically, which is a small but meaningful improvement.

It's important to note that this code affects the *Writer* node only (not the webheads), and that it's been tested in Stage only.  Because of the changes, the transition between formats necessitates a small service outage - this can be managed when the time comes to go to Prod.

Now we wait for bug 1294414.

[0] https://github.com/mozilla-services/svcops/pull/1166

Comment 7

[Daniel Maher [:phrawzty]]

Note that the associated modifications to Puppet have been merged[0] as well; however, it's important to point out that if we try to go to Prod with this config, it *will* fail, since the appropriate certs haven't been generated. I'll get on that straight away.

[0] https://github.com/mozilla-services/puppet-config/pull/2167

Comment 8

[Daniel Maher [:phrawzty]]

(In reply to Daniel Maher [:phrawzty] from comment #7)
> well; however, it's important to point out that if we try to go to Prod with
> this config, it *will* fail, since the appropriate certs haven't been
> generated. I'll get on that straight away.

The remaining small modifications necessary to push a new Prod stack have been merged in Puppet, Ansible, and the secrets repo. (yay!)

The next step is to set up the EIP pool that Prod will ultimately draw from. Once that stack has been instantiated, we can set up the IT bug for the VPN routes.  Once that's in place, we can move forward with a Prod push.  For the first push, we'll avoid the "promote" step (read: switch DNS) in order to make sure things look good before it becomes public-facing.

Comment 9

[Daniel Maher [:phrawzty]]

This bug has been resolved for some time now (yay!); closing as successful. :)