Set up admin back-end for Kinto-Writer (VPN)

RESOLVED FIXED

Status

Cloud Services
Operations
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: phrawzty, Assigned: phrawzty)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Assignee)

Description

2 years ago
The OneCRL stack is driven by Kinto and is meant to provide restricted access to an administration panel (provided by Kinto).  At a minimum, this will involve modifications to the associated Cloudformation templates (et al.), as well as associated manipulations to the LDAP database (new group, add users, etc)[0].


[0] Or modifications made to an existing group; this will need to be verified.
(Assignee)

Updated

2 years ago
Depends on: 1255776
No longer depends on: 1248898
(Assignee)

Updated

2 years ago
Depends on: 1264995
(Assignee)

Updated

2 years ago
Summary: Set up admin back-end for OneCRL (Kinto) → Set up admin back-end for Kinto-Writer (VPN)
(Assignee)

Updated

2 years ago
Blocks: 1284411
What is the ETA for this change ?
Flags: needinfo?(dmaher)
(Assignee)

Comment 2

2 years ago
> What is the ETA for this change ?

I've never done this in the SVCOPS context before, so I'm not sure exactly how much work is required. Also, we have to rely on IT for some portions of the work, and I absolutely don't want to make predictions or statements about their workload or timelines.

Due to my PTO and French Holidays over the next seven work days, I'd rather play it safe and aim for the week of 18 July.  I'll try to narrow down that window as work progresses forward.
Flags: needinfo?(dmaher)
So we're in August now, when do you think we can have it ? thank you
Flags: needinfo?(tblow)
Flags: needinfo?(dmaher)

Updated

2 years ago
Flags: needinfo?(tblow)
(Assignee)

Comment 4

2 years ago
(In reply to Tarek Ziadé (:tarek) from comment #3)
> So we're in August now, when do you think we can have it ? thank you

Hi Tarek,

I've prepared PRs[0][1] and will be reviewing them with Relud when he comes online today. Once these are r+ and tested, we'll be able to spin up the new stacks and verify the resulting Elastic IP. Once that EIP is known, we can file the appropriate bugs with IT in order to have the VPN modifications made.

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1255034
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1284411
Flags: needinfo?(dmaher)
(Assignee)

Updated

2 years ago
Depends on: 1294414
(Assignee)

Comment 5

2 years ago
We've devised a methodology and management process that will allow newly-deployed "Admin" stacks to be spun up and promoted without incurring downtime.  It was necessary to develop this technique in order to ensure that the Kinto-Writer node remained functional even during a new stack deployment, since various bits of automation depend on it being available.

We've got a POC in place, and while it still requires some refinement (notably, the DNS migration is still manual), it has allowed us to set up a pool of reserved EIPs ahead of time, so we can move forward with the IT side of the VPN setup.
(Assignee)

Comment 6

2 years ago
After some further testing and refinement, I'm happy to report that the new Kinto-Writer infra code has been merged into our main branch[0].  Along with the improvements to the management cycle noted above, the DNS promotion works automatically, which is a small but meaningful improvement.

It's important to note that this code affects the *Writer* node only (not the webheads), and that it's been tested in Stage only.  Because of the changes, the transition between formats necessitates a small service outage - this can be managed when the time comes to go to Prod.

Now we wait for bug 1294414.

[0] https://github.com/mozilla-services/svcops/pull/1166
(Assignee)

Comment 7

2 years ago
Note that the associated modifications to Puppet have been merged[0] as well; however, it's important to point out that if we try to go to Prod with this config, it *will* fail, since the appropriate certs haven't been generated. I'll get on that straight away.

[0] https://github.com/mozilla-services/puppet-config/pull/2167
(Assignee)

Comment 8

2 years ago
(In reply to Daniel Maher [:phrawzty] from comment #7)
> well; however, it's important to point out that if we try to go to Prod with
> this config, it *will* fail, since the appropriate certs haven't been
> generated. I'll get on that straight away.

The remaining small modifications necessary to push a new Prod stack have been merged in Puppet, Ansible, and the secrets repo. (yay!)

The next step is to set up the EIP pool that Prod will ultimately draw from. Once that stack has been instantiated, we can set up the IT bug for the VPN routes.  Once that's in place, we can move forward with a Prod push.  For the first push, we'll avoid the "promote" step (read: switch DNS) in order to make sure things look good before it becomes public-facing.
(Assignee)

Updated

2 years ago
Depends on: 1297444
Depends on: 1302052
(Assignee)

Updated

2 years ago
Depends on: 1302356
(Assignee)

Updated

2 years ago
Depends on: 1302418
(Assignee)

Comment 9

2 years ago
This bug has been resolved for some time now (yay!); closing as successful. :)
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.