1008872 - Add QA administrators for QA Puppet infrastructure in SCL3

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As agreed on we will have the following people who will be in charge of admin tasks for puppet again in QA:

Henrik Skupin <[e-mail]>
Kamil Jozwiak <[e-mail]>
Andreea Matei <[e-mail]>
Andrei Eftimie <[e-mail]>

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As instructed by Dustin, the steps to be done here are:

You can add them to $admin_users in qa-config.pp, but also make sure their keys are in modules/ssh/manifests/keys.pp.  Be careful with the latter - it's easy to miss a comma there and bring everyone's puppet runs to a grinding halt.  Callek or Rail should be able to review that.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The SSH keys for Andreea and Andrei I got from the commit access bugs (bug 836695 and bug 905979). Sadly I miss the one from Kamil. Not sure if he has setup something so far.

Kamil, please let me know if you have requested commit access meanwhile, as we talked about during the last mozmill-tests merge.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Admins v1

Kamil doesn't seem to be around those days. So I will go ahead and get at least Andreea and Andrei added.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Talked with Marc and we will go ahead with Andreea and Andrei as admins only. If we really need Kamil, we can add him later. For now having us three should totally be enough.

Landed as:
https://hg.mozilla.org/build/puppet/rev/3eb84c83c047 (default)
https://hg.mozilla.org/build/puppet/rev/414806781eb7 (production)

Dustin, is there anything I have to do on puppetmaster to get those settings active? Or does it automatically pull the latest changes into /etc/puppet/production and apply those?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

It should be automatic!

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hah! I missed to change the repository URL to our new one. My bad! :) I will check back tomorrow if it happened.

Andrei, and Andreea please also check if you can login now.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Dustin, looks like that our puppetmaster doesn't pull and update from the production branch. Nothing happened in the last 10h. Can you please have a look, or at least tell me where I would have to look? Is that done via cron or something else? I cannot find an entry.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ah, looks like I have to really update /etc/puppet/update.sh to pull from our new repository. Will do that now.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Adding the keys and user to the puppet config doesn't seem to be enough. When Andrei wants to login, I see the following in the messages log:

> input_userauth_request: invalid user aeftimie

So it looks like Puppet is not automatically creating new user accounts? Would that have to be done manually? Can't this be automated?

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

The puppet runs aren't succeeding on puppetmaster1 due to ssl errors.

I'd have to look at the certificate chain to figure out why that would happen.  You can try running ./puppetize.sh and entering the deploypass again, but if that doesn't work then most likely one of the CA certificates is expired.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Huh, the puppetmaster's CA inventory.txt looks like

V       181013143010Z           01      unknown /CN=puppetmaster1.qa.scl3.mozilla.com/OU=PuppetMasters
V       181013143202Z           02      unknown /CN=puppetmaster1.qa.scl3.mozilla.com
R       181015111442Z   131016114649Z   03      unknown /CN=mm-ub-1204-32-temp.qa.scl3.mozilla.com
R       181015114649Z   140417102908Z   04      unknown /CN=mm-ub-1204-32-temp.qa.scl3.mozilla.com
V       190416102908Z           05      unknown /CN=mm-ub-1204-32-temp.qa.scl3.mozilla.com
R       190416131437Z   140417131521Z   06      unknown /CN=mm-osx-107.qa.scl3.mozilla.com
R       190416131521Z   140422142850Z   07      unknown /CN=mm-osx-107.qa.scl3.mozilla.com
R       190421142850Z   140422143124Z   08      unknown /CN=mm-osx-107.qa.scl3.mozilla.com
V       190421143125Z           09      unknown /CN=mm-osx-107.qa.scl3.mozilla.com
0x000a 2014-04-21T16:32:11GMT 2019-04-21T16:32:11GMT /CN=Puppet CA: puppetmaster1.qa.scl3.mozilla.com

where that last line is definitely wrong.  Among other reasons, that's the puppetmaster *CA* certificate, which was issued by the root CA, not this CA.  It also seems to have been (will be?) revoked in 2019.  And it's in the wrong format (although perhaps OpenSSL will use that format in 5 years..)  So I'm not sure where it came from.

There's no 0A.pem in the cert dir, so I'm just going to remove that line.  That should allow CRL generation to proceed, which is what is causing the puppet runs to fail in the first place.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

And now,

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Syntax error at 'AAAAB3NzaC1yc2EAAAADAQABAAABAQD6WWr8o147HCzCAkU4Iq81V'; expected ']' at /etc/puppet/production/modules/ssh/manifests/keys.pp:18 on node puppetmaster1.qa.scl3.mozilla.com

which I'll leave to you to fix :)

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ouch!! OMG, I missed to put the newly added keys into quotes. I fixed that now:

https://hg.mozilla.org/qa/puppet/rev/3568cf8e4e61
https://hg.mozilla.org/qa/puppet/rev/4709ac79bf99

I hope that will fix the errors I'm also seeing now.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

With the fix the changes have been applied:

Puppet changes applied at puppetmaster1.qa.scl3.mozilla.com:

diff --git a/modules/ssh/manifests/keys.pp b/modules/ssh/manifests/keys.pp
--- a/modules/ssh/manifests/keys.pp
+++ b/modules/ssh/manifests/keys.pp
[..]

Andreea can't still login. So what's left to do here now?

Comment 15

[Andreea Matei [:AndreeaMatei]]

We have solved this, we needed to change the permissions for our keys. Thanks!