1046749 - create signing servers for v2 mac signing

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

We'll need 10.9 signing machines for bug 1046306. Depending on what we do with ESR builds, we may need to keep 2 10.6 signing machines around to support it.

Comment 1

[[e-mail] (:bhearsum)]

I talked to Amy about hardware a bit:
09:41 < bhearsum> arr: so, there's a 99% chance we're going to need 3 or 4 machines to do signing on 10.9. the exact timeline is TBD, but it's very likely going to be 
                  short. do we have machines around that we can use for that, or is there something i should do to start the process of acquiring them? i'm hoping not to 
                  steal from our test pool, since it's already overloaded
09:42 -!- simone is now known as simone|meeting
09:42 < arr> can we reimage the existing signing servers?
09:43 < arr> or do they need to be up in parallel?
09:43 < bhearsum> not sure yet, we may need to keep 2 around still
09:43 < bhearsum> we can definitely re-use 2 of them, if they can run 10.9
09:43 < bhearsum> i think they were rev4 machines....
09:43 < arr> there's no reason they shouldn't be able to afaik
09:43 < bhearsum> ok
09:44 < arr> we have another spare r5 we can also give you
09:44 < bhearsum> awesome
09:44 < bhearsum> so at the worst, we may need one additional machine, if we want 4 10.9 signing machines and need to keep 2 10.6 ones
09:45 < bhearsum> we survived on 2 signing machines during a downtime though, so i'm not sure we need 4 anymore...
09:45 < arr> we should at least have n+1
09:46 < bhearsum> yeah
09:46 < bhearsum> n might be 2 now
09:46 < bhearsum> we had 4 before because we wanted 2 in scl1 and 2 in scl3
09:47 < arr> we could also likely take some r4s form the snow pool if you want all the same hardware
09:47 < arr> since I think that has lots of spare capacity
09:47 < arr> (though I may be mistaken)
09:48 < bhearsum> ahh
09:48 < bhearsum> ok, cool
09:48 < arr> anyway, we should be able to cobble something together, regardless.  the sooner you let us know requirements, the better, of course  :D
09:48 < bhearsum> understood :)
09:48 < bhearsum> thanks amy!
09:49 < arr> bhearsum: sure thing!
09:50 < arr> (we'll need to move machines around in vlans, for example, so that will take us some time since we have to ask dcops to update the switches)

Comment 2

[[e-mail] (:bhearsum)]

I managed to puppetize a 10.9 machine as a signing server with a few caveats:
* I had to generate new RPMs for nrpe and libevent.
* I also had to copy the 10.6 signmar rpm over to the 10.9 DMGs directory. 
* User creation messed up the first time, but I think that's because I set-up Hiera wrong. The second time through went OK, but the signer_pw_pbkdf2* entries in hiera need to be updated with new secrets - they're currently using the same ones as root because I couldn't figure out how to generate new ones.

I'm now at the point where the signing server app is running on a 10.9 machine. It looks like they'll be no code changes to the app needed nor any puppet manifest changes. If that remains true, this bug will just track migrating/creating 10.9 signing machines. bug 1046747 is tracking the necessary changes to how we invoke codesign and other details of the signature.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

mac-signing1 is out of use at the moment for a couple of freeze ups, I'll get moving on asking DCOps to run diagnostics on it.

Comment 4

[[e-mail] (:bhearsum)]

Note: still need to make sure that our self signed cert works.

Comment 5

[[e-mail] (:bhearsum)]

Per https://bugzilla.mozilla.org/show_bug.cgi?id=1046747#c9, we're going to need to run 10.6 and 10.9 signing servers at the same time -- we cannot support old and new style Firefox packages on one signing server.

So, we need to figure out how many we need of each, and get the 10.9 ones up and running. Hopefully this doesn't mean purchasing any new hardware. I'll figure this out ASAP, and start imaging process for the 10.9 ones.

Dustin, I also need help with this:
(In reply to Ben Hearsum [:bhearsum] from comment #2)
> * User creation messed up the first time, but I think that's because I
> set-up Hiera wrong. The second time through went OK, but the
> signer_pw_pbkdf2* entries in hiera need to be updated with new secrets -
> they're currently using the same ones as root because I couldn't figure out
> how to generate new ones.

Any tips?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

You should be able to set ctlsign's password with 'passwd', and then run
  sudo ruby mtnlion-user-info.rb cltsign
to get the (un-encrypted) info you need.

Amy noted today that the 10.9 image we have on DS definitely isn't 10.9.5 -- it's 10.9.0.  Have you upgraded these hosts by hand?  We may need to generate a new 10.9.5 base image if 10.9.0 doesn't work.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

with all of the passwords the same:

[[e-mail] ~]# passwd cltsign
Changing password for cltsign.
New password: <fake password>
Retype new password: <fake password>
[[e-mail] ~]# env -i ssh cltsign@localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 3c:b7:ce:27:25:5b:58:6d:ed:e8:c4:37:8d:82:b6:fe.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
cltsign@localhost's password: <fake password>
Unauthorized access prohibited
This signing server hosts the following instances:
* port 9110 in /builds/signing/dep-key-signing-server
* port 9100 in /builds/signing/nightly-key-signing-server
* port 9120 in /builds/signing/rel-key-signing-server

It's possible that puppet re-set the password between when you used 'passwd' and when you saw the auth failure.  It's also possible that you've been running puppet against a master that hasn't sync'd the updated secrets yet.

I put the secrets generated from running 'passwd' and then the ruby script into the hiera file on releng-puppet2, then ran puppet against that server, changed cltsign's password, ran puppet, and then logged in successfully:

[[e-mail] ~]# passwd cltsign
Changing password for cltsign.
New password: <fake password>
Retype new password: <fake password>
[[e-mail] ~]# puppet agent --test --server=releng-puppet2.srv.releng.scl3.mozilla.com
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_ruby_interpreter.rb
Info: Loading facts in /var/lib/puppet/lib/facter/env.rb
Info: Loading facts in /var/lib/puppet/lib/facter/existing_slave_trustlevel.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_persistent_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/needs_reboot.rb
Info: Loading facts in /var/lib/puppet/lib/facter/num_masters.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/supermicro_ipmi_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/vmwaretools_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/winrootlp.rb
Info: Caching catalog for r5-mini-002.srv.releng.scl3.mozilla.com
Info: Applying configuration version '68f5d255419e'
Notice: /Stage[users]/Users::Signer::Account/User[cltsign]/password: changed password
Notice: /Stage[users]/Users::Signer::Account/User[cltsign]/salt: salt changed '60d547079f32150bec34bee06b2a7d3eba08c4827485262d41ac65f9ee07683f' to 'fafa52c3fb17e032ed5cb8e885fbd91f383977a657157bd392bcda62ec971a0e'
Notice: /Stage[users]/Users::Signer::Account/User[cltsign]/iterations: iterations changed '25188' to '25000'
Info: /User[cltsign]: Scheduling refresh of Exec[kill-signer-keychain]
Info: /User[cltsign]: Scheduling refresh of Exec[kill-signer-keychain]
Info: /User[cltsign]: Scheduling refresh of Exec[kill-signer-keychain]
Notice: /Stage[users]/Users::Signer::Account/Exec[kill-signer-keychain]: Triggered 'refresh' from 3 events
Notice: /Stage[main]/Disableservices::Common/Exec[disable-panic-reporting]/returns: executed successfully
Notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/File[/var/root/.ssh/known_hosts]/content:
--- /var/root/.ssh/known_hosts  2014-08-06 07:57:56.000000000 -0700
+++ /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20140806-80122-mj3jip   2014-08-06 08:00:13.000000000 -0700
@@ -41,4 +41,3 @@
 symbolpush.mozilla.org,63.245.217.193 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA8C9kssmf2rAl2Y6iS6JONcgArpYJBMVzUwLE8Bd4A4qr1TIqLKUTTSkU3T8/+6lBj8UWmzRNwZ/eXCAquvsm0vSa1PX2shBrcuIi8w8JvyYszTMNseiLJmA7ADZ3NpQFr6KKTyH/JsB+vnbU0lO/KNsUcaFkaSelSrwR8rPmhAxrsxUbWKgSLMCtiaw9m7+WBgh+LpzQJPZh6gbmVWWPi7sQx7XgAsSOxkDQAQR3rCucXAVo/snG993d+etqWZqQzIt1gr2tx326ZywV5p+8lv0tHUtD8GR7lEN5uVp6xzvouXfzrhGIuZNc/GoY1MFBCmBdenF0h3Xvrj0JDHKolw==
 symbols1.dmz.phx1.mozilla.com,10.8.74.48 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA8C9kssmf2rAl2Y6iS6JONcgArpYJBMVzUwLE8Bd4A4qr1TIqLKUTTSkU3T8/+6lBj8UWmzRNwZ/eXCAquvsm0vSa1PX2shBrcuIi8w8JvyYszTMNseiLJmA7ADZ3NpQFr6KKTyH/JsB+vnbU0lO/KNsUcaFkaSelSrwR8rPmhAxrsxUbWKgSLMCtiaw9m7+WBgh+LpzQJPZh6gbmVWWPi7sQx7XgAsSOxkDQAQR3rCucXAVo/snG993d+etqWZqQzIt1gr2tx326ZywV5p+8lv0tHUtD8GR7lEN5uVp6xzvouXfzrhGIuZNc/GoY1MFBCmBdenF0h3Xvrj0JDHKolw==
 update.boot2gecko.org,184.73.70.191 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA7bU2ooQ9VoBBWl7hZqI1QxJmoiKcFqKYXl/i8Jt1cxigXSIOSIiPDu4fiuXpkQzZ2x97Da1QcK1E/vc8p5lRMaOdwxKIreHH1Mq6HEuzYX3cqreDfSe9EflHDZIepRi5RW4NpMsqi+OIF2NEsYQ+oEOvb/YCKHCNOMWlAIY5CUMLZIBO/Qq3K4EzUrVWeqYYyPZlfyI67sj/eiu6OI25DnqGl/cQYYrHwETt93E7a2g1Oi3t9ehhkwKheeEYvVFNLPZQA/6sDrVO4oLdRYMIlL/e8swJBiQg0I5GBi6xmwLB+9bI9iwjKqgRa7wm2shsj8G6/7OnoBlFyuqNThO69w==
-localhost ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTTfpJX6kJp/IehPNoJptLnRFVRXCH9hsY0B3kOI8kji01PBreWhL6TNBCNN1+pcGSxygAK/we2eRcEIqaL78QwhHQm7Jaym0buh8Bm0/I/MBgPPpdESiyINzko22It+tT/uk4WedXvuYBk8AQwmWi6jsfG2ey4y34mZ54rkQlMRaSu+JVP/YMq99rM38frCZuAwcWWfQcwU4Bk/k04+ko93g/owfB1mQG2t53bdoLOjbnbi1ViVC/SFJVoFR1MdiOnOxaX/uXSAIU6A8x9619VIK7w1pCy1SlYbWgwNAILkJm14foy6S7JpebpRGy/46/GseVUKnI0taGPskOZDut

Notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/File[/var/root/.ssh/known_hosts]/content: content changed '{md5}9a7ffd2d2513fe8a3370fbaf94b85fca' to '{md5}3f43f7a9409d1ba3ec5fe971e78c758f'
Notice: Finished catalog run in 66.00 seconds
[[e-mail] ~]# env -i ssh cltsign@localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 3c:b7:ce:27:25:5b:58:6d:ed:e8:c4:37:8d:82:b6:fe.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
cltsign@localhost's password: <real password>
Last login: Wed Aug  6 07:58:09 2014 from localhost
Unauthorized access prohibited
This signing server hosts the following instances:
* port 9110 in /builds/signing/dep-key-signing-server
* port 9100 in /builds/signing/nightly-key-signing-server
* port 9120 in /builds/signing/rel-key-signing-server
[[e-mail] ~]$

So, I think we're good  here..

Comment 8

[[e-mail] (:bhearsum)]

Attachment: adjust signing regex to support v2 mac servers

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8468490 [details] [diff] [review]
adjust signing regex to support v2 mac servers

I think (v2-)? is a more common way to write that in regex, but this works.

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 8468490 [details] [diff] [review]
adjust signing regex to support v2 mac servers

Landed on default+production.

Comment 11

[[e-mail] (:bhearsum)]

All of the new signing servers are up. We're enabling them in production today in bug 1049595.