869498 - Create puppetagain manifests for signing servers

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Amy Rich [:arr] [:arich]]

We've created the following signing servers on vmware in SCL3 to replace those on kvm in scl1 and scl3:

signing4.srv.releng.scl3.mozilla.com
signing5.srv.releng.scl3.mozilla.com
signing6.srv.releng.scl3.mozilla.com

Dustin is working on manifests to set them up under puppetagain.  When that's complete, bhearsum will test.  Once they're signed off on by bhearsum, releng will cut over from using signing1, signing2, and signing3 to these three new servers.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I made good progress on this today.  The plan is to get this working for Linux, and add Darwin in a second patch.

My remaining todo's:

 - signmar RPM -- looks like the binary is at 19.0a1 (in prog)
 - jdk1.6, android_sdk packages
 - new secrets for ip ranges
 - iptables
 - gcc link on darwin
 - handle secrets internally
 - treat signing server as a service, rather than exec
 - use mercurial::repo instead of manually cloning
 - pull more moco-specific config out of templates
 - pub/pvt certs - config/secrets
 - signmar DMG, with all of the .dylib's in the DMG version
 - test-files DMG

I'm working on building signmar from source but .. m-c is hard to build!

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

OK, I'm giving up on building Mozilla-Central from source.  I'll build a .spec that expects to start with a built nightly.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

From a chat with aki, per bug 706243, the jdk and android SDK packages probably aren't very version-sensitive.  We have 1.6 JDK's in the CentOS repos, and Android SDK's in the releng repos, so those should be fine.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

signmar RPM is built:

[[e-mail] firefox]$ rpm -qlp signmar-19.0-1.el6.x86_64.rpm 
/usr/bin/signmar
[[e-mail] firefox]$ rpm -qRp signmar-19.0-1.el6.x86_64.rpm 
libc.so.6()(64bit)  
libc.so.6(GLIBC_2.2.5)(64bit)  
libdl.so.2()(64bit)  
libgcc_s.so.1()(64bit)  
libm.so.6()(64bit)  
libnspr4.so()(64bit)  
libnss3.so()(64bit)  
libnss3.so(NSS_3.2)(64bit)  
libnss3.so(NSS_3.3)(64bit)  
libnss3.so(NSS_3.4)(64bit)  
libnss3.so(NSS_3.5)(64bit)  
libnssutil3.so()(64bit)  
libplc4.so()(64bit)  
libplds4.so()(64bit)  
libpthread.so.0()(64bit)  
libstdc++.so.6()(64bit)  
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rtld(GNU_HASH)  
rpmlib(PayloadIsXz) <= 5.2-1

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug869498.patch

Please pay particular attention to:

- management of private keys: since these are multiline and thus do not fit into a CSV file, I'd like to wait until we use Hiera, even though the old-puppet version installs the keys directly

- per-org config: the idea here is that moco can change the arrangement of and passwords for instances without affecting other orgs.

- password handling: did I get the corresponding passwords correct?

- process handling: am I correct that the system doesn't automatically start the signing server?

Comment 7

[Amy Rich [:arr] [:arich]]

Added in a dependency for 815281 so we can try to fix that in puppetagain.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 770394 [details] [diff] [review]
bug869498.patch

Review of attachment 770394 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> Created attachment 770394 [details] [diff] [review]
> bug869498.patch
> 
> Please pay particular attention to:
> 
> - management of private keys: since these are multiline and thus do not fit
> into a CSV file, I'd like to wait until we use Hiera, even though the
> old-puppet version installs the keys directly

Seems OK to me. The current manifests don't install secret files, either.

> - password handling: did I get the corresponding passwords correct?

I think so...I'd really like to get a machine puppetized against the MoCo org config to verify, though. I can do some inspection + verification with a dev master with that.

> - process handling: am I correct that the system doesn't automatically start
> the signing server?

Yup, that's right. The signing servers require passphrase entry at start-up, so we have to do it by hand.

r- because of the typo + IP range changes. I want to test with a dev master before r+'ing, too.

::: manifests/moco-config.pp
@@ +43,4 @@
>      $nrpe_allowed_hosts = "10.2.71.20,10.12.75.9,127.0.0.1,10.26.75.30"
>      $ntp_server = "ntp.build.mozilla.org"
>  
> +    $signing_tools_repo = 'http://hg.mozilla.org/build/tools'

Does signer_username need to be set in here? I see it in relabs-config.pp

@@ +45,5 @@
>  
> +    $signing_tools_repo = 'http://hg.mozilla.org/build/tools'
> +    $signing_redis_host = 'redis01.build.scl1.mozilla.com'
> +    $signing_mac_id = 'Mozilla'
> +    $signing_allowed_ips = [

Looks like a good time to purge these lists. This should be all of the networks with build or try machines in them. AFAICT, that's:
10.12.40.0/22
10.12.48.0/21
10.250.48.0/22 (mostly tegra stuff, but still contains a few very old build machines sadly)
10.26.40.0/22
10.26.48.0/24 (technically a srv network, but partner-repack1 is in it, and needs to be able to sign)
10.26.52.0/22
10.134.52.0/23
10.134.64.0/23
10.132.52.0/24
10.132.53.0/24
10.132.54.0/24
10.132.64.0/24
10.132.65.0/24
10.132.66.0/24

Not sure about these, I couldn't find any machines in them in inventory, nor any info on any wiki pages
10.26.36.0/22
10.26.44.0/22
10.26.64.0/22

All of this info sourced from inventory, https://mana.mozilla.org/wiki/display/NOC/VLAN+assignments#VLANassignments-Releng, and https://intranet.mozilla.org/Build/Amazon_Networks.

@@ +61,5 @@
> +        '10.130.0.0/16',
> +        '10.132.0.0/16',
> +        '10.134.0.0/16',
> +    ]
> +    $signing_new_token_allowed_ips = [

And this list should be:
10.132.49.112
10.132.49.125
10.132.50.44
10.132.50.56
10.132.50.142
10.132.50.247
10.134.48.196
10.134.48.228
10.134.48.236
10.134.49.62
10.134.49.93
10.134.49.223

::: modules/packages/manifests/gnupg.pp
@@ +5,5 @@
> +class packages::gnupg {
> +    case $::operatingsystem {
> +        CentOS: {
> +            package {
> +                "gnupg2":

We're using gnupg 1.4.5 on the existing signing machines. I assume that this will work fine...but this reminds me that we need to do some validation of a machine set-up with these manifests.

::: modules/packages/manifests/java.pp
@@ +12,5 @@
>          CentOS, Ubuntu: {
> +            package {
> +                # the precise version here isn't terribly important, at least for signing servers,
> +                # as long as it's 1.6.
> +                'java-1.6.0-openjdk-devel':

Is this going to get us installing Java on a bunch of machines other than signing servers, too?

::: modules/signingserver/templates/signing.ini.erb
@@ +22,5 @@
> +# ips that can connect at all
> +allowed_ips = <%= @allowed_ips.join(', ') %>
> +allowed_filenames = .*\.exe,.*\.mar,.*\.dll,.*\.bz2,.*\.zip,.*\.dmg,.*\.tar,.*\.checksums,.*\.bundle,.*SUMS,.*\.apk
> +min_filesize = 10
> +<%# if these change frequently, consider making them puppetagain config options -%>

These don't change all too often. Maybe once or twice a year.

::: modules/toplevel/manifests/server/signing.pp
@@ +68,5 @@
> +                    mac_cert_subject_ou => "43AQ936H96",
> +                    token_secret   => secret('moco_signing_server_release_token_secret'),
> +                    token_secret0  => secret('moco_signing_server_old_token_secret'),
> +                    new_token_auth => "${signing_server_username}:${signing_server_release_password}",
> +                    new_token_auth0=> "${signing_server_username}:${mock_signing_server_repack_password}",

Typo here: mock vs. moco

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Ben Hearsum [:bhearsum] from comment #8)
> Seems OK to me. The current manifests don't install secret files, either.

They do, actually, but OK.

http://hg.mozilla.org/build/puppet-manifests/file/ca7fc686a3d3/modules/signingserver/manifests/instance.pp#l210
   210         "$full_private_ssl_cert":
   211             content => file("/etc/puppet/secrets/signing.server.key"),
   212             mode => 600;

> > - password handling: did I get the corresponding passwords correct?
> 
> I think so...I'd really like to get a machine puppetized against the MoCo
> org config to verify, though. I can do some inspection + verification with a
> dev master with that.

OK - we can either land this as-is and set up a host, or pin a host to a test environment somewhere.  From your r- it sounds like you'd prefer the latter?

> Does signer_username need to be set in here? I see it in relabs-config.pp

It uses the default, cltsign, but it can't hurt to add it.

IP changes accounted for.  10.26.36 is winbuild, 10.26.44 is wintry, and 10.26.64 is try.

> We're using gnupg 1.4.5 on the existing signing machines. I assume that this
> will work fine...but this reminds me that we need to do some validation of a
> machine set-up with these manifests.

I know there were some major API changes to gnupg at one point, but I don't recall if the "2" in "gnupg2" is reflected in the version number - is 1.4.5 a version of gnupg2?  Time will tell..

> Is this going to get us installing Java on a bunch of machines other than
> signing servers, too?

Huh, yes.  I think "packages::java" is too vague.  And installing it on all talos systems, but having it be a no-op on Ubuntu, is yucky.  I'll refactor.

> These don't change all too often. Maybe once or twice a year.

OK, I'll leave the comment in there for next time it gets changed.

> Typo here: mock vs. moco

Fixed.

Now, do you want a new review, or should I set this up on a host somewhere for you?

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Dustin J. Mitchell [:dustin] from comment #9)
> (In reply to Ben Hearsum [:bhearsum] from comment #8)
> > Seems OK to me. The current manifests don't install secret files, either.
> 
> They do, actually, but OK.
> 
> http://hg.mozilla.org/build/puppet-manifests/file/ca7fc686a3d3/modules/
> signingserver/manifests/instance.pp#l210
>    210         "$full_private_ssl_cert":
>    211             content => file("/etc/puppet/secrets/signing.server.key"),
>    212             mode => 600;

Whoops, forgot about that one. I was thinking purely of the actual signing keys.

> > > - password handling: did I get the corresponding passwords correct?
> > 
> > I think so...I'd really like to get a machine puppetized against the MoCo
> > org config to verify, though. I can do some inspection + verification with a
> > dev master with that.
> 
> OK - we can either land this as-is and set up a host, or pin a host to a
> test environment somewhere.  From your r- it sounds like you'd prefer the
> latter?

I don't have a strong preference, but we need to use a new host (or change the IP on an existing one). Otherwise production machines will try to sign with it.

> IP changes accounted for.  10.26.36 is winbuild, 10.26.44 is wintry, and
> 10.26.64 is try.

OK, cool. We need those too, obviously.

> > We're using gnupg 1.4.5 on the existing signing machines. I assume that this
> > will work fine...but this reminds me that we need to do some validation of a
> > machine set-up with these manifests.
> 
> I know there were some major API changes to gnupg at one point, but I don't
> recall if the "2" in "gnupg2" is reflected in the version number - is 1.4.5
> a version of gnupg2?  Time will tell..

I just checked, we use:
gnupg-1.4.5-14.el5_5.1

It looks like "gnupg2" has different binary names, so we'll want plain "gnupg" here.

> Now, do you want a new review, or should I set this up on a host somewhere
> for you?

I'd like to review before landing. I'm happy to do that in parallel with or after testing out a new host.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

gnupg is available in the Fedora repos, but it conflicts with gnupg2.  We can't remove gnupg2:

[[e-mail] ~]# yum remove gnupg2
Loaded plugins: security
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package gnupg2.x86_64 0:2.0.14-4.el6 will be erased
--> Processing Dependency: gnupg for package: debmirror-2.14-2.el6.noarch
--> Processing Dependency: gnupg2 for package: gpgme-1.1.8-3.el6.x86_64
--> Running transaction check
---> Package debmirror.noarch 0:2.14-2.el6 will be erased
---> Package gpgme.x86_64 0:1.1.8-3.el6 will be erased
--> Processing Dependency: libgpgme.so.11()(64bit) for package: pygpgme-0.1-18.20090824bzr68.el6.x86_64
--> Processing Dependency: libgpgme.so.11(GPGME_1.0)(64bit) for package: pygpgme-0.1-18.20090824bzr68.el6.x86_64
--> Processing Dependency: libgpgme.so.11(GPGME_1.1)(64bit) for package: pygpgme-0.1-18.20090824bzr68.el6.x86_64
--> Running transaction check
---> Package pygpgme.x86_64 0:0.1-18.20090824bzr68.el6 will be erased
--> Processing Dependency: pygpgme for package: yum-3.2.29-22.el6.centos.noarch
--> Running transaction check
---> Package yum.noarch 0:3.2.29-22.el6.centos will be erased
--> Processing Dependency: yum >= 3.2.23-10 for package: createrepo-0.9.8-4.el6.noarch
--> Processing Dependency: yum >= 3.0 for package: yum-plugin-fastestmirror-1.1.30-10.el6.noarch
--> Processing Dependency: yum >= 2.4 for package: mock-1.1.21-1.el6.noarch
--> Processing Dependency: yum >= 3.2.29 for package: yum-utils-1.1.30-10.el6.noarch
--> Processing Dependency: yum >= 3.2.18 for package: yum-plugin-security-1.1.30-10.el6.noarch
--> Processing Dependency: yum >= 2.4 for package: mock_mozilla-1.0.3-1.el6.noarch
--> Running transaction check
---> Package createrepo.noarch 0:0.9.8-4.el6 will be erased
---> Package mock.noarch 0:1.1.21-1.el6 will be erased
---> Package mock_mozilla.noarch 0:1.0.3-1.el6 will be erased
---> Package yum-plugin-fastestmirror.noarch 0:1.1.30-10.el6 will be erased
---> Package yum-plugin-security.noarch 0:1.1.30-10.el6 will be erased
---> Package yum-utils.noarch 0:1.1.30-10.el6 will be erased
--> Finished Dependency Resolution
Error: Trying to remove "yum", which is protected
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

They have overlapping binary names:

[[e-mail] ~]# rpm -qlp /data/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64/gnupg2-2.0.14-4.el6.x86_64.rpm | grep bin
/usr/bin/gpg
/usr/bin/gpg-agent
/usr/bin/gpg-connect-agent
/usr/bin/gpg-zip
/usr/bin/gpg2
/usr/bin/gpgconf
/usr/bin/gpgkey2ssh
/usr/bin/gpgparsemail
/usr/bin/gpgsplit
/usr/bin/gpgv
/usr/bin/gpgv2
/usr/bin/watchgnupg
/usr/sbin/addgnupghome
/usr/sbin/applygnupgdefaults
[[e-mail] ~]# rpm -qlp /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/x86_64/os/Packages/gnupg-1.4.11-3.fc15.x86_64.rpm  | grep bin
/usr/bin/gpg
/usr/bin/gpg-zip
/usr/bin/gpgsplit
/usr/bin/gpgv
/usr/bin/lspgpot

And everything I can find indicates that the two are interchangeable - the internal architecture was different, but the interfaces are compatible.  Shall we try it with gnupg2 and see if the signing works?

Comment 12

[[e-mail] (:bhearsum)]

Oh, huh. I guess I was looking at CentOS 5 packages. Let's just try gnupg2 then. I'd be surprised if we're doing anything complicated enough that the difference matters.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug869492-r2.patch

I'll set this up on a demo host in a few minutes.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

OK, I set this up on signing4.srv.releng.scl3, pinned to the test environment on releng-puppet2.build.scl1.mozilla.com.  It's using real secrets.  Ben, have at it!

Note that there were two trivial tweaks to the latest patch: the secret(..) for moco_signing_server_repack_password had a trailing space, and I removed the creation of $testfile_dir, which is anyway created by the test file package.

As a note to self, the signing hosts in comment 0 weren't kickstarted, so they'll need that done before we put them into production.

Comment 16

[[e-mail] (:bhearsum)]

I'll try to get some tests running today. If I can't get to it today, tomorrow for sure.

Comment 17

[[e-mail] (:bhearsum)]

(In reply to Dustin J. Mitchell [:dustin] from comment #15)
> Note that there were two trivial tweaks to the latest patch: the secret(..)
> for moco_signing_server_repack_password had a trailing space, and I removed
> the creation of $testfile_dir, which is anyway created by the test file
> package.

I think these may not have made it into the patch...I still see both of these present in the latest patch. You can fix those on landing though.

I've got some tests running in staging now. I'll r+ the patch if/when they pass.

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Yes, they were tweaks *to*, not *in* the patch ;)

Comment 19

[[e-mail] (:bhearsum)]

Whoops, I forgot that dev-master01 needs to be in new_token_allowed_ips, too. That's 10.12.48.14.

Comment 20

[[e-mail] (:bhearsum)]

And a note for me: make sure to copy ALL secrets (including server/signing.server.key) when installing them onto the new servers.

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Ben Hearsum [:bhearsum] from comment #19)
> Whoops, I forgot that dev-master01 needs to be in new_token_allowed_ips,
> too. That's 10.12.48.14.

Got it.

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 772873 [details] [diff] [review]
bug869492-r2.patch

Review of attachment 772873 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good in my tests, ship it!

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

Backed out - it was causing

Thu Jul 11 09:03:09 -0700 2013 /Stage[main]//Resources[firewall] (err): Failed to generate additional resources using 'generate': Command iptables_save is missing

on macs.

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

Actually, I only backed out the site.pp modification which caused the problem.

Comment 25

[Dustin J. Mitchell [:dustin] (he/him)]

The corrected site.pp got checked in as part of bug 889058.  I'm going to re-kickstart all three hosts now (which will, incidentally, get them the clocksource=pit fix too).

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

OK, all four hosts should be up and ready to go.  Over to ben to verify and close this bug if that's good enough.

Comment 27

[[e-mail] (:bhearsum)]

(In reply to Dustin J. Mitchell [:dustin] from comment #26)
> OK, all four hosts should be up and ready to go.  Over to ben to verify and
> close this bug if that's good enough.

Just checking: you meant three, right? I see a signing4 through 6, but no 7.

Comment 28

[[e-mail] (:bhearsum)]

Had to land a bustage fix for allowed_ips: https://hg.mozilla.org/build/puppet/rev/fd105ae4e5f7

Turns out that new_token_allowed_ips also need to be in allowed_ips.

Things seem to be looking fine now. I'll take care of completing the swap in bug 893867.

Comment 30

[Dustin J. Mitchell [:dustin] (he/him)]

We want to limit $admin_users on these hosts to just a few folks.

Comment 31

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug869498.patch

Comment 32

[Dustin J. Mitchell [:dustin] (he/him)]

[[e-mail] ~]# ls -al /home
total 52
drwxr-xr-x. 13 root      root      4096 Jul 23 13:54 .
dr-xr-xr-x. 24 root      root      4096 Jul 17 13:57 ..
drwx------   2 root      root      4096 Jul 23 13:54 archive
drwxr-xr-x   3 arr       arr       4096 Jul 17 11:52 arr
drwxr-xr-x   3 bhearsum  bhearsum  4096 Jul 18 06:21 bhearsum
drwxr-xr-x   3 catlee    catlee    4096 Jul 17 11:52 catlee
drwx------   4 cltsign   cltsign   4096 Jul 18 13:49 cltsign
drwxr-xr-x   3 coop      coop      4096 Jul 17 11:54 coop
drwxr-xr-x   5 dmitchell dmitchell 4096 Jul 17 11:53 dmitchell
drwxr-xr-x   3 joduinn   joduinn   4096 Jul 17 11:52 joduinn
drwxr-xr-x   3 jwatkins  jwatkins  4096 Jul 17 11:52 jwatkins
drwxr-xr-x   3 nthomas   nthomas   4096 Jul 17 11:52 nthomas
drwxr-xr-x   3 raliiev   raliiev   4096 Jul 17 11:52 raliiev

Comment 33

[[e-mail] (:bhearsum)]

Thanks for the quick turnaround on this Dustin.