1389542 - configure new tc-host-secrets servers in releng puppet

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[:dhouse]

Need to add nodes in releng puppet for the new tc-host-secrets servers in scl3 and mdc1 and then configure how the tc-host-secrets module+package is installed and set up.

Comment 1

[:dhouse]

This will be similar to how it was set up on the puppet-masters in bug 1341654

Comment 2

[:dhouse]

Attachment: Add taskcluster_host_secrets server class and node def

Comment 3

[:dhouse]

Attachment: add fw role for tc_host_secrets

Comment 4

[Jake Watkins [:dividehex]]

Comment on attachment 8898868 [details] [diff] [review]
add fw role for tc_host_secrets

Review of attachment 8898868 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/fw/manifests/profiles/taskcluster_host_secrets.pp
@@ +6,5 @@
> +
> +    case $::fqdn {
> +        /.*\.(scl3|mdc1)\.mozilla\.com/: {
> +            include ::fw::roles::taskcluster_host_secrets
> +            include ::fw::roles::ssh_from_rejh

Missing nrpe_from_nagios role; we'll definitely need nagios checks.  Also, we're not entirely ready to limit this to rejh only since no one in releng is even enrolled in duo yet.  Please change this to ssh_from_anywhere in the meantime.  When we are ready and announce the activation of the jumphost restrictions, I'll make sure to swap it back out for ssh_from_rejh.

::: modules/fw/manifests/roles/taskcluster_host_secrets.pp
@@ +8,5 @@
> +    fw::rules { 'allow_taskcluster_host_secrets_http':
> +        sources =>  $::fw::networks::dc_test,
> +        app     => 'http'
> +    }
> +}
\ No newline at end of file

For this role, the file (and class) should be named 'taskcluster_host_secrets_from_dc_test.pp' to stay in accordance with the naming standard of <type_of_service>_from_<its_sources>.  Also, title the fw:rules with 'allow_taskcluster_host_secrets_from_dc_test'. Sticking with this standard gives much better readability. The See other roles for examples.  (I should also update the fw wiki docs since it isn't obvious)

Comment 5

[Jake Watkins [:dividehex]]

Comment on attachment 8898855 [details] [diff] [review]
Add taskcluster_host_secrets server class and node def

Review of attachment 8898855 [details] [diff] [review]:
-----------------------------------------------------------------

::: manifests/moco-nodes.pp
@@ +20,5 @@
>      include generic_worker::disabled
>  }
>  
> +# taskcluster-host-secrets hosts
> +node /tc-host-secrets\d+\.srv\.releng\.(mdc1|scl3)\.mozilla\.com/ {

Please put a ^ in the front just to be more explicit.  I think that should be a standard going forward for node def regex.

/^tc-host-secrets\d+\.srv\.releng\.(mdc1|scl3)\.mozilla\.com/

@@ +21,5 @@
>  }
>  
> +# taskcluster-host-secrets hosts
> +node /tc-host-secrets\d+\.srv\.releng\.(mdc1|scl3)\.mozilla\.com/ {
> +    $aspects          = [ 'low-security' ]

I can only assume that since this service is handling giving out security tokens, it should probably be treated with a much higher security aspect.  Possibly 'high' or 'maximum'?  Maybe :dustin would know.

Comment 6

[Jake Watkins [:dividehex]]

Dustin, which sec aspect should be used here?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I don't remember what the details of each level are, but yes -- higher than "low"!

Also, last I knew there wasn't much difference above "high", so let's just say "high".

Comment 8

[:dhouse]

Attachment: add tc-host-secrets node def

Comment 9

[:dhouse]

Attachment: add fw role for tc_host_secrets

Comment 10

[:dhouse]

Attachment: add fw role for tc_host_secrets

Comment 11

[Jake Watkins [:dividehex]]

Comment on attachment 8899600 [details] [diff] [review]
add fw role for tc_host_secrets

Review of attachment 8899600 [details] [diff] [review]:
-----------------------------------------------------------------

Perfect! thanks

Comment 12

[:dhouse]

Applied node def creation:
remote:   https://hg.mozilla.org/build/puppet/rev/cbdcc2d7ea62f71a8f10cb30bbfd4cdcb954f5be
Travis passed. Pushed to prod:
remote:   https://hg.mozilla.org/build/puppet/rev/a6e93e6ffd423eb28bf2c77926b193e6702bc6eb

Comment 13

[:dhouse]

Comment on attachment 8899600 [details] [diff] [review]
add fw role for tc_host_secrets

Pushed the fw role for tc-host-secrets to prod.
remote:   https://hg.mozilla.org/build/puppet/rev/43a08ca645e1557720fe94caaa178e404a23fd51

Comment 14

[Dragos Crisan [:dragrom]]

Comment on attachment 8899566 [details] [diff] [review]
add tc-host-secrets node def

LGTM

Comment 15

[Dragos Crisan [:dragrom]]

Comment on attachment 8899600 [details] [diff] [review]
add fw role for tc_host_secrets

LGTM

Comment 16

[:dhouse]

Attachment: allow changing host-secrets port. set tc-host-secretsN to port 80

Need to be able to configure which port tc-host-secrets listens on:

[[e-mail] ~]$ cat /var/log/taskcluster-host-secrets.stderr.log 
Tue, 29 Aug 2017 15:21:20 GMT typed-env-config Config file missing: user-config.yml
Tue, 29 Aug 2017 15:21:20 GMT taskcluster-lib-validate Attempting to set constants by file: /opt/taskcluster-host-secrets/code/schemas/constants.yml
Tue, 29 Aug 2017 15:21:20 GMT taskcluster-lib-validate finished walking tree of schemas
Tue, 29 Aug 2017 15:21:21 GMT taskcluster-client Calling: awsS3Credentials, retry: 0
Tue, 29 Aug 2017 15:21:21 GMT taskcluster-client Success calling: awsS3Credentials, (0 retries)
Tue, 29 Aug 2017 15:21:22 GMT taskcluster-lib-docs { ETag: '"9c02caf90740342c701ae4a093b85330"',
  PartNumber: 1,
  receivedSize: 3183,
  uploadedSize: 3183 }
Tue, 29 Aug 2017 15:21:22 GMT taskcluster-lib-docs { Location: 'https://taskcluster-raw-docs.s3.amazonaws.com/taskcluster-host-secrets%2Flatest.tar.gz',
  Bucket: 'taskcluster-raw-docs',
  Key: 'taskcluster-host-secrets/latest.tar.gz',
  ETag: '"08443b35e8ec6564027e2bb9a072945c-1"' }
Tue, 29 Aug 2017 15:21:22 GMT base:app Server listening on port 80

Comment 17

[Jake Watkins [:dividehex]]

Comment on attachment 8902324 [details] [diff] [review]
allow changing host-secrets port. set tc-host-secretsN to port 80

Review of attachment 8902324 [details] [diff] [review]:
-----------------------------------------------------------------

I think this is a fine solution but one thing I'd like to steer clear of in the future is the excessive use of node scope variables.  I know I've defaulted to them in the past but I think we should make a point to putting things in moco-config.pp and using $::fqdn selector switch instead.  In the meantime, this works and we can go back and remove it once the new hosts are running and the old service is removed from the puppetmasters.

So... Ship it! :-)

Comment 18

[Dragos Crisan [:dragrom]]

Comment on attachment 8902324 [details] [diff] [review]
allow changing host-secrets port. set tc-host-secretsN to port 80

Review of attachment 8902324 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

::: manifests/moco-nodes.pp
@@ +31,3 @@
>  # taskcluster-host-secrets hosts
>  node /^tc-host-secrets\d+\.srv\.releng\.(mdc1|scl3)\.mozilla\.com/ {
>      $aspects          = [ 'high-security' ]

arrange the = sign

Comment 19

[:dhouse]

Comment on attachment 8902324 [details] [diff] [review]
allow changing host-secrets port. set tc-host-secretsN to port 80

remote:   https://hg.mozilla.org/build/puppet/rev/8522549e51d108355b022b3061a36e72790965e4
travis passed. pushed to prod:
remote:   https://hg.mozilla.org/build/puppet/rev/6c4a88d95a2a8552718dd4688550bd378d9dc062

Comment 20

[:dhouse]

(In reply to Dragos Crisan [:dragrom] from comment #18)
> Comment on attachment 8902324 [details] [diff] [review]
> allow changing host-secrets port. set tc-host-secretsN to port 80
> 
> Review of attachment 8902324 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lgtm
> 
> ::: manifests/moco-nodes.pp
> @@ +31,3 @@
> >  # taskcluster-host-secrets hosts
> >  node /^tc-host-secrets\d+\.srv\.releng\.(mdc1|scl3)\.mozilla\.com/ {
> >      $aspects          = [ 'high-security' ]
> 
> arrange the = sign

Amended for aligning the equals:
remote:   https://hg.mozilla.org/build/puppet/rev/6aa02e22a075b0491cb465bab05d684f36763f86
travis passed. pushed to prod
remote:   https://hg.mozilla.org/build/puppet/rev/89c50bdb3259f38d2bed0ea55652ce5ec023590b

Comment 21

[:dhouse]

Attachment: apply host-based firewall to tc-host-secrets hosts

This allows only port 80(from test/wintest in scl3/mdc1), nagios, and ssh (from anywhere with logging).

Comment 22

[:dhouse]

Comment on attachment 8905206 [details] [diff] [review]
apply host-based firewall to tc-host-secrets hosts

Applied host-based fw patch
remote:   https://hg.mozilla.org/build/puppet/rev/5f6f6561cd8c5f24284a13d2d7095f781e7f4bcf
Travis passed. Pushed to production:
remote:   https://hg.mozilla.org/build/puppet/rev/59e0531c26f446c5298f6260c7f6377d1ee17d70