795060 - Create separate ssh keys for AWS

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

Ideally different set of keys per region. Creating the keys is easy. Making sure they're in authorized_keys everywhere is the hard part.

Comment 1

[Rail Aliiev [:rail]]

Anything I missed in the list below?

The keys should be deployed to the following hosts:

* stage.m.o
* aus3-staging.m.o
* relengweb1.dmz.scl3.mozilla.com (SYMBOL_SERVER_HOST, the same ip for build.mozilla.org (tb SYMBOL_SERVER_HOST))
* hg.mozilla.org (only if we need to run release builds in AWS)
* if we want to run fuzzing jobs in AWS  8-)
  ** hg.mozilla.org/private/fuzzing
  ** pvtbuilds2.dmz.scl3.mozilla.com
  
Still need to plan key to host:directory maps.

Comment 2

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] from comment #1)
> * hg.mozilla.org (only if we need to run release builds in AWS)

blocklist update also requires this key/host

Comment 3

[Rail Aliiev [:rail]]

This is the updated list of keys, hosts, usernames:

Production
----------

auspush:
  aus3-staging (ffxbld, tbirdbld)

b2gbld_dsa:
  pvtbuilds2.dmz.scl3.mozilla.com (b2gbld)
  stage.mozilla.org (b2gbld)

ffxbld_dsa
  hg.mozilla.org (ffxbld)
  stage.mozilla.org (ffxbld)
  symbols1.dmz.phx1.mozilla.com (ffxbld)
  aus3-staging.mozilla.org (ffxbld)
  pvtbuilds2.dmz.scl3.mozilla.com (ffxbld)

tbirdbld_dsa
  hg.mozilla.org (tbirdbld)
  stage.mozilla.org (tbirdbld)
  symbols1.dmz.phx1.mozilla.com (tbirdbld)
  aus3-staging.mozilla.org (tbirdbld)

xrbld_dsa
  stage.mozilla.org (xrbld)
  symbols1.dmz.phx1.mozilla.com (xrbld)
  

Try
---

b2gtry_dsa:
  pvtbuilds2.dmz.scl3.mozilla.com (b2gtry)
  stage.mozilla.org (b2gtry)

trybld_dsa:
  stage.mozilla.org (trybld)
  symbols1.dmz.phx1.mozilla.com (trybld)

Comment 4

[Rail Aliiev [:rail]]

Generated keys:

production:
for k in auspush b2gbld_dsa ffxbld_dsa tbirbbld_dsa xrbld_dsa; do echo $k; ssh-keygen -t dsa -C "aws-$k" -f $k ; done


try:
for k in b2gtry_dsa trybld_dsa; do echo $k; ssh-keygen -t dsa -C "aws-$k" -f $k ; done

Comment 5

[Rail Aliiev [:rail]]

Amy, Dustin:

do you know who would be the best person to contact with for deploying the keys on the servers?

I believe that hg.m.o uses LDAP to suck the keys, but I'm not sure if it allows to store  multiple ssh keys per person.

AFAIK, the following ones should be deployed by puppet (or puppet via LDAP):

aus3-staging.mozilla.org
pvtbuilds2.dmz.scl3.mozilla.com
stage.mozilla.org
symbols1.dmz.phx1.mozilla.com

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Multiple keys will work with LDAP.  I think you're talking about user keys, so I'm not sure what to make of the list of four hostnames in comment 5.

Comment 7

[Rail Aliiev [:rail]]

Just to clarify the issue. I generated private keys and need to deploy the corresponding public ones to those machines to users' ~/.ssh/authorized_keys.

Comment 8

[Rail Aliiev [:rail]]

This is the updated list of keys, hosts, usernames. I added b2gbld and b2gtry to symbols1.dmz.phx1.mozilla.com to be ready for it:

Production
----------

auspush:
  aus3-staging (ffxbld, tbirdbld)

b2gbld_dsa:
  pvtbuilds2.dmz.scl3.mozilla.com (b2gbld)
  stage.mozilla.org (b2gbld)
  symbols1.dmz.phx1.mozilla.com (b2gbld)

ffxbld_dsa
  hg.mozilla.org (ffxbld)
  stage.mozilla.org (ffxbld)
  symbols1.dmz.phx1.mozilla.com (ffxbld)
  aus3-staging.mozilla.org (ffxbld)
  pvtbuilds2.dmz.scl3.mozilla.com (ffxbld)

tbirdbld_dsa
  hg.mozilla.org (tbirdbld)
  stage.mozilla.org (tbirdbld)
  symbols1.dmz.phx1.mozilla.com (tbirdbld)
  aus3-staging.mozilla.org (tbirdbld)

xrbld_dsa
  stage.mozilla.org (xrbld)
  symbols1.dmz.phx1.mozilla.com (xrbld)
  

Try
---

b2gtry_dsa:
  pvtbuilds2.dmz.scl3.mozilla.com (b2gtry)
  stage.mozilla.org (b2gtry)
  symbols1.dmz.phx1.mozilla.com (b2gtry)

trybld_dsa:
  stage.mozilla.org (trybld)
  symbols1.dmz.phx1.mozilla.com (trybld)

Comment 9

[Rail Aliiev [:rail]]

us-east-1 is using the new keys.

Comment 10

[Rail Aliiev [:rail]]

It turns out that Thunderbird uses a different user for try uploads:


Command ['ssh', '-o', 'IdentityFile=~/.ssh/trybld_dsa', '[e-mail]', 'mktemp -d'] returned non-zero exit code: 255

Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Comment 11

[Michael Henry [:tinfoil]]

I've reached out to the opsec and nobody specifically remembers making this recommendation.  Perhaps this was something that came out of a conversation?  Either way....

We don't have a strong stance on this at this time.  Currently our perspective is that AWS is an extension of the current build environment.  While we would certainly like more keys, if it's seriously causing delays, then lets work around it until we can implement something better.