1186302 - Add host keys for new upload hosts in GPO

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Similar to bug 1186300 but for GPO. I'll attach the new known_hosts we'd like to use.

This would be for the windows build slaves, although bug 1185873 implies that we use the same GPO for testers too.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: known_hosts

This is the same file we have on linux, after bug 1186300 modifies it.

FTR, this removes aus3-staging.m.o, bm-remote-talos-webhost-0{1,2,3}.build.mozilla.org, several old buildbot masters, cvs.m.o, preproduction-stage, a scl1 puppet host, servo-puppet1, and update.boot2gecko.org. It adds 6 new hosts
 upload.{trybld,ffxbld,tbirdbld}.productdelivery.{stage,prod}.mozaws.net

Q, would it be possible to roll this out in the next few days ?

Comment 2

[Q]

Test GPOs look good should we go for a Monday rollout?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Sounds good, I'll be around from about 2pm Pacific, or go earlier if buildduty is game. Partial rollout first, or go the whole hog ?

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

From Q in bug 1186300: updating ix-0060 - ix-0089 updating confirmed that 082 got the update.

Unfortunately we're going to need to redo this, the server side got changed on us.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: known_hosts v2

With apologies for needing to repackage this.

Comment 6

[Q]

On it now.

Comment 7

[Q]

Emergency pushing to replace this file should be out in the next 15 minutes to any host who got the v1 file.

Comment 8

[Amy Rich [:arr] [:arich]]

Nick: just to double check, have you also added this to puppet for the puppetized windows machines?

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Yes, that's being done in bug 1186300.

Comment 10

[Q]

Nick, I think we are good for site wide rollout. What do you think ?

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Yes, lets go ahead. I can't see any problems on the slaves that already got the updated file.

Comment 12

[Q]

Last update didn't take. This is out in the wild and looking good so far.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Everything is updated except for a windows puppet hosts, which I'll take to bug 1186300.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: known_hosts v3

We needed to fix a couple of issues in puppet, described over at https://bugzilla.mozilla.org/show_bug.cgi?id=1186300#c18, so I'm reopening this bug to fix them in GPO too.

The true deadline is a few days before Oct 13, but there's no reason to wait that long as the diff (with truncated lines) looks like this:

45,48c45,48
< upload.tbirdbld.productdelivery.stage.mozaws.net ssh-rsa AAAAB3NzaC1yc2EAAAADA
< upload.trybld.productdelivery.prod.mozaws.net ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB
< upload.ffxbld.productdelivery.prod.mozaws.net ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB
< upload.tbirdbld.productdelivery.prod.mozaws.net ssh-rsa AAAAB3NzaC1yc2EAAAADAQ
\ No newline at end of file
---
> upload.tbirdbld.productdelivery.stage.mozaws.net,52.3.217.123 ssh-rsa AAAAB3Nz
> upload.trybld.productdelivery.prod.mozaws.net,52.27.28.242 ssh-rsa AAAAB3NzaC1
> upload.ffxbld.productdelivery.prod.mozaws.net,52.11.216.122 ssh-rsa AAAAB3NzaC
> upload.tbirdbld.productdelivery.prod.mozaws.net,52.88.134.149 ssh-rsa AAAAB3Nz

ie only changes to not hosts not in use yet, and adding a newline at the end of the file.

Comment 15

[Amy Rich [:arr] [:arich]]

I thought we verified GPO independently? Is it actually broken?

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

When we first verified only two of the staging hosts had been created, so we checked just those (we could add the others because they have the same host sig). It turns out that if you do something like 
  ssh -i ~/.ssh/ffxbld_rsa upload.ffxbld.productdelivery.prod.mozaws.net
then ssh will add a line to known_hosts with the actual IP address and host sig (harmless). The lack of newline (at the end of the currently deployed file) means it appends to the line for upload.tbirdbld.productdelivery.prod.mozaws.net and busts it. So I'm adding the newline, and while I'm here adding the IPs now we have them.

Comment 17

[Q]

Added the new line but my updated was eaten. I think we should be good here.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

I checked b-2008-ix-0067, which had rebooted about 5 hours ago, and it didn't have a trailing newline or attachment 8665635 [details]. Could we deploy that attachment, with CRLF line endings added if you prefer ?

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Hi, what's the the state here ? We'll need to use this from Oct 13. If I can help by modifying the file on the share directly then just let me know.

Comment 20

[Q]

Let me jump on the share and take a look. I also need to check that the replace is taking

Comment 21

[Q]

Apologies, it looks like there was a file mismatch. That has been corrected I spotted checked five systems I forced pushed to and the lines looked correct and the new line is at that bottom of the file. I will keep an eye on it.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Spot checks look good to me too.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

All done here ? The migration is tomorrow, 1pm Pacific.

Comment 24

[Q]

Yes Sir, everything should be set.

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Awesome, thanks for all your patience with the multiple deploys.