852970 - Deploy current xrbld keys to all build slaves

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Rail Aliiev [:rail]]

It happened for 20.0b6 build while trying to upload symbols for xulrunner. It sounds like some of the servers are missing the xrbld key. I retriggered 2 builds for now.

See also bug 804995


debug1: Reading configuration data /c/Users/cltbld/.ssh/config
debug1: Applying options for *mozilla.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to symbolpush.mozilla.org [63.245.217.193] port 22.
debug1: Connection established.
debug1: identity file /c/Users/cltbld/.ssh/xrbld_dsa type 2
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.6
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'symbolpush.mozilla.org' is known and matches the RSA host key.
debug1: Found key in /c/Users/cltbld/.ssh/known_hosts:19
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering public key: /c/Users/cltbld/.ssh/id_dsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: /c/Users/cltbld/.ssh/xrbld_dsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
lost connection

Comment 1

[Brandon Burton [:solarce]]

Investigating, as I'm somewhat familiar with the web service side of this

Comment 2

[Brandon Burton [:solarce]]

Seeing failure from some nodes, roughly 1/3rd today

[[e-mail]1 log]# grep xr secure | grep 'Mar 20' | grep Failed | wc -l
32
[[e-mail]1 log]# grep xr secure | grep 'Mar 20' | grep -v Failed | wc -l
96

But we only see Zeus's IP in /var/log/secure :/

I turned on the basic Zeus logging and we can most likely correlate times to determine which nodes are failing if rail's investigation doesn't find which nodes are misbehaving

Comment 3

[Rail Aliiev [:rail]]

xrbld private key on some machines is different, back to releng to investigate

Comment 4

[Chris Cooper [:coop] (he/him)]

This sounds like a fix for Puppet.