869560 - multiple failed uploads with ssh permission denied

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task, P1)

Description

[Chris AtLee [:catlee]]

trees are closed right now because some hosts can't upload files to upload1

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Systems folks - I see some messages about puppet having trouble with the ffxbld ssh key.  Can you have a look?

Comment 2

[Amy Rich [:arr] [:arich]]

catlee said that enough jobs were burned to close the trees, but that every machine he checked afterwards worked.  AJ: can you cross check the ssh log files to see if the errors corresponded to in between some puppet runs that broke/fixed things?

Comment 3

[Adrian J Fernandez [:Aj]]

Logs shows changes starting from May  7 09:27:41 relating to users that have ffxbld in their name (there's also a stage user). The last notice relating to the ffxbld user was @ May  7 10:00:06

Could provide snippets of logs (from /var/log/messages) but not sure if bug Security would need to be changed.

Comment 4

[Justin Dow [:jabba]]

This was caused by pushing a new SSH authorized_keys provider in puppet, which was tested successfully beforehand, but seemed to give some unexpected results in our infrastructure (causing ssh keys to be removed from $USER/.ssh/authorized_kes. We've reverted all work relating to that and will proceed with CAB process before attempting this change again.

Comment 5

[Chris AtLee [:catlee]]

aus3-staging is also affected

`ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org` from bld-lion-r5-{068,071} is failing

Comment 6

[Adrian J Fernandez [:Aj]]

For historical purposes the issue has long been fixed (right after Comment 5). Will punt over to jabba to clarify was done at the end.

Comment 7

[Mark Banner (:standard8)]

We've just hit this on Thunderbird Nightly builds (Linux & Mac - windows failed due to other reasons), example logs:

https://tbpl.mozilla.org/php/getParsedLog.php?id=22725700&tree=Thunderbird-Trunk
https://tbpl.mozilla.org/php/getParsedLog.php?id=22725924&tree=Thunderbird-Trunk
https://tbpl.mozilla.org/php/getParsedLog.php?id=22727002&tree=Thunderbird-Trunk

Comment 8

[Justin Dow [:jabba]]

What's the public key that are being used by those? And what username?

Comment 9

[Mark Banner (:standard8)]

(In reply to Justin Dow [:jabba] from comment #8)
> What's the public key that are being used by those? And what username?

I used to know that, but I'm not totally certain, so I'll redirect to Chris as he's on buildduty.

Comment 10

[Chris AtLee [:catlee]]

the log says things like this:
Executing: ['bash', '-c', 'ssh -l tbirdbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Thunderbird/comm-central/Linux_x86-gcc3/20130507030514/en-US']

So we're using the auspush key to login as tbirdbld.

Comment 11

[Justin Dow [:jabba]]

Ok, this had the same problem as the ffxbld user. That key never was put into ldap, so puppet didn't know to put it back after the little snafu yesterday. I've added it and made sure that puppet added it back now.