multiple failed uploads with ssh permission denied


Status Graveyard
Server Operations
5 years ago
3 years ago


(Reporter: catlee, Assigned: jabba)



(Whiteboard: [buildduty])



5 years ago
trees are closed right now because some hosts can't upload files to upload1
Systems folks - I see some messages about puppet having trouble with the ffxbld ssh key.  Can you have a look?
Assignee: catlee → afernandez
Component: Release Engineering: Automation (General) → Server Operations
QA Contact: catlee → shyam
catlee said that enough jobs were burned to close the trees, but that every machine he checked afterwards worked.  AJ: can you cross check the ssh log files to see if the errors corresponded to in between some puppet runs that broke/fixed things?
Logs shows changes starting from May  7 09:27:41 relating to users that have ffxbld in their name (there's also a stage user). The last notice relating to the ffxbld user was @ May  7 10:00:06

Could provide snippets of logs (from /var/log/messages) but not sure if bug Security would need to be changed.

Comment 4

5 years ago
This was caused by pushing a new SSH authorized_keys provider in puppet, which was tested successfully beforehand, but seemed to give some unexpected results in our infrastructure (causing ssh keys to be removed from $USER/.ssh/authorized_kes. We've reverted all work relating to that and will proceed with CAB process before attempting this change again.
Last Resolved: 5 years ago
Resolution: --- → FIXED

Comment 5

5 years ago
aus3-staging is also affected

`ssh -l ffxbld -i ~/.ssh/auspush` from bld-lion-r5-{068,071} is failing
Resolution: FIXED → ---
For historical purposes the issue has long been fixed (right after Comment 5). Will punt over to jabba to clarify was done at the end.
Assignee: afernandez → jdow
Severity: blocker → normal
We've just hit this on Thunderbird Nightly builds (Linux & Mac - windows failed due to other reasons), example logs:

Comment 8

5 years ago
What's the public key that are being used by those? And what username?
Flags: needinfo?(mbanner)
(In reply to Justin Dow [:jabba] from comment #8)
> What's the public key that are being used by those? And what username?

I used to know that, but I'm not totally certain, so I'll redirect to Chris as he's on buildduty.
Flags: needinfo?(mbanner) → needinfo?(catlee)

Comment 10

5 years ago
the log says things like this:
Executing: ['bash', '-c', 'ssh -l tbirdbld -i ~/.ssh/auspush mkdir -p /opt/aus2/incoming/2/Thunderbird/comm-central/Linux_x86-gcc3/20130507030514/en-US']

So we're using the auspush key to login as tbirdbld.
Flags: needinfo?(catlee)

Comment 11

5 years ago
Ok, this had the same problem as the ffxbld user. That key never was put into ldap, so puppet didn't know to put it back after the little snafu yesterday. I've added it and made sure that puppet added it back now.
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
Product: → Graveyard
You need to log in before you can comment on or make changes to this bug.