multiple failed uploads with ssh permission denied

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
P1
normal
RESOLVED FIXED
5 years ago
3 years ago

People

(Reporter: catlee, Assigned: jabba)

Tracking

Details

(Whiteboard: [buildduty])

(Reporter)

Description

5 years ago
trees are closed right now because some hosts can't upload files to upload1
Systems folks - I see some messages about puppet having trouble with the ffxbld ssh key.  Can you have a look?
Assignee: catlee → afernandez
Component: Release Engineering: Automation (General) → Server Operations
QA Contact: catlee → shyam
catlee said that enough jobs were burned to close the trees, but that every machine he checked afterwards worked.  AJ: can you cross check the ssh log files to see if the errors corresponded to in between some puppet runs that broke/fixed things?
Logs shows changes starting from May  7 09:27:41 relating to users that have ffxbld in their name (there's also a stage user). The last notice relating to the ffxbld user was @ May  7 10:00:06

Could provide snippets of logs (from /var/log/messages) but not sure if bug Security would need to be changed.
(Assignee)

Comment 4

5 years ago
This was caused by pushing a new SSH authorized_keys provider in puppet, which was tested successfully beforehand, but seemed to give some unexpected results in our infrastructure (causing ssh keys to be removed from $USER/.ssh/authorized_kes. We've reverted all work relating to that and will proceed with CAB process before attempting this change again.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 5

5 years ago
aus3-staging is also affected

`ssh -l ffxbld -i ~/.ssh/auspush aus3-staging.mozilla.org` from bld-lion-r5-{068,071} is failing
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
For historical purposes the issue has long been fixed (right after Comment 5). Will punt over to jabba to clarify was done at the end.
Assignee: afernandez → jdow
Severity: blocker → normal
We've just hit this on Thunderbird Nightly builds (Linux & Mac - windows failed due to other reasons), example logs:

https://tbpl.mozilla.org/php/getParsedLog.php?id=22725700&tree=Thunderbird-Trunk
https://tbpl.mozilla.org/php/getParsedLog.php?id=22725924&tree=Thunderbird-Trunk
https://tbpl.mozilla.org/php/getParsedLog.php?id=22727002&tree=Thunderbird-Trunk
(Assignee)

Comment 8

5 years ago
What's the public key that are being used by those? And what username?
Flags: needinfo?(mbanner)
(In reply to Justin Dow [:jabba] from comment #8)
> What's the public key that are being used by those? And what username?

I used to know that, but I'm not totally certain, so I'll redirect to Chris as he's on buildduty.
Flags: needinfo?(mbanner) → needinfo?(catlee)
(Reporter)

Comment 10

5 years ago
the log says things like this:
Executing: ['bash', '-c', 'ssh -l tbirdbld -i ~/.ssh/auspush aus3-staging.mozilla.org mkdir -p /opt/aus2/incoming/2/Thunderbird/comm-central/Linux_x86-gcc3/20130507030514/en-US']

So we're using the auspush key to login as tbirdbld.
Flags: needinfo?(catlee)
(Assignee)

Comment 11

5 years ago
Ok, this had the same problem as the ffxbld user. That key never was put into ldap, so puppet didn't know to put it back after the little snafu yesterday. I've added it and made sure that puppet added it back now.
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.