872598 - missing home directory on mpt jumphost

Status: VERIFIED FIXED

(Infrastructure & Operations :: Infrastructure: Other, task)

Description

[Robert Kaiser]

Somewhere in the last days, suddenly the mpt jumphost changed and is missing a home directory for me.

In my worfkflow of getting stuff up for my custom reports on crashes, I rely on checking in some adjustments to me own repo and then run a nice nested ssh command like this:

> hg push && ssh-add && ssh vpn "ssh crashanalysis.dmz.phx1.mozilla.com 'cd /mnt/crashanalysis/rkaiser/crash-report-tools/ && hg pull -u && chmod -R a+r /mnt/crashanalysis/rkaiser/crash-report-tools/{overview*,dashboard}'"
[...hg spew]
Identity added: [...]
Could not chdir to home directory /home/rkaiser: No such file or directory
Could not create directory '/home/rkaiser/.ssh'.
Host key verification failed.

Also, when I logged into the mpt jumphost, I usually had a bash history available (so I don't need to remember the host names I tend to access) and didn't always need to confirm the ssh fingerprints of hosts I access (as long as this is always asked and then deleted, it's no security feature but only annoying anyhow for everyone except the three people in the world who actually remember and check those fingerprints manually).

Comment 1

[Peter Radcliffe [:pir]]

:jabba found archived homedirs in /home/archive, from 3am May 10th:
-rw-r--r--    1 root root      2861 May 10 03:09 rkaiser.tar.gz
and is restoring those homedirs.

Theory is the homedir cleanup script running when the host could not connect to ldap since this is a legacy host which is managed directly from ldap and not via puppet.

My missing home directory is not in that set, however, so there may be other missing directories which would need to be manually recreated.

I've suggested the homedir cleanup script be tweaked to not let this happen again before the legacy hosts are retired.

Comment 2

[Justin Dow [:jabba]]

I've brought up the issue with :kang and we'll discuss if some better intelligence can be built into the script. This host is slated to be replaced soon with something with more stable user management vs. using LDAP directly, so it's mostly a temporary problem if it does re-occur.

Comment 3

[Robert Kaiser]

verified that I get my home directory again. Thanks a lot!