694427 - cvs.mozilla.org giving errors when trying to check out "Permission denied (publickey,gssapi-with-mic)."

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Starting at about 9:50 this morning, Camino tinderboxen (which still pull build configs from cvs) stopped working.

After some inspection, it seems that cvs.mozilla.org started responding with "Permission denied (publickey,gssapi-with-mic)." errors for Camino tinderboxen ("caminobld").

Further, I just tried to cvs up one of my trees for CVS-related files, and I get the same error.

Did something happen to CVS today?  I see bug 694249, but that (as far as I can tell) seems like Bear changed his key and needed the new key added to CVS; in this bug's case, however, neither caminobld nor I have changed our keys.

Comment 1

[Mike Taylor [:bear]]

even after the updating of key and redoing my account, I'm still getting errors like you have posted

Comment 2

[Kai Engert [:KaiE:]]

me too

Comment 3

[Jake Maul [:jakem]]

Where are you guys logging in from? a VPN of some kind or... ?

Comment 4

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I get the same issue with:
    
    cvs -d:ext:[e-mail]:/cvsroot co NSS NSPR

from the Mozilla SF office "Mozilla" wifi, no VPN.

Comment 5

[Mike Taylor [:bear]]

I always have the buildvpn active

Comment 6

[Dave Miller [:justdave]]

dumitru and I are investigating.  All evidence so far says it's quite likely related to bug 694249 somehow.

Comment 7

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

FWIW, I'm just a regular old external-person-over-public-Internet, but the Camino tinderboxen are on the Mozilla community network.

This error happened once before, maybe 6 months ago?  I'd search for the old bug, but at this moment I'm away from the house and office on a very flaky wireless connection :P

Comment 8

[Dave Miller [:justdave]]

as part of the other bug, bear got deleted and readded.  The addcvsuser script got run accidentally without an argument when re-adding him the first time.  It's apparently missing a sanitycheck to ensure that you passed in a username (which will be fixed shortly).  The script does a chmod 700 on the newly-created user homedirectory after creating the user.  chmod 700 /cvsdisk/jail/users/$USERNAME with a null $USERNAME caused /cvsdisk/jail/users to become mode 700, thus locking everyone out from seeing their own home directories (and thus preventing sshd from finding your ssh keys since it setuids to your userid before reading it).

chmod 711 on /cvsdisk/jail/users fixed it.

Comment 10

[Dave Miller [:justdave]]

Just for the record, the sanitycheck to make sure a username gets passed was added to all 6 variants of the script.  It also turns out that useradd is quite happy to add a user with a null name to the system (which it successfully did), which is why the whole thing didn't die and acted on an empty username.  It had been counting on the exit code from useradd to tell it if it was legal or not (and of course useradd succeeded).

Comment 11

[Kai Engert [:KaiE:]]

Still broken for me.

Comment 12

[Dave Miller [:justdave]]

This appears to be an unrelated problem specific to your account.  Can you file a new bug? Go ahead and assign it to me, I'm looking into it already (but from looking at the logs, it looks like it's not affecting anyone other than you).

Comment 13

[Kai Engert [:KaiE:]]

Dave, thanks a lot for working on this! I was already asleep when you asked me to file a new bug. I now found the messages you sent to me on IRC.

I confirm this is now fixed for me!