586620 - Can't check out from cvs.mozilla.org / cvs-mirror.mozilla.org

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Mark Banner (:standard8)]

The comm-central tinderboxes are trying to checkout from cvs-mirror and are failing, I can't check out locally either.

Both cvs-mirror.mozilla.org and cvs.mozilla.org are failing, afaik there is no planned downtime for these.

$ cvs -d :pserver:[e-mail]:/cvsroot -q checkout -P -r LDAPCSDK_6_0_6D_MOZILLA_RTM -d c-sdk mozilla/directory/c-sdk
cvs [checkout aborted]: unrecognized auth response from cvs-mirror.mozilla.org: execle: cvs: No such file or directory

$ cvs -d :pserver:[e-mail]:/cvsroot -q checkout -P -r LDAPCSDK_6_0_6D_MOZILLA_RTM -d c-sdk mozilla/directory/c-sdk
cvs [checkout aborted]: connect to cvs.mozilla.org(63.245.208.155):2401 failed: Operation timed out

Comment 1

[Shyam Mani [:fox2mike]]

Looking into this.

Comment 2

[Shyam Mani [:fox2mike]]

Okay, I'm unsure of what the issue is at this point. We rebooted dm-cvs01 and 02 yesterday and they came back fine. xinetd is running and I show the cvspserver running on 2401, but connections from the outside don't seem to make it to the box.

Comment 3

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Talos is failing as well, and that's a really blocking issue for any push on mozilla central too. Please update us on situation.

Comment 4

[Shyam Mani [:fox2mike]]

So jailkit is to blame for this.

06:41:14 < dmoore> fox2mike: hmm... that directory got touched at 4:00 this morning
06:41:25  * fox2mike looks for crons
06:41:40 < dmoore> along with lib
06:42:00 < fox2mike> [root@dm-cvs02 bin]# cat /etc/cron.daily/jk_update 
06:42:00 < fox2mike> #!/bin/bash
06:42:01 < fox2mike> /usr/sbin/jk_update -c /cvs/jail
06:42:23 < fox2mike> and from crontab 
06:42:23 < fox2mike> 02 4 * * * root run-parts /etc/cron.daily
06:43:24 < fox2mike> jailkit
06:43:29 < dmoore> I think we're getting warmer
06:43:34 < fox2mike> [root@dm-cvs02 bin]# rpm -qf /usr/sbin/jk_update
06:43:35 < fox2mike> jailkit-2.11-1.el5.rf
06:43:40 < fox2mike> I bet my ass off
06:43:47 < fox2mike> that jailkit was updated yesterday
06:44:11 < fox2mike> okay dmoore who's the daddy now? :p
06:44:12 < fox2mike> [root@dm-cvs02 bin]# grep jailkit /var/log/yum.log
06:44:13 < fox2mike> Aug 11 07:58:10 Updated: jailkit-2.11-1.el5.rf.i386
06:44:33 < dmoore> [root@dm-cvs02 log]# jk_update -c /cvs/jail
06:44:33 < dmoore> removing outdated file /cvs/jail/usr/bin/cvs
06:44:34 < dmoore> lol

So that's the cause. jailkit was updated to a new version and that basically removed the cvs binary, which caused cvs-mirror to hose itself.

Working on fixing it.

Comment 5

[Shyam Mani [:fox2mike]]

Dave's our jailkit man and is taking over to fix this.

Comment 6

[Dave Miller [:justdave]]

Should be fixed now.

dmoore wasn't showing you the entire output from jk_update.  It was removing the cvs binary because it was old, and then was supposed to put back the new one, but that didn't happen because the script crashed before it got that far.

We picked up a new version of jailkit along with the rest of the security updates the other night.  The new version of jailkit doesn't like symlinks, and the path to the jail that was in the config was a symlink.  Changing the config to use the real path to the jail fixed most of the problem.

The other change was that the new version of ld has an /etc/ld.so.conf.d/ directory with additional config to load in it, and jailkit was only configured to copy /etc/ld.so.conf itself into the jail.  Adding the /etc/ld.so.conf.d/ directory to the jail configuration resolved the remaining issues.