345185 - Checking in to cvs.mozilla.org is much slower since the colo move

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[:Gavin Sharp [email: [e-mail]]]

Since the downtime on the 15th (Saturday), I've noticed that checkins take a lot more time. It takes almost 3-5 seconds per file, where is used to take no more than a second. Martijn said that he has noticed this too, and it's been slow for all of my checkins since then, so it seems that this isn't an isolated case.

Comment 1

[matthew zeier [:mrz]]

I believe this was DNS related - there was no in-addr.arpa for dm-cvs01.mozilla.org.  I've added and as soon as DNS refreshes you should start to see quicker checkins. 

Comment 2

[:Gavin Sharp [email: [e-mail]]]

Assuming DNS has refreshed by now, this still isn't fixed. I just checked in a small change to a single file, and the checkin was still much slower than usual. Others have noticed this too, the fix doesn't seem to have made a difference.

Comment 3

[Jeremy Orem [:oremj]]

Is there a certain spot in the checkin where it seems slower than normal?

Comment 4

[:Gavin Sharp [email: [e-mail]]]

It hangs for up to 10 seconds after each file (after the "Checking in <foo>; <...> done" message appears). Sometimes multiple files (2 or 3) will go through before it hangs.

Comment 5

[Martijn Wargers (dead)]

Yes, that's what I'm seeing too.

Comment 6

[Max Kanat-Alexander]

Yes, I can confirm that checkins are still quite slow, even from a server behind the firewall (landfill).

Comment 7

[Reed Loden [:reed]]

er, did you mean to reassign to marcia?

Comment 8

[Reed Loden [:reed]]

What's the status of this issue?

Comment 9

[:Gavin Sharp [email: [e-mail]]]

I committed a file with cvs -t, and the part that freezes is:

S-> run_popen(/cvsroot/CVSROOT/dolog.pl -h 127.0.0.1 -r /cvsroot [e-mail],w)

I can attach a full log if that would be useful.

Comment 10

[Dave Miller [:justdave]]

The /etc/resolv.conf file inside the chroot jail had the old nameserver IPs from the old colo in it.  I just copied the root-level resolv.conf into the jail, this should fix it.

Since everything is managed by dhcp now, I added the following to /etc/dhclient-exit-hooks to be run after a lease is obtained:

# Copy the freshly-generated /etc/resolv.conf into the chroot jail
if [ -e /etc/resolv.conf ]; then
  cp -pf /etc/resolv.conf /cvsdisk/jail/etc/resolv.conf
else
  # if there isn't one, nuke the one in the jail, too.
  rm -f /cvsdisk/jail/etc/resolv.conf
fi


If this doesn't fix it, please reopen.

Comment 11

[:Gavin Sharp [email: [e-mail]]]

Checking in is now fast again! Thanks Dave.