76462 - cvs mirror spewing errors

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[timeless]

cvs-mirror is virtually unusable.  Most times i try to cvs diff, i get a 
timeout.  right now i'm using a pull script (config/fast-update.pl) to cvs 
update files, it's only supposed to update 24hrs worth of files. I have a good 
net connection (and it's past 3am, so even normal college students should be 
sleeping).

Comment 1

[André Dahlqvist]

I have seen this too since last night, and it still happens this morning. When
doing a checkout I often get "resource temporarily unavailable", and when it
sometimes actually starts it's *horribly* slow.

Comment 2

[James Green]

It's unusable from here in the UK as well, I'm not seeing any error message,
just no response, or a response if you care to wait an hour or two.

Comment 3

[NilsE]

It does "work", it's just veeeeeeeryyyyy slow. I'm running an CVS update for 5
hours now and strace the cvs process. About 2 files per hour get through.

Comment 4

[R.K.Aa.]

cvs [login aborted]: recv() from server cvs-mirror.mozilla.org: Connection reset
by peer

Comment 5

[Dan Mosedale (:dmosedale, :dmose)]

It's conceivable that there are networking problems between you and cvs-mirror.
 Try searching http://freshmeat.net/ for a program called "mtr" and install
that.  It can be extremely helpful diagnosing this sort of thing.

Comment 6

[Brian King [:kinger]]

If it is happening for different people on different networks, surely it is 
related to the cvs server's network link to the outside world.

Comment 7

[R.K.Aa.]

there is no problem ping'ing the server - response times seem normal - no
packetloss.

Comment 8

[Dan Mosedale (:dmosedale, :dmose)]

Sure, but mtr can provide you with useful info about where the problem actually
is, whether it's directly at sourceforge, or at one of the carriers along the
way (and which carrier).

Comment 9

[R.K.Aa.]

Attachment: mtr: all looks fine

Comment 10

[James Green]

Attachment: mtr over 50 echos: no significant problems

Comment 11

[James Green]

The report I've attached is based on 50 echos sent.. the minimum neccessary for
any accurate reporting, and shows some packet loss, but not nearly enough to
cause any problems.

Comment 12

[R.K.Aa.]

Attachment: dacapo

Comment 13

[James Green]

Sourceforge crew are now rebooting the box to work on it.

Comment 14

[R.K.Aa.]

started this 40 minutes ago. This is how far it's gotten - and how far it's
getting. In another hour or two the server will burp out a handfull files, give
me a lock on something - and then let me hang again.
Tried that once before - hang with a lock for 40 minutes. Then i didn't dare sit
on it any longer so i broke the pull. Afterwards i got thrown out with the
"disconnected by peer" several times when trying to log in again.
That's not a cvs "feature". Something is just plain wrong - and it's on the server.

 make -f mozilla/client.mk checkout
checkout start: Wed Apr 18 22:46:25 CEST 2001
cvs -q -z3 co -P mozilla/client.mk mozilla/build/unix/modules.mk
M mozilla/client.mk
make[1]: Entering directory `/home/dark/DISK'
cvs -q -z3 co -P -r NSPRPUB_CLIENT_BRANCH mozilla/nsprpub

Comment 15

[Dan Mosedale (:dmosedale, :dmose)]

JG: In my experience, 10% packet loss is more than enough to cause significant
problems with CVS.

Comment 16

[André Dahlqvist]

The box is back up, and I'm marking this one FIXED.

Comment 17

[James Green]

Reopening since it's now spewing with problems, updating summary.

Comment 18

[James Green]

OK the story is that Drew is apparently offsite, and I'm talking with a guy via
irc at VA who has physical access to the box but no root.

It spent an alarmingly long time shutting down each service/daemon, rebooted
fine, and looked ok for a few mins. Now people including myself are getting
unrecoverable locks and other fun problems. The guy at VA is paging Drew for
further assistance.

Comment 19

[R.K.Aa.]

Are someone pulling full sources via 300bps modems?
In addition to locks queueing up at an alarming rate, it quits here for me now:

cvs [server aborted]: unable to parse
/cvsroot/mozilla/modules/libimg/jpgcom/win32.order,v; `state' not in the
expected place
gmake[1]: *** [real_checkout] Error 1
gmake[1]: Leaving directory `/home/dark/DISK'
gmake: *** [checkout] Error 2
[dark@localhost mozilla]$ 

I deleted the whole file + occurance of it in CVS/Entries, but that didn't
change anything.

Comment 20

[Boris Zbarsky [:bzbarsky]]

I get:

cvs -d :pserver:[e-mail]:/cvsroot -q -z 3 co -P -D Wed
Apr 18 17:50:23 EDT 2001 mozilla/modules/libpr0n
cvs [server aborted]: unable to parse
/cvsroot/mozilla/modules/libpr0n/Makefile.in,v; `state' not in the expected place

do we just have corrupt RCS files floating around?

Comment 21

[André Dahlqvist]

If you try to login to the server you get:

cvs [login aborted]: connect to cvs-mirror.mozilla.org:2401 failed: Connection
refused

Comment 22

[R.K.Aa.]

The sympthoms kinda indicate the box has been about to swap itself to death.
Perhaps a leak - more likely just plain low on RAM/swapspace.
Someone in n.p.m.builds told the badness started around the time Slashdot
announced that the QT version of moz was now in CVS :)

Comment 23

[James Green]

cvs access has been switched off. I've been liasing with Patrick at VA onsite
and he's now rebuilding the entire mirror... We could have just rsynced again,
but who knows what else could have been around the corner.

The rebuild is from thelizzard.mozilla.org so it's taking a while to complete...
I'll keep ya updated.

Comment 24

[James Green]

OK, the rebuild happened, and we got cvs errors... Dawn got in telephone contact
with Patrick and ran a disk check, this turned out to be too long a process so
the disk was formatted and a new mirror built.

This mirror is finished but only half the size of the original. *shrug*

The current problem is that it won't give us anonymous cvs access. It's 3.20am
here and I can't stay up any longer, so I'll come back tomorrow and hope it's fixed.

(Our thanks to Patrick who has lept into this problem at no notice and spent
most of his day on it.)

Comment 25

[Matthias Versen [:Matti]]

I mark this fixed.

My checkout works without problems now !

Comment 26

[Patrick D. Ashmore]

I believe I have fixed the problem. I just did a CVS checkout to my workstation
and it seems to have pulled everything in just fine.

Please let me know if there still exists any problems. Thanks!

Comment 27

[Brian King [:kinger]]

I can confirm it is working as well.

Comment 28

[timeless]

vrfy anonymous checkouts from cvs-mirror are working for me right now, thanks 
everyone.

Comment 29

[R.K.Aa.]

(.. or could have been a full /var/log or /tmp directory...)

Downloaded source.tar.gz from April 15 and just ran a good checkout.
Only files missing were these - prolly has a natural explanation:

cvs server: cannot open directory
/cvsroot/mozilla/layout/html/tests/frameset/core: No such file or directory
cvs server: skipping directory mozilla/layout/html/tests/frameset/core
cvs server: cannot open directory /cvsroot/mozilla/layout/html/tests/table/core:
No such file or directory
cvs server: skipping directory mozilla/layout/html/tests/table/core
cvs server: cannot open directory /cvsroot/mozilla/layout/html/tests/tags: No
such file or directory
cvs server: skipping directory mozilla/layout/html/tests/tags