43716 - cvs-mirror's disk full?

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task, P3)

Description

[Ben Bucksch (:BenB)]

I'm getting things like

cvs [server aborted]: cannot open /tmp/cvsPr9s6F: No space left on device

and partly garbage data from cvs-mirror. Assuming I can trust |df|, I have
enough space.

Also,

$ nslookup 198.186.203.47
Server:  ns.de.uu.net
Address:  192.76.144.66
Name:    cvs-mirror.mozilla.org
Address:  198.186.203.47

looks odd to me, as mozilla.org servers
- used to have name of lizards
- are usually in the 207.200.73.0/255.255.255.0 net

Comment 1

[Frank de Lange]

Same here, but I get all sorts of 'cvs [server aborted]: cannot open
CVS/Entries.Log: No space left on device' messages. Definitively something fishy
going on.

Comment 2

[Ben Bucksch (:BenB)]

cvs-mirror being at sourceforge seems to be OK. REASSIGNing to
<[e-mail]>.

Comment 3

[Risto Kotalampi]

I'm rebooting the server. Hopefully it comes back up. The load average was > 40 
but didn't quickly see much else wrong except some kernel panic messages.

Comment 4

[Risto Kotalampi]

Ok, looks good now.

Comment 5

[Risto Kotalampi]

I also run rsync with -c (runs every night anyway) to make sure files are 
identical.

Drew, I'm leaving this open - please close when you've seen it and ack'd.

Comment 6

[Ben Bucksch (:BenB)]

What was the problem (to make sure it doesn't happen again)?

Comment 7

[Risto Kotalampi]

I don't know. Load was > 40. I bounced inetd, no help. I killed all cvs 
processes, no help. I'm leaving it to Sourceforge/Drew to determine because it's 
their server.

Comment 8

[Ben Bucksch (:BenB)]

BTW: rko, thanks for the fast hotfix. At least I could co now :).

Comment 9

[Drew Streib]

Not quite sure what happened, but I'm analyzing log files now. Kernel version is
2.2.16+ (which is current). I'll monitor this closely. -drew