83243 - corruption of cvs mirror on lounge - IO error encountered

Status: RESOLVED WORKSFORME

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dawn Endico]

bonsai was broken again, indicating corrupted files in lounge's 
mirror of cvs. There were 4 rsyncs going so i killed them all and
restarted my own with -c to do a checksum. The command was copied
from crontab. 

Some problems:

1) IO Error encountered. Is the disk having problems?

2) why is it complaining about #cvs files? shouldn't those be ignored.
One of the rsync instances is missing "--exclude="#cvs*". Should it also
exclude lock files?

3) my rsync hasn't finished yet and it seems to be hung. or maybe its just
being quiet now that its no longer deleting lock files.

rsync -actl --delete --exclude="/CVSROOT/*" --exclude="#cvs*"
cvs.mozilla.org::mozilla /cvsroot/
mozilla/webshell/src/#cvs.rfl.thelizard.mozilla.org.2225: No such file or
directory
mozilla/webshell/tests/#cvs.lock: No such file or directory
mozilla/extensions/vixen/build/winbuild/#cvs.lock: No such file or directory
mozilla/netwerk/protocol/res/public/#cvs.rfl.thelizard.mozilla.org.2374: No such
file or directory
mozilla/gc/boehm/xSYM/#cvs.rfl.thelizard.mozilla.org.2554: No such file or
directory
mozilla/embedding/components/windowwatcher/public/#cvs.rfl.thelizard.mozilla.org.2571:
No such file or directory
IO error encountered - skipping file deletion

Comment 1

[Risto Kotalampi]

1) IO errors: this is rather common. When rsync scans the disk things change 
before they are copied. Ie. a file is found but it's not there later.

2) why --cvs-ignore was disables: see http://bugzilla.mozilla.org/show_bug.cgi?
id=80587

3) crc option takes some time.

Comment 2

[Dawn Endico]

yes, i understand why cvs-ignore was disabled. However, it was replaced
by --exclude="#cvs*", and it seems like this argument is being ignored,
otherwise I wouldn't have seen all the 'no such file or directory'
messages for the lock files. right?

reopening this bug because corruption in this mirror is a pretty regular
occurance and it needs to be solved. Not sure what's causing it but please
investigate. When file corruption occurs, bonsai is unusable for the 
affected files until the rsync -c happens, which is once per day at 5am.

Corruption has happened twice in the last couple days. Each time, there
were multiple copies of rsync (and the starter shell) running at once.
None of these processes were doing checksums. It seems like we have corruption
because of overlapping rsyncs.

Comment 3

[Dawn Endico]

reassigning to new serverops owner

Comment 4

[Risto Kotalampi]

--cvs-ignore was not replaced by --exclude="#cvs", that was already there.

Comment 5

[Risto Kotalampi]

I made a quick script and replaced it to cron (like wouldn't have anything 
better to do Saturday afternoon): /usr/local/bin/sync_cvs.pl. This will do some 
locking and won't let two rsyncs to run same time. It will do crc with an 
option -c.

Comment 6

[Risto Kotalampi]

Moving tickets to Ray.

Comment 7

[Risto Kotalampi]

Mass changing IC's ticket to reflect current situation.

mozilla.org, AOL employees:

If you want IC to look at issues reported in bugzilla, please open a Helpdesk
ticket and ask it to be routed to AOL R1 Server Operations. We currently have no
way to handle comprehensive problem resolution through bugzilla. This is not a
change in the way we are supporting mozilla.org - we are still supporting you on
the level as before. IC's support is based on Helpdesk ticket system - not
bugzilla which only few hard-core people are looking at. 

Also, projects are handled elsewhere - not in bugzilla. If you have projects you
need us to deliver please feel free to contact me directly.

Summa summarum: tickets -> Helpdesk
Project initiations -> [e-mail]

Comment 8

[Dave Miller [:justdave]]

Is this still an issue?  We appear to still have the script in place that sets a
lock, and the new servers are much faster, to boot.  I haven't heard of anyone
reporting this in a long time.

Comment 9

[Gervase Markham [:gerv]]

No-one seems to think this is still an issue - closing.

Gerv