57907 - cvs-mirror.mozilla.org is not updating

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task, P3)

Description

[David Krause]

cvs-mirror.mozilla.org is not updating.  I haven't seen any new files appear in
two days now.  cvs stat shows that it thinks I am up-to-date while actually
using bonsai to compare file versions the cvs-mirror server is behind.

Comment 1

[timeless]

In the past this was caused by being out of disk space.

Comment 2

[Jason Eager]

I can definately confirm this. Is there any way to pull from the main cvs in the
meantime?

Comment 3

[Drew Streib]

I get an access denied from cvs.mozilla.org. Could you check the
permissions/allowed hosts, Dawn?

Comment 4

[Dawn Endico]

hosts allow seems to be correct.

from the rsync log file, here is the last successful rsync from source forge
and the most recent failure. They're both from the same IP address so this
looks like a dns problem, not a config problem. /etc/rsyncd.conf was last
changed sept 26.  OTOH, nslookup seemed to behave correctly.  The sept 26
change set max connections to 1. Could that be the problem? Also while
we're at it could we fix the lines in the conf file that are causing all
the error messages? (transfer logging, use chroot, etc)

this seems like the problem is on thelizard. reassining to risto.

2000/10/22 19:19:52 [27202] rsync on mozilla from sourceforge.net (198.186.203.33)
2000/10/22 19:20:24 [27202] wrote 1142229 bytes  read 94 bytes  total size 930499981

2000/10/25 12:29:46 [2940] reverse name lookup failed
2000/10/25 12:29:46 [2940] rsync denied on module mozilla from unknown
(198.186.203.33)

root@thelizard:/etc/> nslookup 198.186.203.33
Server:  ns.netscape.com
Address:  198.95.251.10

Name:    sourceforge.net
Address:  198.186.203.33

Comment 5

[Dan Mosedale (:dmosedale, :dmose)]

It looks as though someone has changed the firewall config on mozilla.org's net
so that it only lets DNS packets to ns.netscape.com (198.95.251.10) through, and
not those to ns2.netscape.com, our backup server, or any other servers for that
matter.  I've worked around this by commenting out the backup server in
/etc/resolv.conf, but this is really bad, since we now have no backup server.
Please fix the firewall rules as soon as possible.

Comment 6

[Dawn Endico]

Mosedale's change didn't make any difference. the attempted rsync at
15:19 failed again.

Note that this firewall horkage is causing other problems as well.
We can no longer send mail between thelizard and lounge. Since mail is
broken this is keeping people from checking in to cvs.mozilla.org
and/or keeping checkins from showing up in bonsai (bonsai.mozilla.org
on lounge.mozilla.org).

This breakage keeps CPD engineers from working. Marking severity as blocker.

Oct 25 15:22:31 thelizard.mozilla.org sendmail[9456]: e9PMLRn09456:
ruleset=check_mail, arg1=[e-mail], relay=localhost
[127.0.0.1], reject=451 4.1.8 [e-mail]... Sender
domain must resolve


ntpdate has also been broken since oct 3
Oct 25 14:00:04 thelizard.mozilla.org ntpdate[6465]: no server suitable for
synchronization found


also it looks like weekly dumps have been failing have been failing
since sept.

USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
root      9842  0.0  0.1  848  704 pts/3    S 15:33:58  0:00 grep vxdump
root      2285  0.0  0.1 2080  640 ?        S   Sep 29  0:00 /usr/sbin/vxdump 6
root      2286  0.0  0.1 2080  640 ?        S   Sep 29  0:00 /usr/sbin/vxdump 6
root     22611  0.0  0.1 2080  648 ?        S   Oct 13  0:00 /usr/sbin/vxdump 6
root     22612  0.0  0.1 2080  648 ?        S   Oct 13  0:00 /usr/sbin/vxdump 6
root     24400  0.0  0.1 2080  640 ?        S   Oct 06  0:00 /usr/sbin/vxdump 6
root     24402  0.0  0.1 2080  640 ?        S   Oct 06  0:00 /usr/sbin/vxdump 6
root     27554  0.0  0.1 2080  648 ?        S   Oct 20  0:00 /usr/sbin/vxdump 6
root     27555  0.0  0.1 2080  648 ?        S   Oct 20  0:00 /usr/sbin/vxdump 6

I called the help desk and filed ticket 265177 which is being sent to
the network team.

reassigning to harishm since I think rko is out of town.

Comment 7

[(not reading, please use [e-mail] instead)]

this turned commercial trunk tinderbox red.  zoinks.

Comment 8

[(not reading, please use [e-mail] instead)]

adding cls to the cc list.

Comment 9

[Risto Kotalampi]

Dumps... don't worry. If the tape is full it leaves these zombies behind. 
Usually it fills before the level 6 backups... thus all those leftover 
processes.

Comment 10

[Risto Kotalampi]

Seems like sourceforge's reverse DNS had been changes, thus *.sourceforge.net 
didn't match anymore and didn't let rsync. I've added cvs-mirror's IP to the 
allowed hosts and it seems to sync fine now. 

Harish, could you verify that this is fixed and if it is please close the 
ticket.

Comment 11

[Risto Kotalampi]

Running commands from command line seems to be syncing fine to cvs-mirror. Btw, 
I also raised limits for rsync max hosts. It was set to 1 because that was when 
we had some preformance problems with MacCVS clients loading the server and in 
elimination tried that too and forgot to put it back to what it was (3).

Comment 12

[Harish Murukanandan]

Verified with Dawn Endico. And everything except the backup dns server is fixed.
There is another ticket alerady open for that.

Comment 13

[David Krause]

Verified.  I just updated my tree and all the changes were retrieved.