432570 - talos machines can't reach cvs-mirror

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[:Gavin Sharp [email: [e-mail]]]

Talos machines try to run:
 argv: ['cvs', '-d', ':pserver:[e-mail]:/cvsroot', 'co', '-d', 'talos', 'mozilla/testing/performance/talos']

Starting at about 20:17:02, a bunch of machines turned red, with an error running that command out:

cvs [checkout aborted]: reading from server: Connection reset by peer
program finished with exit code 1

http://qm-rhel02:2006/ shows 8 machines red at the moment.

Comment 1

[Aravind Gottipati [:aravind]]

cvs-mirror should be up now.

Comment 2

[matthew zeier [:mrz]]

Netscaler health check is a ping check on dm-cvs02 so it inadvertently came back up when the physical host came back up.  

nslb02> disable service svc-dm-cvs02-cvs
 Done
nslb02> show gslb  vserver glb-cvs-mirror

*snip*

1)      cvs-mirror-remote (63.245.213.100: 2401) - TCP State: UP        Weight: 1
                 Dynamic Weight: 0       Cumulative Weight: 1
        Location: Europe.Netherlands.*.Amsterdam.GlobalSwitch.*
2)      cvs-mirror-local (63.245.209.14: 2401) - TCP State: DOWN        Weight: 1
                 Dynamic Weight: 0       Cumulative Weight: 1
        Location: NorthAmerica.UnitedStates.California.SanJose.MPT.*

They should eventually find the right CVS mirror - the TTL is short.

Comment 3

[:Gavin Sharp [email: [e-mail]]]

Machines going green again, thanks.