547131 - talos-r3 master or slaves unwell

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We've had several bunches of talos runs fail out when the slaves lose contact with the master. In particular at 15:22 (5 slaves), 15:37 (17), 17:30-18:00 (50+).

Possibly fallout from bug 546731 ? Master is also gobbling up 1.5G right now, but CPU usage is OK.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

There are lots of "talos dirty" runs that are stuck at "downloading to dirtyMaxDBs.zip". I feel like restarting the master.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Details of talos failures

This is all the builds today that ended in exception, meaning they lost connection between slave and master.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

dmoore, do the times correlate with any work you were doing ?

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

talos-r3 master got a stop/start in a quiet period, with a purge_events for good measure. talos-pool has a similar problem with stalled builds but I haven't touche that.

Comment 5

[[e-mail] (:bhearsum)]

The last time we had machines fail in downloading dirtyDBs.zip it was because the slaves had a bad version of Twisted IIRC.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

There were 4 more premature disconnects at 22:11 PST on the 18th, two each of fedora and leopard boxes. Nothing else between then and now. Seems to me there could be network congestion issue between the slaves in MV and the master in MPT, so I think we should leave this open to see what happens when MV arrives back at work. Alternatively, twice as many slaves connecting to talos-master may be saturating the network connection to communicate logs and download files. Filed bug 547602 to add munin monitoring.

Note that there are other issues causing problems for r3 jobs
 * timing out downloading symbols on mac - should be fixed by bug 546939
 * "talos dirty" jobs for XP and Leopard are timing out retrieving dirtyMaxDBs.zip from the master, bug 547600

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

alice: is this still happening?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: More recent failures

Still some failures occurring. This is all the 'exception' results since those in attachment 427704 [details], including the 4 I mentioned in comment 6. I verified that a couple are lost connections between master and slave.

Comment 9

[alice nodelman [:alice] [:anode]]

Is this still occurring now that rev3 master is in production?

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: But wait there's more

Don't know which of these line up with our recent problems.

Comment 11

[alice nodelman [:alice] [:anode]]

Nothing new here since the 9th - still an issue?

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

One yesterday (tracemonkey-xp-v8 218 at 2010-03-23 00:24:51 UTC), then lots a week ago, otherwise fine.