573784 - New round of disconnects

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Chris AtLee [:catlee]]

Recently we've been having disconnects of slaves, and problems slaves uploading to stage.

This bug will track these new occurrences.

mv-moz2-linux-ix-slave12.build.mozilla.org to stage.mozilla.org:22 at 10:55:36

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

eg http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1277266826.1277269662.22081.gz

scp codesize-auto.log stage.mozilla.org:/home/ftp/pub/firefox/tinderbox-builds/mozilla-central-linux/
codesize-auto.log                               0%    0     0.0KB/s   --:-- ETA
codesize-auto.log                              46% 8208KB   8.0MB/s   00:01 ETA
codesize-auto.log                              73%   13MB   7.7MB/s   00:00 ETA
codesize-auto.log                              73%   13MB   6.9MB/s   00:00 ETA
codesize-auto.log                              73%   13MB   6.2MB/s   00:00 ETA
codesize-auto.log                              73%   13MB   5.6MB/s   00:00 ETA
codesize-auto.log                              73%   13MB   5.0MB/s   00:00 ETA
codesize-auto.log                              73%   13MB   4.5MB/s - stalled -
codesize-auto.log                              73%   13MB   4.1MB/s - stalled -
...
codesize-auto.log                              73%   13MB   0.0KB/s - stalled -
codesize-auto.log                              73%   13MB   0.0KB/s - stalled -
Read from remote host stage.mozilla.org: Connection timed out
lost connection
program finished with exit code 1
elapsedTime=936.794230

mv-moz2-linux-ix-slave12 at Tue Jun 22 21:49:39 2010, so Castro -> MPT.

Comment 2

[Chris AtLee [:catlee]]

12:42:01 mw32-ix-slave04 disconnected from production-master01
12:40:54 mw32-ix-slave18 disconnected from production-master01

Comment 3

[Chris AtLee [:catlee]]

June 25th, 05:14:40 mw32-ix-slave19:
Read from remote host stage.mozilla.org: Connection reset by peer
lost connection

Comment 4

[matthew zeier [:mrz]]

Nothing since June, inclined to close (and these hosts are moving out of Castro anways).

Comment 5

[Chris AtLee [:catlee]]

Just had one today:

Tue Sep 28 15:11:48 2010 w32-ix-slave34 disconnected from buildbot-master1.build.mozilla.org

Want to re-open or file a new bug?

Comment 6

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1285738466.1285738734.10370.gz

Comment 7

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1285790807.1285792040.23341.gz

Comment 8

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1285790550.1285792729.26223.gz

Comment 9

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1285793286.1285794176.932.gz
http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1285793286.1285794203.1033.gz

Comment 10

[Chris AtLee [:catlee]]

We suspect most (all?) of the above due to physical bumping of the machines / power / network cables while others are getting moved to Santa Clara.