600523 - w32-ix-slave34,36 having connectivity issues

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Chris AtLee [:catlee]]

We've had it disconnect over a dozen times since hooking it up yesterday.  It usually gets disconnected while running hg.

Current IP address: 10.12.48.60.

Comment 1

[Chris AtLee [:catlee]]

http://buildbot-master1.build.mozilla.org:8010/buildslaves/w32-ix-slave34?numbuilds=100

Anything that says 'retry' or 'exception' is most likely a disconnect.

Comment 2

[Chris AtLee [:catlee]]

w32-ix-slave36 is also having problems.

Comment 3

[Chris AtLee [:catlee]]

Attachment: screenshot

I managed to catch this when I logged into slave 36.  Notice the balloon popup on the bottom right.

Comment 4

[Derek Moore [:dmoore]]

We're going to swap network cables and switch ports on our next trip to Internap in an attempt to isolate (what is likely) a hardware problem.

Comment 5

[Justin Lazaro [:jlaz] (use needinfo)]

Changed the network cables and switch port, let's see if there's any improvement

Comment 6

[Chris AtLee [:catlee]]

w32-ix-slave34 just hit problems at Fri Oct 1 03:05:17 2010

did you reboot these machines?  they shouldn't have reconnected to the master.

Comment 7

[Justin Lazaro [:jlaz] (use needinfo)]

I did reboot them, my apologies.  What kind of activity are you seeing now?

Comment 8

[Chris AtLee [:catlee]]

(In reply to comment #7)
> I did reboot them, my apologies.  What kind of activity are you seeing now?

slave 34 hit a few disconnects last night, and I've shutdown buildbot for now.  Are you seeing anything on the switch?

Comment 9

[Justin Lazaro [:jlaz] (use needinfo)]

Will file a bug with IX since this might be hardware-related

Comment 10

[[e-mail] (:bhearsum)]

(In reply to comment #9)
> Will file a bug with IX since this might be hardware-related

Any update here?

Comment 11

[Justin Lazaro [:jlaz] (use needinfo)]

Contacted IX support through email to get this looked at, as well as the other machines experiencing various issues

Comment 12

[Justin Lazaro [:jlaz] (use needinfo)]

w32-ix-slave34
w32-ix-slave36

These machines were taken by Chris Williams of IX Systems, to investigate the network connectivity issues.  Will report back when I receive an update from IX

Comment 13

[Justin Lazaro [:jlaz] (use needinfo)]

Updating the bug here from the email:

Asset #4700 - Reported issue: NIC link resets/speed changes in Windows
- Issue not observed on our test network.
- Event log has record of link changes, indicating NIC is downgrading to 100Mbit due to link integrity issues.
- Suggest installing current Intel NIC driver from ftp://ftp.supermicro.nl/driver/LAN/Intel/PRO_v15.5.zip
- Admin credentials were not provided so we cannot test driver or run additional diagnostics.

Comment 14

[[e-mail] (:bhearsum)]

Given that they haven't seen the issue on their network, are we sure it's not faulty hardware or cabling at SCL?

Comment 15

[matthew zeier [:mrz]]

Comment #4 suggests that was tried.

Comment 16

[Justin Dow [:jabba]]

Both machines are back online at Internap.