651542 - [tracker] IX systems with hardware issues (round 2)

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Amy Rich [:arr] [:arich]]

IX systems that are broken (some of them again, after returning):

Newly broken:

buildbot-master1 (bug 644991)
linux-ix-slave23 (bug 650655)
linux-ix-slave29 (bug 643397)
linux64-ix-slave03 (bug 643614)
linux64-ix-slave14 (bug 646161)
linux64-ix-slave17 (bug 650805)
linux64-ix-slave35 (bug 648312)  (possibly)

Still broken:

linux-ix-slave13 (bug 651325)

Comment 1

[Amy Rich [:arr] [:arich]]

linux-ix-slave34 (bug 652529)

Comment 2

[Zandr Milewski [:zandr]]

Two updates (this bug makes more sense than bug 650355 for this):

As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=650335#c10, iX is going to provide the test script they're using. I actually got that late this afternoon, and will take a look at running it tomorrow. They PXE boot FreeBSD, but the script looks reasonably portable. This will give us apples-to-apples data. They did acknowledge that they're seeing repeat offenders in the machines they haven't finished yet.

I'm approaching the end of my rope with these things, though. The likely replacement (with MUCH better management) is an HP that won't start shipping for another week, though. I'm hesitant to jump on to a new platform sight unseen, (especially since this price point is new territory for HP) but we do have a few coming as soon as they actually start shipments.

Comment 3

[Chris Cooper [:coop] (he/him)]

mv-moz2-linux-ix-slave13 segfaulted during an scp today:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1304712089.1304713765.15362.gz

Log excerpt:

http://pastebin.mozilla.org/1220064

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

mv-moz2-linux-ix-slave13 hasn't had any further trouble since then.  Let's not piggyback bugs, anyway.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I'm closing this in favor of these machines landing in a batch for bug 655304.