1004813 - Investigate Windows 8 machines that are still out of action

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Armen [:armenzg]]

I don't know if all are related to the graphics situation. Please debug each case with Q.
t-w864-ix-018
t-w864-ix-020
t-w864-ix-021
t-w864-ix-026
t-w864-ix-063
t-w864-ix-070

Video of non-working fakemoni.vbs: https://bugzilla.mozilla.org/attachment.cgi?id=8415450

Comment 1

[Phil Ringnalda (:philor)]

Or maybe if we tell Q this bug exists, he'll debug it on his own, since it appears nobody is going to "debug each case with" him.

Comment 2

[Justin Wood (:Callek)]

Q, any estimation of effort needed here, fixing time, etc?

Comment 3

[Chris Cooper [:coop] (he/him)]

Q: ping - we could really use the extra capacity provided by these 6 nodes. When will you or Mark be able to start investigating?

Comment 4

[Justin Wood (:Callek)]

Since we're tracking the individual slaves in the Buildduty Queue, I'm handing this bug over to the relops component, and additionally needinfo to amy since its been almost two weeks since my own needinfo, and almost a month since the initial self-assigned nature of the bug.

Comment 5

[Amy Rich [:arr] [:arich]]

I'll add this to the list of things to work on in the following week.

Comment 6

[Mark Cornmesser [:markco] OOO 2024/04/15]

t-w864-ix-003.wintest.releng.scl3.mozilla.com should be added to this list.

Comment 7

[Q]

Please re-enable 
t-w864-ix-003
and
t-w864-ix-021

Comment 8

[Justin Wood (:Callek)]

what was the cause of the issue, what was the remedy, is this a systemic issue, random issue, or hardware issue?

Is there any progress/plan/etc with the other machines, remedies to prevent more falling into this state, etc.?

(p.s. sorry about other bug mini-rant went too fast and saw that one was resolved, and thought I was looking at this one)

Comment 9

[Q]

Please re-enable 
t-w864-ix-042

Comment 10

[Q]

Commenting in each individual machine bugs attached here. The only one with a misconfiguration was 021.

Comment 11

[Amy Rich [:arr] [:arich]]

coop: so have you guys been adding nodes back into the pool as they're being fixed?

Comment 12

[Chris Cooper [:coop] (he/him)]

(In reply to Amy Rich [:arich] [:arr] from comment #11)
> coop: so have you guys been adding nodes back into the pool as they're being
> fixed?

No, or at least, not until now. I've re-enabled the slaves from comment #7 and comment #9.

Comment 13

[Amy Rich [:arr] [:arich]]

Adding t-w864-ix-001 to this list since it had issues in bug 1036627 (no graphics card shows up).

Comment 14

[Justin Wood (:Callek)]

any clue what the status is here? Last comments were from july... and we have (at least) 5 machines out of commission due to this, in one of our largest pained platforms

Comment 15

[Amy Rich [:arr] [:arich]]

coop asked that we drop priority in favor of other bugs.

Comment 16

[Phil Ringnalda (:philor)]

Have we successfully reimaged and put other Win8 slaves back into production since? If so, fine, focus on buying more instead, but I have the feeling that this bug is actually either "we can't reimage Win8 slaves and get working graphics" or "it's pretty nearly totally random whether or not a reimaged Win8 slave will have working graphics and our bus number for figuring out whether or not one does and fixing it if it does not is exactly 1" which doesn't bode especially well for a pile of new ones.

Comment 17

[Amy Rich [:arr] [:arich]]

The problem is that these *specific* slaves have a variety of different and hard to diagnose problems.

Comment 18

[Phil Ringnalda (:philor)]

Tough to tell for sure, but I think the answer to my question is that bug 1011673 was as close as we've come to a success since May, while we've added three more things to this since then.

Comment 19

[Q]

All machines in slavealloc minus:
t-w864-ix-002 bug 1079396 - loaned to jrmuizel        
t-w864-ix-003 bug 1080023 - loaned to markco      
t-w864-ix-144 Bug 1093488 - Change on-board video setting for moved win8 machines     
t-w864-ix-148 Bug 1093488 - Change on-board video setting for moved win8 machines     
t-w864-ix-156 Bug 1093488 - Change on-board video setting for moved win8 machines


are back in after a hand check of each.