Investigate Windows 8 machines that are still out of action

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: armenzg, Assigned: Q)

Tracking

(Blocks: 1 bug)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/483] [time=10:00])

(Reporter)

Description

3 years ago
I don't know if all are related to the graphics situation. Please debug each case with Q.
t-w864-ix-018
t-w864-ix-020
t-w864-ix-021
t-w864-ix-026
t-w864-ix-063
t-w864-ix-070

Video of non-working fakemoni.vbs: https://bugzilla.mozilla.org/attachment.cgi?id=8415450
(Reporter)

Updated

3 years ago
No longer depends on: 977615
(Reporter)

Updated

3 years ago
Blocks: 1004815
(Reporter)

Updated

3 years ago
Blocks: 889309
(Reporter)

Updated

3 years ago
No longer blocks: 974684
(Reporter)

Updated

3 years ago
Blocks: 851280
No longer blocks: 889309
No longer blocks: 905365
No longer blocks: 1004815
Or maybe if we tell Q this bug exists, he'll debug it on his own, since it appears nobody is going to "debug each case with" him.
Blocks: 935916, 856691, 883870
(Assignee)

Updated

3 years ago
Assignee: nobody → q
Blocks: 1027101
Q, any estimation of effort needed here, fixing time, etc?
Flags: needinfo?(q)

Comment 3

3 years ago
Q: ping - we could really use the extra capacity provided by these 6 nodes. When will you or Mark be able to start investigating?
Since we're tracking the individual slaves in the Buildduty Queue, I'm handing this bug over to the relops component, and additionally needinfo to amy since its been almost two weeks since my own needinfo, and almost a month since the initial self-assigned nature of the bug.
Component: Buildduty → RelOps
Flags: needinfo?(arich)
Product: Release Engineering → Infrastructure & Operations
QA Contact: bugspam.Callek → arich
Version: unspecified → other
I'll add this to the list of things to work on in the following week.
Flags: needinfo?(arich)
t-w864-ix-003.wintest.releng.scl3.mozilla.com should be added to this list.
Blocks: 851185
(Assignee)

Comment 7

3 years ago
Please re-enable 
t-w864-ix-003
and
t-w864-ix-021
Flags: needinfo?(q)
what was the cause of the issue, what was the remedy, is this a systemic issue, random issue, or hardware issue?

Is there any progress/plan/etc with the other machines, remedies to prevent more falling into this state, etc.?

(p.s. sorry about other bug mini-rant went too fast and saw that one was resolved, and thought I was looking at this one)
Flags: needinfo?(q)
Flags: needinfo?(arich)
(Assignee)

Comment 9

3 years ago
Please re-enable 
t-w864-ix-042
(Assignee)

Comment 10

3 years ago
Commenting in each individual machine bugs attached here. The only one with a misconfiguration was 021.
Flags: needinfo?(q)
Flags: needinfo?(arich)
coop: so have you guys been adding nodes back into the pool as they're being fixed?
Flags: needinfo?(coop)
(In reply to Amy Rich [:arich] [:arr] from comment #11)
> coop: so have you guys been adding nodes back into the pool as they're being
> fixed?

No, or at least, not until now. I've re-enabled the slaves from comment #7 and comment #9.
Flags: needinfo?(coop)
Adding t-w864-ix-001 to this list since it had issues in bug 1036627 (no graphics card shows up).
Blocks: 866691
(Assignee)

Updated

3 years ago
Whiteboard: [time=10:00]
any clue what the status is here? Last comments were from july... and we have (at least) 5 machines out of commission due to this, in one of our largest pained platforms
Flags: needinfo?(q)
Flags: needinfo?(arich)
coop asked that we drop priority in favor of other bugs.
Flags: needinfo?(q)
Flags: needinfo?(arich)
Have we successfully reimaged and put other Win8 slaves back into production since? If so, fine, focus on buying more instead, but I have the feeling that this bug is actually either "we can't reimage Win8 slaves and get working graphics" or "it's pretty nearly totally random whether or not a reimaged Win8 slave will have working graphics and our bus number for figuring out whether or not one does and fixing it if it does not is exactly 1" which doesn't bode especially well for a pile of new ones.
The problem is that these *specific* slaves have a variety of different and hard to diagnose problems.
Tough to tell for sure, but I think the answer to my question is that bug 1011673 was as close as we've come to a success since May, while we've added three more things to this since then.

Updated

3 years ago
Whiteboard: [time=10:00] → [kanban:engops:https://kanbanize.com/ctrl_board/6/457] [time=10:00]

Updated

3 years ago
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/457] [time=10:00] → [kanban:engops:https://kanbanize.com/ctrl_board/6/460] [time=10:00]

Updated

3 years ago
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/460] [time=10:00] → [time=10:00]

Updated

3 years ago
Whiteboard: [time=10:00] → [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]

Updated

3 years ago
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1622] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1622] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1623] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1623] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1628] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1628] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1629] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1629] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1630] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]
(Assignee)

Comment 19

3 years ago
All machines in slavealloc minus:
t-w864-ix-002 bug 1079396 - loaned to jrmuizel        
t-w864-ix-003 bug 1080023 - loaned to markco      
t-w864-ix-144 Bug 1093488 - Change on-board video setting for moved win8 machines     
t-w864-ix-148 Bug 1093488 - Change on-board video setting for moved win8 machines     
t-w864-ix-156 Bug 1093488 - Change on-board video setting for moved win8 machines


are back in after a hand check of each.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1630] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/483] [time=10:00]
You need to log in before you can comment on or make changes to this bug.