1104741 - Tree closed for Windows 8 Backlog

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Carsten Book [:Tomcat]]

-> philor: 2038 win8 jobs pending, and only 25 slaves that have done a job in the last 5 hours?

pete: could you take a look, thanks!

Comment 1

[Carsten Book [:Tomcat]]

closed fx-team, m-i and mc for this problem

Comment 2

[Pete Moore [:pmoore][:pete]]

Rebooted 151 windows 8 slaves:

t-w864-ix-001
t-w864-ix-004
t-w864-ix-005
t-w864-ix-006
t-w864-ix-008
t-w864-ix-009
t-w864-ix-010
t-w864-ix-012
t-w864-ix-013
t-w864-ix-014
t-w864-ix-015
t-w864-ix-016
t-w864-ix-017
t-w864-ix-018
t-w864-ix-019
t-w864-ix-020
t-w864-ix-021
t-w864-ix-022
t-w864-ix-023
t-w864-ix-024
t-w864-ix-026
t-w864-ix-027
t-w864-ix-028
t-w864-ix-029
t-w864-ix-030
t-w864-ix-031
t-w864-ix-032
t-w864-ix-033
t-w864-ix-034
t-w864-ix-035
t-w864-ix-036
t-w864-ix-037
t-w864-ix-039
t-w864-ix-040
t-w864-ix-041
t-w864-ix-042
t-w864-ix-044
t-w864-ix-046
t-w864-ix-047
t-w864-ix-048
t-w864-ix-049
t-w864-ix-050
t-w864-ix-051
t-w864-ix-052
t-w864-ix-053
t-w864-ix-054
t-w864-ix-055
t-w864-ix-056
t-w864-ix-058
t-w864-ix-059
t-w864-ix-060
t-w864-ix-061
t-w864-ix-062
t-w864-ix-063
t-w864-ix-065
t-w864-ix-066
t-w864-ix-067
t-w864-ix-068
t-w864-ix-069
t-w864-ix-070
t-w864-ix-071
t-w864-ix-072
t-w864-ix-073
t-w864-ix-074
t-w864-ix-075
t-w864-ix-076
t-w864-ix-078
t-w864-ix-079
t-w864-ix-080
t-w864-ix-081
t-w864-ix-082
t-w864-ix-083
t-w864-ix-084
t-w864-ix-085
t-w864-ix-086
t-w864-ix-087
t-w864-ix-088
t-w864-ix-089
t-w864-ix-090
t-w864-ix-091
t-w864-ix-092
t-w864-ix-093
t-w864-ix-094
t-w864-ix-095
t-w864-ix-096
t-w864-ix-100
t-w864-ix-101
t-w864-ix-102
t-w864-ix-103
t-w864-ix-104
t-w864-ix-105
t-w864-ix-106
t-w864-ix-107
t-w864-ix-108
t-w864-ix-109
t-w864-ix-110
t-w864-ix-112
t-w864-ix-113
t-w864-ix-114
t-w864-ix-115
t-w864-ix-116
t-w864-ix-118
t-w864-ix-119
t-w864-ix-120
t-w864-ix-121
t-w864-ix-122
t-w864-ix-123
t-w864-ix-124
t-w864-ix-125
t-w864-ix-126
t-w864-ix-127
t-w864-ix-129
t-w864-ix-130
t-w864-ix-131
t-w864-ix-132
t-w864-ix-133
t-w864-ix-134
t-w864-ix-135
t-w864-ix-136
t-w864-ix-137
t-w864-ix-138
t-w864-ix-139
t-w864-ix-140
t-w864-ix-141
t-w864-ix-142
t-w864-ix-143
t-w864-ix-144
t-w864-ix-145
t-w864-ix-146
t-w864-ix-148
t-w864-ix-149
t-w864-ix-150
t-w864-ix-151
t-w864-ix-152
t-w864-ix-153
t-w864-ix-154
t-w864-ix-155
t-w864-ix-156
t-w864-ix-157
t-w864-ix-158
t-w864-ix-159
t-w864-ix-160
t-w864-ix-161
t-w864-ix-162
t-w864-ix-163
t-w864-ix-164
t-w864-ix-165
t-w864-ix-166
t-w864-ix-167
t-w864-ix-169
t-w864-ix-170

Comment 3

[Chris Cooper [:coop] (he/him)]

Attachment: Current display on t-w864-ix-043

Comment 4

[Amy Rich [:arr] [:arich]]

According to coop, some just have the windows explorer window up.

The conjecture is that something systemwide went out last night that caused a w8 hosts to change resolution *during* a job and then at subsequent boots the machines have not started buildbot due to an incorrect resolution.

Comment 5

[Chris Cooper [:coop] (he/him)]

Other slave just seem to have an explorer window open:

http://imgur.com/40KhulF

Slaves in this state:
* t-w864-ix-011
* t-w864-ix-025
* t-w864-ix-111

Comment 6

[Q]

Nivida was trying to update driver version at least according to event viewer starting on 11/24/2014 at around 1930 PST. At boot the driver version was being reverted and the process started again. We explicitly turn off the "nvidia update service" but it appears that isn't enough on win 8. The geforce experience ( which we don't use anyway) needs to be disabled. This is doen via the nvidia_updates gpo with a registry remove of :
"NvBackend"="\"C:\\Program Files (x86)\\NVIDIA Corporation\\Update Core\\NvBackend.exe\""
"ShadowPlay"="C:\\windows\\system32\\rundll32.exe C:\\windows\\system32\\nvspcap64.dll,ShadowPlayOnSystemStart"

after a remove and reboot on a test machine the update cycle stopped. This is holding true on other machines as well.

Comment 7

[Chris Cooper [:coop] (he/him)]

I've rebooted all t-w864-ix machines now via slaveapi. They should be back at the proper resolution and taking jobs within 30min.

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I've reopened the trees, since we're running out of new jobs to catch the slaves acting badly from, and I don't feel like reading try failures all night.

Comment 9

[Pete Moore [:pmoore][:pete]]

Zero pending Windows 8 test jobs. Closing!

Thanks Q for the awesome troubleshooting and resolution!

Pete