1156356 - All Trees Closed due to issue with windows machines

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Justin Wood (:Callek)]

[12:16:19]	RyanVM|sheriffduty	WTH, suddenly getting oddball Windows failures across multiple trees
[12:16:30]	RyanVM|sheriffduty	and xp/7/8 are all affected
[12:16:31]	RyanVM|sheriffduty	https://treeherder.mozilla.org/logviewer.html#?job_id=9068992&repo=mozilla-inbound
[12:17:18]	bkelly	is there an issue with treeherder mis-marking successes as failures on windows at the moment?
[12:17:31]	RyanVM|sheriffduty	I literally just pinged about the same issue

Filing now while discussion ongoing.

Comment 1

[Justin Wood (:Callek)]

Attachment: IRC log until ~ now

Comment 2

[Justin Wood (:Callek)]

To summarize current theory: Bug 1155257 broke it by being deployed // changing far more then the subset of jobs it was expected to.

We see a 0 byte file tooltool changed about an hour ago on at least one of the XP machines.

The Treeherder job is failing due to not finding minidump_stackwalk which is downloaded via tooltool

Comment 3

[Mark Cornmesser [:markco]]

The GPO has been corrected, and the tooltool file that is being pushed out now is the one originally pushed out 2014-10-21. 

What caused the issue was the GPO source file was altered while I was creating a testing environment for the new tooltool.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Everything is cleaned up and reopened.