1480839 - OSX Exceptions

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Comment 1

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

So, for the moment I quarantined those machines to not take jobs and I'll try a revert for them.

Comment 2

[Dorel Luca [:dluca]]

Trees are closed for this issue

Comment 3

[:dhouse]

This was triggered by my putting generic-worker v10.11.2 onto the staging pool.

These workers are not in the production worker-type, but in https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010-beta

They picked up production jobs, when I tested this change. Example:

https://tools.taskcluster.net/groups/fUM3vxG7TO2JKc2_decHfA/tasks/eivABcy-Q3qAr7XHMsCU-Q/details

Confirmed in the logs the machine has the wrong, production workerType listed (does not show up in the worker explorer under that type):
```
Aug 03 10:23:42 t-yosemite-r7-380.test.releng.mdc1.mozilla.com generic-worker:   "workerId": "t-yosemite-r7-380",
Aug 03 10:23:42 t-yosemite-r7-380.test.releng.mdc1.mozilla.com generic-worker:   "workerType": "gecko-t-osx-1010", 
[...]
Aug 03 10:23:43 t-yosemite-r7-380.test.releng.mdc1.mozilla.com generic-worker: 2018/08/03 10:23:43 Reclaiming task eivABcy-Q3qAr7XHMsCU-Q at 2018-08-03 17:40:43.181 +0000 UTC 
```

Comment 4

[:dhouse]

These workers are quarantined until the fix is applied. I'll include the fix in my patch for bug 1480412

Comment 5

[:dhouse]

Attachment: fixing the workerType

Comment 6

[:dhouse]

I've applied the fix on the four staging machines, still pinned to my puppet environment. And they all correctly show the:
  "workerType": "gecko-t-osx-1010-beta",

Comment 7

[:dhouse]

Comment on attachment 8997535 [details] [review]
fixing the workerType

This was r+'d and merged through bug 1480412

Comment 8

[Intermittent Failures Robot]

101 failures in 472 pushes (0.214 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 51
* autoland: 50

Platform breakdown:
* osx-10-10: 93
* macosx64-qr: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1480839&startday=2018-08-03&endday=2018-08-03&tree=all

Comment 9

[:dhouse]

The workerType fix prevented this from repeating.