1474299 - (t-yosemite-r7-357) [MDC1] t-yosemite-r7-357 problem tracking

Status: RESOLVED WONTFIX

(Infrastructure & Operations :: RelOps: Hardware, defect)

Comment 2

[Zsolt Fay [:zfay]]

Worker was missing from TC. Was able to ssh into it and kick off the re-imaging process, however it still hasn't returned to TC yet. Checked puppet mail for a puppet report about it got found this: 

Sat Jul 28 05:45:56 -0700 2018 Puppet (err): Could not find command 'generic-worker'
Sat Jul 28 05:45:56 -0700 2018 /Stage[main]/Generic_worker/Exec[create gpg key]/returns (err): change from notrun to 0 failed: Could not find command 'generic-worker'
Sat Jul 28 05:48:21 -0700 2018 /Stage[main]/Mig::Agent::Daemon/Exec[restart mig] (err): Failed to call refresh: /bin/kill -s 2 $(/usr/local/bin/mig-agent -q=pid); /usr/local/bin/mig-agent returned 1 instead of one of [0]
Sat Jul 28 05:48:21 -0700 2018 /Stage[main]/Mig::Agent::Daemon/Exec[restart mig] (err): /bin/kill -s 2 $(/usr/local/bin/mig-agent -q=pid); /usr/local/bin/mig-agent returned 1 instead of one of [0]

Comment 3

[:dhouse]

(In reply to Zsolt Fay [:zsoltfay] from comment #2)
> Worker was missing from TC. Was able to ssh into it and kick off the
> re-imaging process, however it still hasn't returned to TC yet. Checked
> puppet mail for a puppet report about it got found this: 
> 
> Sat Jul 28 05:45:56 -0700 2018 Puppet (err): Could not find command
> 'generic-worker'
> Sat Jul 28 05:45:56 -0700 2018 /Stage[main]/Generic_worker/Exec[create gpg
> key]/returns (err): change from notrun to 0 failed: Could not find command
> 'generic-worker'
> Sat Jul 28 05:48:21 -0700 2018 /Stage[main]/Mig::Agent::Daemon/Exec[restart
> mig] (err): Failed to call refresh: /bin/kill -s 2
> $(/usr/local/bin/mig-agent -q=pid); /usr/local/bin/mig-agent returned 1
> instead of one of [0]
> Sat Jul 28 05:48:21 -0700 2018 /Stage[main]/Mig::Agent::Daemon/Exec[restart
> mig] (err): /bin/kill -s 2 $(/usr/local/bin/mig-agent -q=pid);
> /usr/local/bin/mig-agent returned 1 instead of one of [0]

This error is ignore-able. It is a dependency issue in puppet on the first run and is resolved on the first reboot.

Comment 4

[:dhouse]

No problems found with the worker. The reimage+puppet took 
The taskcluster worker explorer yesterday failed to display it (a missing task id): https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc1/t-yosemite-r7-357
But it is fine now.

https://papertrailapp.com/systems/2114253042/events

Comment 5

[Roland Mutter Michael (:rmutter)]

Seems like this worker is offline. Last logs from papertrail: 

Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class AMDRadeonX4000_AMDAccelDevice.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class AMDRadeonX4000_AMDAccelSharedUserClient.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class AMDSIVideoContext.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class Gen6DVDContext.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class IGAccelDevice.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class IGAccelSharedUserClient.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class IGAccelVideoContextMain.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class IGAccelVideoContextMedia.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class IGAccelVideoContextVEBox.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class IOBluetoothDeviceUserClient.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class IOHIDParamUserClient.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the IOKit user-client class IOSurfaceRootUserClient.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the mach service named com.apple.AirPlayXPCHelper.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named AirPlay.driver requires extending the sandbox for the mach service named com.apple.wirelessproxd.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com coreaudiod: 2018-11-28 06:16:22.829336 AM [AirPlay] BTLE discovery removing all devices
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com coreaudiod: 2018-11-28 06:16:22.831014 AM [AirPlay] Resetting AWDL traffic registration.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com coreaudiod: 2018-11-28 06:16:22.831307 AM [AirPlay] Deregister AirPlay traffic for AWDL at MAC 00:00:00:00:00:00 with target infra non critical PeerIndication=0 err=82
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named BluetoothAudioPlugIn.driver requires extending the sandbox for the IOKit user-client class IOBluetoothDeviceUserClient.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named BluetoothAudioPlugIn.driver requires extending the sandbox for the mach service named com.apple.blued.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com com.apple.audio.DriverHelper: The plug-in named BluetoothAudioPlugIn.driver requires extending the sandbox for the mach service named com.apple.bluetoothaudiod.
Nov 28 06:16:22 t-yosemite-r7-357.test.releng.mdc1.mozilla.com coreaudiod: Logging disabled

Comment 6

[:dhouse]

re-racked after shipping to SF/applestore(andrew)/ and back. Reimaged and back in service
looks good so far today

Comment 7

[Roland Mutter Michael (:rmutter)]

Reopening this as it seems every tasks run on this machine ends as exception. I'm going to quarantine this machines until investigations are done.

Comment 8

[Adrian Pop]

reimaged machine. will check later if the issue still persists.

Comment 9

[Adrian Pop]

Attachment: multiple_jobs.png

Comment 10

[Adrian Pop]

I will quarantine the machine, as it started more jobs in the same time.

Dave, can you please take a look ?

Comment 11

[Adrian Pop]

I have rebooted the machine and let it run a job, but still it was an exception