1524793 - [MDC1] t-yosemite-r7-336 is unreachable

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Hardware, task)

Description

[Automatic Bug Generator]

Reboot t-yosemite-r7-336.test.releng.mdc1.mozilla.com. 10.49.56.120
Requested by mozilla-auth0/ad|Mozilla-LDAP|zfay
Relops controller action failed:
2019-02-02T20:06:18.878170 ssh_reboot -l roller -i ssh.key TimeoutExpired
2019-02-02T20:08:19.862142 snmp_reboot pdu1.rit40.ops.releng.mdc1.mozilla.com ba5 --delay 60 Exception
2019-02-02T20:08:19.866299 ipmi ipmi_reset KeyError
2019-02-02T20:08:19.868317 ipmi ipmi_cycle KeyError
2019-02-02T20:08:19.870105 ipmi ipmi_on KeyError

Comment 1

[Adrian Pop]

the machine seems to be up and running and taking jobs.
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc1/t-yosemite-r7-336
We will close the bug for now. If the problem will persist in the future, we will re-open this bug.

Comment 2

[Roland Mutter Michael (:rmutter)]

Looks like the worker is offline again. It doesn't respond to ping. Worth mentioning that last job ended as exception 6 hours ago.
Papertrail piece of log:

Feb 09 02:56:09 t-yosemite-r7-336.test.releng.mdc1.mozilla.com CoreServicesUIAgent: unexpected message <OS_xpc_error: <error: 0x7fff784eac60> { count = 1, contents =#012#011#011"XPCErrorDescription" => <string: 0x7fff784eaf70> { length = 18, contents = "Connection invalid" }#012#011}>
Feb 09 02:56:09 t-yosemite-r7-336 kernel: firefox (map: 0xffffff802658a5a0) triggered DYLD shared region unnest for map: 0xffffff802658a5a0, region 0x7fff92600000->0x7fff92800000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
Feb 09 02:56:09 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x107c65a60] loaded
Feb 09 02:56:14 t-yosemite-r7-336.test.releng.mdc1.mozilla.com CoreServicesUIAgent: unexpected message <OS_xpc_error: <error: 0x7fff784eac60> { count = 1, contents =#012#011#011"XPCErrorDescription" => <string: 0x7fff784eaf70> { length = 18, contents = "Connection invalid" }#012#011}>
Feb 09 02:56:15 t-yosemite-r7-336 kernel: firefox (map: 0xffffff8025aed4b0) triggered DYLD shared region unnest for map: 0xffffff8025aed4b0, region 0x7fff92600000->0x7fff92800000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
Feb 09 02:56:15 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x10f665a60] loaded
Feb 09 02:56:25 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x10f665a60] unloaded
Feb 09 02:56:34 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: Persistent UI failed to open file file:///Users/cltbld/Library/Saved%20Application%20State/org.mozilla.nightly.savedState/window_1.data: No such file or directory (2)
Feb 09 02:56:39 t-yosemite-r7-336.test.releng.mdc1.mozilla.com CoreServicesUIAgent: unexpected message <OS_xpc_error: <error: 0x7fff784eac60> { count = 1, contents =#012#011#011"XPCErrorDescription" => <string: 0x7fff784eaf70> { length = 18, contents = "Connection invalid" }#012#011}>
Feb 09 02:56:40 t-yosemite-r7-336 kernel: firefox (map: 0xffffff8022989b40) triggered DYLD shared region unnest for map: 0xffffff8022989b40, region 0x7fff92600000->0x7fff92800000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
Feb 09 02:56:40 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x107d65a60] loaded
Feb 09 02:56:50 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x107d65a60] unloaded
Feb 09 02:57:03 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: Persistent UI failed to open file file:///Users/cltbld/Library/Saved%20Application%20State/org.mozilla.nightly.savedState/window_1.data: No such file or directory (2)
Feb 09 02:57:25 t-yosemite-r7-336 kernel: process firefox[1035] caught causing excessive wakeups. Observed wakeups rate (per sec): 993; Maximum permitted wakeups rate (per sec): 150; Observation period: 300 seconds; Task lifetime number of wakeups: 45006
Feb 09 02:57:25 t-yosemite-r7-336 com.apple.xpc.launchd: (com.apple.ReportCrash[1040]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.ReportCrash
Feb 09 02:57:25 t-yosemite-r7-336.test.releng.mdc1.mozilla.com ReportCrash: Invoking spindump for pid=1035 wakeups_rate=993 duration=46 because of excessive wakeups
Feb 09 02:57:25 t-yosemite-r7-336.test.releng.mdc1.mozilla.com spindump: Error grabbing microstackshots: 53
Feb 09 02:57:25 t-yosemite-r7-336.test.releng.mdc1.mozilla.com spindump: No microstackshots found
Feb 09 02:58:57 t-yosemite-r7-336.test.releng.mdc1.mozilla.com plugin-container: LSExceptions [0x116849a60] loaded
Feb 09 02:58:58 t-yosemite-r7-336 com.apple.xpc.launchd: (com.apple.xpc.launchd.user.29.100005.Aqua): Caller not allowed to perform action: plugin-containe.1041, action = pid-local registration, code = 1: Operation not permitted, uid = 29, euid = 29, gid = 20, egid = 20, asid = 100005
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com appleeventsd: <rdar://problem/11489077> A sandboxed application with pid 1041, "Firefox NightlyCP Plugin Content (Test Plug-in)" checked in with appleeventsd, but its code signature could not be validated ( either because it was corrupt, or could not be read by appleeventsd ) and so it cannot receive AppleEvents targeted by name, bundle id, or signature. Error=ERROR: #100001 { "NSDescription"="SecCodeCopyGuestWithAttributes() returned 100001, -." } (handleMessage()/appleEventsD.cp #2100) com.apple.root.default-qos
Feb 09 02:58:58 t-yosemite-r7-336 com.apple.xpc.launchd: (com.apple.xpc.launchd.user.29.100005.Aqua): Caller not allowed to perform action: plugin-containe.1041, action = pid-local registration, code = 1: Operation not permitted, uid = 29, euid = 29, gid = 20, egid = 20, asid = 100005
Feb 09 02:58:58 t-yosemite-r7-336 com.apple.xpc.launchd: (com.apple.xpc.launchd.user.29.100005.Aqua): Caller not allowed to perform action: plugin-containe.1041, action = pid-local registration, code = 1: Operation not permitted, uid = 29, euid = 29, gid = 20, egid = 20, asid = 100005
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com plugin-container: Failed to allocate communication port for com.apple.CFPasteboardClient; this is likely due to sandbox restrictions
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com sandboxd: ([50]): appleeventsd(50) deny file-read-data /Users/cltbld/tasks/task_1549708724/build/application/Firefox Nightly.app/Contents/MacOS/plugin-container.app
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com sandboxd: ([50]): appleeventsd(50) deny file-read-metadata /Users/cltbld/tasks/task_1549708724/build/application/Firefox Nightly.app/Contents/MacOS/plugin-container.app/Contents
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com sandboxd: ([50]): appleeventsd(50) deny file-read-metadata /Users/cltbld/tasks/task_1549708724/build/application/Firefox Nightly.app/Contents/MacOS/plugin-container.app
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com sandboxd: ([50]): appleeventsd(50) deny file-read-data /Users/cltbld/tasks/task_1549708724/build/application/Firefox Nightly.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container
Feb 09 02:59:00 t-yosemite-r7-336.test.releng.mdc1.mozilla.com diagnosticd: Peer Validation Failed: SecCodeCheckValidity status=-67063
Feb 09 02:59:08 t-yosemite-r7-336.test.releng.mdc1.mozilla.com plugin-container: LSExceptions [0x116849a60] unloaded
Feb 09 08:13:30 rejh1.srv.releng.mdc1.mozilla.com sshd: error: connect_to t-yosemite-r7-336.test.releng.mdc1.mozilla.com port 22: failed.

Comment 3

[:dhouse]

it has been taking jobs. It must have recovered at the last attempted reboot.