Closed Bug 1475132 (t-yosemite-r7-336) Opened 8 years ago Closed 7 years ago

[MDC1] t-yosemite-r7-336 problem tracking

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rmutter, Unassigned)

References

Details

Stdio forwarding request failed: Session open refused by peer > ssh_exchange_identification: Connection closed by remote host
ssh works for me on this host puppet last ran today, and the ssh auth_keys include the CIDuty keys: ``` [root@t-yosemite-r7-336.test.releng.mdc1.mozilla.com ~]# grep mutter .ssh/authorized_keys ssh-rsa AAAA[...]== rolandmutter_SV [root@t-yosemite-r7-336.test.releng.mdc1.mozilla.com ~]# ls -la /var/lib/puppet/state/last_run_summary.yaml -rw-r--r-- 1 root puppet 722 Jul 23 08:12 /var/lib/puppet/state/last_run_summary.yaml ``` Please reopen this bug if CIDuty is not able to ssh into this host now.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
When I checked this machine, I found that it had hit a timeout on its last job and so it was left with the worker process not running. It needed rebooted (rebooted by roller). ``` [root@t-yosemite-r7-336.test.releng.mdc1.mozilla.com ~]# uptime 12:23 up 3:19, 2 users, load averages: 2.44 2.16 2.02 [root@t-yosemite-r7-336.test.releng.mdc1.mozilla.com ~]# ls -ltr /Users/cltbld/tasks/task*/logs total 2312 -rw-r--r-- 1 cltbld staff 0 Jul 23 09:04 log_fatal.log -rw-r--r-- 1 cltbld staff 0 Jul 23 09:04 log_critical.log -rw-r--r-- 1 cltbld staff 10957 Jul 23 09:04 localconfig.json -rw-r--r-- 1 cltbld staff 477 Jul 23 09:33 log_error.log -rw-r--r-- 1 cltbld staff 574 Jul 23 09:33 log_warning.log -rw-r--r-- 1 cltbld staff 529206 Jul 23 09:33 log_raw.log -rw-r--r-- 1 cltbld staff 627566 Jul 23 09:33 log_info.log [root@t-yosemite-r7-336.test.releng.mdc1.mozilla.com ~]# cat /Users/cltbld/tasks/task*/logs/log_error.log 09:16:52 ERROR - TEST-UNEXPECTED-TIMEOUT | automation.py | application timed out after 370 seconds with no output 09:16:52 ERROR - Force-terminating active process(es). 09:33:32 ERROR - timed out after 1000 seconds of no output 09:33:32 ERROR - Return code: -15 09:33:32 ERROR - No suite end message was emitted by this harness. 09:33:32 ERROR - # TBPL FAILURE # 09:33:32 ERROR - The mochitest suite: browser-chrome-chunked ran with return status: FAILURE ```
Alias: t-yosemite-r7-336
After reboot, this machine has been running tasks: https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc1/t-yosemite-r7-336
Status: RESOLVED → VERIFIED
Machine has issues again. SSH unresponsive and does not appear in taskcluster.
Sorry, the machine was available in taskcluster (was looking into the wrong MDC), rebooted and the SSH became available, started the re-image process that came with a SUCCESS termination message a few minutes after. Waiting for the machine to take jobs.
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Depends on: 1524793

the machine seems to be up and running and taking jobs.
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010/workers/mdc1/t-yosemite-r7-336
We will close the bug for now. If the problem will persist in the future, we will re-open this bug.

Status: REOPENED → RESOLVED
Closed: 7 years ago → 7 years ago
Resolution: --- → FIXED

Looks like the worker is offline again. It doesn't respond to ping. Worth mentioning that last job ended as exception 6 hours ago.
Papertrail piece of log:

Feb 09 02:56:09 t-yosemite-r7-336.test.releng.mdc1.mozilla.com CoreServicesUIAgent: unexpected message <OS_xpc_error: <error: 0x7fff784eac60> { count = 1, contents =#012#011#011"XPCErrorDescription" => <string: 0x7fff784eaf70> { length = 18, contents = "Connection invalid" }#012#011}>
Feb 09 02:56:09 t-yosemite-r7-336 kernel: firefox (map: 0xffffff802658a5a0) triggered DYLD shared region unnest for map: 0xffffff802658a5a0, region 0x7fff92600000->0x7fff92800000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
Feb 09 02:56:09 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x107c65a60] loaded
Feb 09 02:56:14 t-yosemite-r7-336.test.releng.mdc1.mozilla.com CoreServicesUIAgent: unexpected message <OS_xpc_error: <error: 0x7fff784eac60> { count = 1, contents =#012#011#011"XPCErrorDescription" => <string: 0x7fff784eaf70> { length = 18, contents = "Connection invalid" }#012#011}>
Feb 09 02:56:15 t-yosemite-r7-336 kernel: firefox (map: 0xffffff8025aed4b0) triggered DYLD shared region unnest for map: 0xffffff8025aed4b0, region 0x7fff92600000->0x7fff92800000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
Feb 09 02:56:15 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x10f665a60] loaded
Feb 09 02:56:25 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x10f665a60] unloaded
Feb 09 02:56:34 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: Persistent UI failed to open file file:///Users/cltbld/Library/Saved%20Application%20State/org.mozilla.nightly.savedState/window_1.data: No such file or directory (2)
Feb 09 02:56:39 t-yosemite-r7-336.test.releng.mdc1.mozilla.com CoreServicesUIAgent: unexpected message <OS_xpc_error: <error: 0x7fff784eac60> { count = 1, contents =#012#011#011"XPCErrorDescription" => <string: 0x7fff784eaf70> { length = 18, contents = "Connection invalid" }#012#011}>
Feb 09 02:56:40 t-yosemite-r7-336 kernel: firefox (map: 0xffffff8022989b40) triggered DYLD shared region unnest for map: 0xffffff8022989b40, region 0x7fff92600000->0x7fff92800000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
Feb 09 02:56:40 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x107d65a60] loaded
Feb 09 02:56:50 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: LSExceptions [0x107d65a60] unloaded
Feb 09 02:57:03 t-yosemite-r7-336.test.releng.mdc1.mozilla.com firefox: Persistent UI failed to open file file:///Users/cltbld/Library/Saved%20Application%20State/org.mozilla.nightly.savedState/window_1.data: No such file or directory (2)
Feb 09 02:57:25 t-yosemite-r7-336 kernel: process firefox[1035] caught causing excessive wakeups. Observed wakeups rate (per sec): 993; Maximum permitted wakeups rate (per sec): 150; Observation period: 300 seconds; Task lifetime number of wakeups: 45006
Feb 09 02:57:25 t-yosemite-r7-336 com.apple.xpc.launchd: (com.apple.ReportCrash[1040]): Endpoint has been activated through legacy launch(3) APIs. Please switch to XPC or bootstrap_check_in(): com.apple.ReportCrash
Feb 09 02:57:25 t-yosemite-r7-336.test.releng.mdc1.mozilla.com ReportCrash: Invoking spindump for pid=1035 wakeups_rate=993 duration=46 because of excessive wakeups
Feb 09 02:57:25 t-yosemite-r7-336.test.releng.mdc1.mozilla.com spindump: Error grabbing microstackshots: 53
Feb 09 02:57:25 t-yosemite-r7-336.test.releng.mdc1.mozilla.com spindump: No microstackshots found
Feb 09 02:58:57 t-yosemite-r7-336.test.releng.mdc1.mozilla.com plugin-container: LSExceptions [0x116849a60] loaded
Feb 09 02:58:58 t-yosemite-r7-336 com.apple.xpc.launchd: (com.apple.xpc.launchd.user.29.100005.Aqua): Caller not allowed to perform action: plugin-containe.1041, action = pid-local registration, code = 1: Operation not permitted, uid = 29, euid = 29, gid = 20, egid = 20, asid = 100005
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com appleeventsd: <rdar://problem/11489077> A sandboxed application with pid 1041, "Firefox NightlyCP Plugin Content (Test Plug-in)" checked in with appleeventsd, but its code signature could not be validated ( either because it was corrupt, or could not be read by appleeventsd ) and so it cannot receive AppleEvents targeted by name, bundle id, or signature. Error=ERROR: #100001 { "NSDescription"="SecCodeCopyGuestWithAttributes() returned 100001, -." } (handleMessage()/appleEventsD.cp #2100) com.apple.root.default-qos
Feb 09 02:58:58 t-yosemite-r7-336 com.apple.xpc.launchd: (com.apple.xpc.launchd.user.29.100005.Aqua): Caller not allowed to perform action: plugin-containe.1041, action = pid-local registration, code = 1: Operation not permitted, uid = 29, euid = 29, gid = 20, egid = 20, asid = 100005
Feb 09 02:58:58 t-yosemite-r7-336 com.apple.xpc.launchd: (com.apple.xpc.launchd.user.29.100005.Aqua): Caller not allowed to perform action: plugin-containe.1041, action = pid-local registration, code = 1: Operation not permitted, uid = 29, euid = 29, gid = 20, egid = 20, asid = 100005
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com plugin-container: Failed to allocate communication port for com.apple.CFPasteboardClient; this is likely due to sandbox restrictions
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com sandboxd: ([50]): appleeventsd(50) deny file-read-data /Users/cltbld/tasks/task_1549708724/build/application/Firefox Nightly.app/Contents/MacOS/plugin-container.app
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com sandboxd: ([50]): appleeventsd(50) deny file-read-metadata /Users/cltbld/tasks/task_1549708724/build/application/Firefox Nightly.app/Contents/MacOS/plugin-container.app/Contents
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com sandboxd: ([50]): appleeventsd(50) deny file-read-metadata /Users/cltbld/tasks/task_1549708724/build/application/Firefox Nightly.app/Contents/MacOS/plugin-container.app
Feb 09 02:58:58 t-yosemite-r7-336.test.releng.mdc1.mozilla.com sandboxd: ([50]): appleeventsd(50) deny file-read-data /Users/cltbld/tasks/task_1549708724/build/application/Firefox Nightly.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container
Feb 09 02:59:00 t-yosemite-r7-336.test.releng.mdc1.mozilla.com diagnosticd: Peer Validation Failed: SecCodeCheckValidity status=-67063
Feb 09 02:59:08 t-yosemite-r7-336.test.releng.mdc1.mozilla.com plugin-container: LSExceptions [0x116849a60] unloaded
Feb 09 08:13:30 rejh1.srv.releng.mdc1.mozilla.com sshd: error: connect_to t-yosemite-r7-336.test.releng.mdc1.mozilla.com port 22: failed.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

is active/taking jobs
looks like it recovered at the last problem time

Status: REOPENED → RESOLVED
Closed: 7 years ago → 7 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.