1260442 - (t-yosemite-r7-0022) t-yosemite-r7-0022 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Comment 1

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1260443)

Comment 2

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1262749)

Comment 3

[Vlad Ciobancai [:vladC] [:vciobancai]]

The slave was re-imaged by Dcops and we will monitor the slave to check if the issue occurs again

Comment 4

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1264353)

Comment 5

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1272766)

Comment 6

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1282298)

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

This machine is spending a lot of CPU time running coreaudiod (~80%) and launchd (~40%) and should probably be reimaged. Disabled in slavealloc.

Comment 8

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

The slave was re-imaged and re-enabled in slavealloc.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

It still has high CPU after the reimage, eg 

uptime:
15:42  up  1:03, 2 users, load averages: 1.92 1.93 1.95

From top:
Processes: 143 total, 4 running, 3 stuck, 136 sleeping, 464 threads                                                                                     15:42:50
Load Avg: 2.22, 2.00, 1.97  CPU usage: 11.88% user, 23.1% sys, 65.9% idle    SharedLibs: 948K resident, 0B data, 0B linkedit.
MemRegions: 10902 total, 356M resident, 62M private, 106M shared. PhysMem: 2125M used (1009M wired), 14G unused.
VM: 349G vsize, 1025M framework vsize, 0(0) swapins, 0(0) swapouts. Networks: packets: 325414/314M in, 185425/136M out.
Disks: 18921/684M read, 58445/1282M written.

PID   COMMAND      %CPU TIME     #TH  #WQ  #PORT MEM    PURG   CMPR PGRP PPID STATE    BOOSTS        %CPU_ME %CPU_OTHRS UID  FAULTS    COW   MSGSENT
213   coreaudiod   79.7 50:30.24 4/1  1/1  165-  18M    0B     0B   213  1    running  *0[1]         14.4828 0.00000    202  40738039+ 193   285121913+
1     launchd      43.3 27:14.73 5/1  4    2149  9048K  0B     0B   1    0    running   0[0]         0.00000 14.48262   0    6101      278   244447900+
1803  top          1.1  00:00.18 1/1  0    19    1960K  0B     0B   1803 1795 running  *0[1]         0.00000 0.00000    0    13915+    79    70763+
0     kernel_task  0.0  00:12.23 95/4 0    2     686M   0B     0B   0    0    running   0[0]         0.00000 0.00000    0    15417     0     164403912+
...

Hardware fault ? Disabled in slavealloc because it's probably distorting talos results.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Looks like this is a pool wide problem, filed as bug 1285483.  aobreja says he re-enabled this slave and it's working ok so lets close this bug out.

Comment 11

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1287168)

Comment 12

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back in production and taking jobs.

Comment 13

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1365405)

Comment 14

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online.

Comment 15

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1383181)

Comment 16

[Alin Selagea [:aselagea]]

Machine is online.

Comment 17

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online after re-image.