644991 - disable masters on buildbot-master1 due to slow drive

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

Depending on how jittery the drive is feeling, I get hdparm reuslts ranging from 70-95MB/s on buildbot-master1.  The machine has been up for 97 days, and has plenty of these messages in dmesg:
ata1: spurious interrupt (irq_stat 0x8 active_tag -84148995 sactive 0x8)
ata1: spurious interrupt (irq_stat 0x8 active_tag -84148995 sactive 0x2)

will reboot and see what the machine feels like afterwards

Comment 1

[Chris AtLee [:catlee]]

Running a SMART self test causes a whole ton of those spurious interrupt messages

Comment 2

[Zandr Milewski [:zandr]]

Before we can take this out of service, we need to move the slaves off of it. Moving to RelEng, throw it back when the slaves have been moved.

Comment 3

[Armen [:armenzg]]

The slaves are off the master.

I wasn't aware that we are lacking so many masters.

I have bumped the version to critical as it affects our current capacity.

Comment 4

[Armen [:armenzg]]

Reducing as there is not much more that you can do besides getting a healthy master.

Comment 5

[Armen [:armenzg]]

Sorry for the noise. There are still some slaves connected.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage: 

1) Before we can hand this over to IT, we have to power off all the master instances running on this box.

2) Per triage right now, the try and build master instances are still running. The test master instance is already off. 

3) Because of load on other buildbot masters, we cannot power off this master just yet, no matter how sick it is. Once the other new masters (04,06) are fully online, and sharing load, we can revisit.

Pushing this bug to catlee, who is working on setting up 04,06, and therefore will know when its safe to power off buildbot-master1.

Comment 7

[Zandr Milewski [:zandr]]

Any update on this? Would like to get this shut down so we can send it out for repair.

Comment 8

[Armen [:armenzg]]

zandr we are waiting on disabling masters from this host once we are done the work of setting more masters up on SJC (bug 656413).

Adding dependency.

Comment 9

[Chris AtLee [:catlee]]

This is ready to go back to IX.

Comment 10

[Chris AtLee [:catlee]]

Attachment: Remove buildbot-master1 from masters json

Comment 11

[Armen [:armenzg]]

In which bug is this host tracked to be sent back to IX?

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 536417 [details] [diff] [review]
Remove buildbot-master1 from masters json

Remove them from slavealloc, too?

Comment 13

[Chris AtLee [:catlee]]

(In reply to comment #12)
> Comment on attachment 536417 [details] [diff] [review] [review]
> Remove buildbot-master1 from masters json
> 
> Remove them from slavealloc, too?

Yeah, might as well. Is that doable easily?

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Via mysql, yes - I took care of it.