1256375 - Rebooting through slaveapi fails with "Expecting value: line 2 column 1 (char 1)"

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

e.g. https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=t-xp32-ix&name=t-xp32-ix-095 though it affects all flavors of test and build slaves.

We've already lost 13% of the WinXP slaves, so if there aren't any events like a temporary slavealloc outage which takes out a huge swath of slaves, figure three or four days before it becomes a blocker.

Comment 1

[Kim Moir [:kmoir] ET]

I reverted 


http://hg.mozilla.org/build/slave_health/rev/49cab2f5cf4c

since it looks like the change to slave health might be the problem

Comment 2

[Kim Moir [:kmoir] ET]

that didn't make a difference so relanding

Comment 3

[Kim Moir [:kmoir] ET]

Attachment: bug1256375.patch

remove references to mozpool and devices.json etc that are preventing reboots

Comment 4

[Alin Selagea [:aselagea]]

I managed to connect to almost all of the 13 slaves mentioned above - some were running jobs, so I only rebooted the idle ones.

Another problem here is that their status from Slave Health dashboard does not change because /slave_health/json/test-t-xp32-ix.json file does not get updated (that is valid for other pools as well). 

If we open the json file, it can be noticed that it says: generated: "2016-03-14T15:00:09.008130Z"..which does not look right.

Comment 5

[Kim Moir [:kmoir] ET]

Comment on attachment 8730454 [details] [diff] [review]
bug1256375.patch

Now following instructions here 
https://wiki.mozilla.org/ReleaseEngineering/Applications/SlaveAPI
to deploy to dev, then prod

Comment 6

[Kim Moir [:kmoir] ET]

Comment on attachment 8730454 [details] [diff] [review]
bug1256375.patch

actually I'm getting 

remote: abort: could not lock repository /repo/hg/mozilla/build/slaveapi: Permission denied
abort: unexpected response: empty string

and can't land it there

Comment 7

[Phil Ringnalda (:philor)]

Win7 slaves dead so far (with 3232 pending jobs): t-w732-ix-173 t-w732-ix-047 t-w732-ix-026 t-w732-ix-126 t-w732-ix-028 t-w732-ix-041 t-w732-ix-258 t-w732-ix-281 t-w732-ix-031 t-w732-ix-162 t-w732-ix-150

Comment 8

[Kim Moir [:kmoir] ET]

I added back buildfarm/mobile/devices.json until we can get my commit rights sorted out in bug  	1257283.

Comment 9

[Kim Moir [:kmoir] ET]

http://hg.mozilla.org/build/tools/rev/1f5aca5158d9

Comment 10

[Kim Moir [:kmoir] ET]

Comment on attachment 8730454 [details] [diff] [review]
bug1256375.patch

really checked in this time but in git

Comment 11

[Kim Moir [:kmoir] ET]

Hit this error when I restarted production slaveapi so I have reverted the version of slaveapi puppet/production to 1.5.0

2016-03-17 11:26:08,442 - INFO - t-w732-ix-026 - Getting inventory info
2016-03-17 11:26:08,629 - ERROR - t-w732-ix-026 - Something went wrong while processing!
2016-03-17 11:26:08,630 - ERROR - t-w732-ix-026 - Traceback (most recent call last):
2016-03-17 11:26:08,630 - ERROR - t-w732-ix-026 - 
2016-03-17 11:26:08,630 - ERROR - t-w732-ix-026 -   File "/builds/slaveapi/prod/lib/python2.7/site-packages/slaveapi/processor.py", line 64, in _worker
2016-03-17 11:26:08,630 - ERROR - t-w732-ix-026 -     res, msg = action(slave, *args, **kwargs)
2016-03-17 11:26:08,630 - ERROR - t-w732-ix-026 - 
2016-03-17 11:26:08,630 - ERROR - t-w732-ix-026 -   File "/builds/slaveapi/prod/lib/python2.7/site-packages/slaveapi/actions/reboot.py", line 32, in reboot
2016-03-17 11:26:08,631 - ERROR - t-w732-ix-026 -     slave.load_inventory_info()
2016-03-17 11:26:08,631 - ERROR - t-w732-ix-026 - 
2016-03-17 11:26:08,631 - ERROR - t-w732-ix-026 -   File "/builds/slaveapi/prod/lib/python2.7/site-packages/slaveapi/slave.py", line 60, in load_inventory_info
2016-03-17 11:26:08,631 - ERROR - t-w732-ix-026 -     info = Machine.load_inventory_info(self)
2016-03-17 11:26:08,631 - ERROR - t-w732-ix-026 - 
2016-03-17 11:26:08,631 - ERROR - t-w732-ix-026 -   File "/builds/slaveapi/prod/lib/python2.7/site-packages/slaveapi/machines/base.py", line 44, in load_inventory_info
2016-03-17 11:26:08,631 - ERROR - t-w732-ix-026 -     if info["pdu_fqdn"]:
2016-03-17 11:26:08,632 - ERROR - t-w732-ix-026 - 
2016-03-17 11:26:08,632 - ERROR - t-w732-ix-026 - TypeError: 'NoneType' object has no attribute '__getitem__'
2016-03-17 11:26:08,632 - ERROR - t-w732-ix-026 - 
2016-03-17 11:26:08,632 - ERROR - t-w732-ix-026 -

Comment 12

[Kim Moir [:kmoir] ET]

Attachment: bug1256375fixbustage.patch

found the problem, my previous patch removed the removed statement from get_system by accident

Comment 13

[Kim Moir [:kmoir] ET]

Attachment: bug1256375fixbustage.patch

I built a slaveapi 1.6.1 with this patch + bumping the slaveapi version in puppet.  I deployed it to the production slaveapi instance and it all seems to be working now - I can reboot machines etc. Now all I have to do is land these patches on g.m.org once it up again.

Comment 14

[Justin Wood (:Callek)]

...err reopened until kim can land the patches

Comment 15

[Kim Moir [:kmoir] ET]

Landed