918803 - buildbot-master42,43 have very few or no pandas attached

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

On Wednesday I noticed that buildbot-master42 had no pandas attached.  It should have 522-544, 258-307, 320-381.  I reimaged the pandas with the android image, they attached and ran some jobs.  The same problem is happening today.  The master isn't showing any buildslaves attached and many of the devices have error.flgs.  I verified the entries in the devices.json look good.  The same thing appears to be happening for buildbot-master43, it looks like only has a couple of pandas attached.

Comment 1

[Armen [:armenzg]]

kmoir, is this something you or pmoore could help investigate?
I don't think I will have time as buildduty to look into it as there are other bugs that I have to deal with.

Comment 2

[Kim Moir [:kmoir] ET]

Yes, I'm looking at it.  Bm43 looks good this morning, I'm investigating why bm42 continues to have problems.

Comment 3

[Kim Moir [:kmoir] ET]

Both masters were in the same state this morning, very few pandas attached.  

I looked at the logs on mozpool and it looked like most of the pandas ran selftest.py which passed but then on the foopies a connection could not be made thus they didn't connect to the master. 

http://mobile-imaging-010.p10.releng.scl1.mozilla.com/ui/log.html?device=panda-0546

They also didn't show the image as android in mozpool, only blank.  

I'm not sure what's happening here, Callek do you have any suggestions? If not I'll talk to the ateam.

I reimaged the devices to get them to connect, since our current capacity is low with that many pandas down.

Comment 4

[Kim Moir [:kmoir] ET]

I talked to Callek about this yesterday, he thinks that changing the scripts so that everything is rebooted via mozpool (bug 919533 and bug 889967) will resolve this problem.  In the interim, I reimaged all the pandas and they seem to be staying up for the last 24 hours.

Comment 5

[Kim Moir [:kmoir] ET]

This is not a problem any more.  Not sure if this resolved itself since the devices are all managed by mozpool. Will work on resolving bug 919533 and bug 889967 in any case.