1134958 - (get_unallocated_slaves) exceptions.AttributeError: 'NoneType' object has no attribute 'slavename'

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Chris Cooper [:coop] (he/him)]

Build/try masters have been hitting this error intermittently:

2015-02-19 18:36:05-0800 [-] Unhandled Error
        Traceback (most recent call last):
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/process/builder.py", line 517, in _claim_buildreqs
            sb = self._choose_slave(available_slaves)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/process/builder.py", line 548, in _choose_slave
            return self.nextSlave(self, available_slaves)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/misc.py", line 267, in _nextSlave
            return func(builder, available_slaves)
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/misc.py", line 458, in _nextSlave
            J = JacuzziAllocator()
        --- <exception caught here> ---
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/misc.py", line 407, in get_slaves
            # Filter the list of available slaves by the set the service
          File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/misc.py", line 352, in get_unallocated_slaves
            
        exceptions.AttributeError: 'NoneType' object has no attribute 'slavename'
        
2015-02-19 18:36:05-0800 [-] JacuzziAllocator 466645968: unhandled exception
2015-02-19 18:36:05-0800 [-] JacuzziAllocator 466645968: try 1/3; sleeping 10 and trying again

Buildduty first noticed this error on Feb 17. Unsure if this is related to the current issues with pending jobs or not.

I'll attach a full log shortly in case the extra context (and slave list) is helpful.

Comment 1

[Chris Cooper [:coop] (he/him)]

I grabbed a full rotated log off of bm77 and uploaded it to people:

http://people.mozilla.org/~coop/bug1134958/twistd.log.bz2

Comment 2

[Chris AtLee [:catlee]]

we should filter out partly disconnected slaves then.

something like

available_slaves = [s for s in available_slaves if s.slave is not None]

in each of get_unallocated_slaves and get_slaves