Closed Bug 1136765 Opened 9 years ago Closed 9 years ago

Intermittent "exceptions.AttributeError: 'NoneType' object has no attribute 'slave_status'" slave reboot exception at the end of runs

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: RyanVM, Unassigned)

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

[buildbotcustom] Bug 1136765 - manage exception when starting a disconnection.patch
1.75 KB, patch
bhearsum
: review+
massimo
: checked-in+
Details | Diff | Splinter Review 
We've been seeing this intermittently recently. They come at the end of the run and cause otherwise-successful runs to show up as infra exceptions on Treeherder, which adds a lot of noise.
Hi Ben,

try:
   graceful = self.build.slavebuilder.slave.slave_status.getGraceful()
except AttributeError:
graceful = False
ops, I sent last comment before it was completed!

This error seems to be caused by an uncaught exception here [1]
I was thinking about adding the following lines:

> 
> try:
>    graceful = self.build.slavebuilder.slave.slave_status.getGraceful()
> except AttributeError:
> graceful = False

Ben, does it make sense? Is there anything else we can do to mitigate this problem?


[1] https://hg.mozilla.org/build/buildbotcustom/file/982a73ec1aba/steps/misc.py#l486
Flags: needinfo?(bhearsum)
(In reply to Massimo Gervasini [:mgerva] from comment #175)
> ops, I sent last comment before it was completed!
> 
> This error seems to be caused by an uncaught exception here [1]
> I was thinking about adding the following lines:
> 
> > 
> > try:
> >    graceful = self.build.slavebuilder.slave.slave_status.getGraceful()
> > except AttributeError:
> > graceful = False
> 
> Ben, does it make sense? Is there anything else we can do to mitigate this
> problem?
> 
> 
> [1]
> https://hg.mozilla.org/build/buildbotcustom/file/982a73ec1aba/steps/misc.
> py#l486

I'm not entirely sure what you're suggesting. Setting graceful = False doesn't seem like it would do anything - that variable isn't used anywhere else already.

Maybe it would be better to check if self.build.slavebuilder.slave is None though, rather than catching an AttributeError after the fact. I _think_ it would make sense to return SKIPPED in that case as well - since the slave isn't connected there doesn't to be any benefit from trying to disconnect it.
Flags: needinfo?(bhearsum)
Thanks Ben,

changes with this patch:
* added check for self.build.slavebuilder.slave is None.  
* removed "else:"
Attachment #8593354 - Flags: review?(bhearsum)
Comment on attachment 8593354 [details] [diff] [review]
[buildbotcustom] Bug 1136765 - manage exception when starting a disconnection.patch

Review of attachment 8593354 [details] [diff] [review]:
-----------------------------------------------------------------

::: steps/misc.py
@@ +483,5 @@
>      def start(self):
>          # If a graceful shutdown was requested it doesn't make sense to reboot
>          # the slave - so let's not do anything!
> +        if self.build.slavebuilder.slave is None:
> +            # bug 1136765

Please add a more descriptive comment here rather than just a bug number. r=me with that fixed when you land.
Attachment #8593354 - Flags: review?(bhearsum) → review+
Comment on attachment 8593354 [details] [diff] [review]
[buildbotcustom] Bug 1136765 - manage exception when starting a disconnection.patch

Thanks Ben!

landed: https://hg.mozilla.org/build/buildbotcustom/rev/785722efec9b
Attachment #8593354 - Flags: checked-in+
Inactive; closing (see bug 1180138).
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: