528212 - sendchange sometimes hangs

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

There have been a few cases over the past few days of sendchange steps never completing.  There's nothing in the logs to indicate why.  In at least one case, the remote buildbot master was blocked in FtpPoller, and restarting it allowed the local sendchange to complete.

We should add our own timeout handling to sendchange instead of relying on the twisted stack to time out.

Comment 1

[[e-mail] (:bhearsum)]

I'm pretty sure you were looking into this Catlee. Throw it back if you weren't.

Comment 2

[Chris AtLee [:catlee]]

Attachment: fun with twisted!

This adds a 60 second timeout to the sendchange operation, as well as making it interruptable.

Comment 3

[Chris AtLee [:catlee]]

Attachment: fun with object oriented twisted!

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 414336 [details] [diff] [review]
fun with object oriented twisted!

http://hg.mozilla.org/build/buildbotcustom/rev/18df53580c86

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

pm01 got his change when it restarted at 2000 PST. pm02 is pending, I'll try to find a quiet time later tonight.

Comment 6

[Chris AtLee [:catlee]]

Well, sendchanges don't hang any more...

Comment 7

[Chris AtLee [:catlee]]

Attachment: Fix error handling

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 416921 [details] [diff] [review]
Fix error handling

>diff --git a/steps/misc.py b/steps/misc.py
>--- a/steps/misc.py
>+++ b/steps/misc.py
>@@ -302,17 +302,18 @@ class SendChangeStep(BuildStep):
>     master: %s
>     branch: %s
>     revision: %s
>     comments: %s
>     user: %s
>     files: %s""" % (self.master, self.branch, self.revision, self.comments, self.user, self.files))
>             return self.sendChange()
>         except KeyError:
>-            return self.finished(Failure())
>+            self.addCompleteLog("errors", str(res))

I don't think 'res' is defined here. Maybe you want 'except KeyError, res', too?

>+            return self.finished(FAILURE)
> 
>     def sendChange(self):
>         d = self.sender.send(self.branch, self.revision, self.comments, self.files, self.user)
>         if self.timeout:
>             d = errbackAfter(d, self.timeout)
>         d = InterruptableDeferred(d)
>         self._interrupt = d.interrupt
>         d.addCallback(self.sendChangeSuccess)
>@@ -340,25 +341,25 @@ class SendChangeStep(BuildStep):
>                     return self.sendChangeFailed(reason)
>                 self._interrupt = interrupt
>                 return d
> 
>             self.step_status.setText(['sendchange to', self.master, 'failed'])
>             if self.warnOnFailure:
>                 self.step_status.setText2(['sendchange'])
>             self.addCompleteLog("errors", str(res))
>-            return BuildStep.finished(self, FAILURE)
>+            return self.finished(FAILURE)
>         except:
>             log.msg("Error processing sendchange failure")
>             log.err()
>-            return BuildStep.finished(self, FAILURE)
>+            return self.finished(FAILURE)
> 
>     def sendChangeSuccess(self, results):
>         self.step_status.setText(['sendchange to', self.master, 'ok'])
>-        return BuildStep.finished(self, SUCCESS)
>+        return self.finished(SUCCESS)
> 
>     def interrupt(self, reason):
>         self.retries = 0
>         if self._interrupt:
>             self._interrupt("Cancelled sendchange")
>             self._interrupt = None
> 
> class DownloadFile(ShellCommand):

Comment 9

[Chris AtLee [:catlee]]

Attachment: Fix error handling

Comment 10

[Chris AtLee [:catlee]]

Comment on attachment 417499 [details] [diff] [review]
Fix error handling

changeset:   550:2e4ffa62ff34