Closed Bug 947376 Opened 10 years ago Closed 10 years ago

AWS spot instances should always automatically retrigger when disconnected

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: RyanVM, Assigned: rail)

References

Details

Attachments

(1 file, 1 obsolete file)

We see purple disconnects somewhat regularly on the spot instance machines when disconnected. They should auto-restart without intervention.

https://tbpl.mozilla.org/php/getParsedLog.php?id=31577207&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=31576771&tree=Fx-Team
Maybe retry on 
remoteFailed: [Failure instance: Traceback: <class 'twisted.spread.pb.DeadReferenceError'>: Calling Stale Broker
Blocks: 935533
Attached patch retry on this error (obsolete) — Splinter Review
Assignee: nobody → rail
Attachment #8347745 - Flags: review?(catlee)
Comment on attachment 8347745 [details] [diff] [review]
retry on this error

Review of attachment 8347745 [details] [diff] [review]:
-----------------------------------------------------------------

That error happens further down...why not something that matches Failure instance: Traceback (failure with no frames): <class 'twisted.spread.pb.PBConnectionLost'>?
Attached patch PBConnectionLostSplinter Review
(In reply to Chris AtLee [:catlee] from comment #3)
> Comment on attachment 8347745 [details] [diff] [review]
> retry on this error
> 
> Review of attachment 8347745 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> That error happens further down...why not something that matches Failure
> instance: Traceback (failure with no frames): <class
> 'twisted.spread.pb.PBConnectionLost'>?

Sure, good idea.
Attachment #8349446 - Flags: review?(catlee)
Attachment #8347745 - Attachment is obsolete: true
Attachment #8347745 - Flags: review?(catlee)
Attachment #8349446 - Flags: review?(catlee) → review+
In production.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.