947376 - AWS spot instances should always automatically retrigger when disconnected

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

We see purple disconnects somewhat regularly on the spot instance machines when disconnected. They should auto-restart without intervention.

https://tbpl.mozilla.org/php/getParsedLog.php?id=31577207&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=31576771&tree=Fx-Team

Comment 1

[Rail Aliiev [:rail]]

Maybe retry on 
remoteFailed: [Failure instance: Traceback: <class 'twisted.spread.pb.DeadReferenceError'>: Calling Stale Broker

Comment 2

[Rail Aliiev [:rail]]

Attachment: retry on this error

Comment 3

[Chris AtLee [:catlee]]

Comment on attachment 8347745 [details] [diff] [review]
retry on this error

Review of attachment 8347745 [details] [diff] [review]:
-----------------------------------------------------------------

That error happens further down...why not something that matches Failure instance: Traceback (failure with no frames): <class 'twisted.spread.pb.PBConnectionLost'>?

Comment 4

[Rail Aliiev [:rail]]

Attachment: PBConnectionLost

(In reply to Chris AtLee [:catlee] from comment #3)
> Comment on attachment 8347745 [details] [diff] [review]
> retry on this error
> 
> Review of attachment 8347745 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> That error happens further down...why not something that matches Failure
> instance: Traceback (failure with no frames): <class
> 'twisted.spread.pb.PBConnectionLost'>?

Sure, good idea.

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8349446 [details] [diff] [review]
PBConnectionLost

https://hg.mozilla.org/build/buildbotcustom/rev/42bf30ebc1d0

Comment 6

[Chris Cooper [:coop] (he/him)]

In production.