946238 - All trees closed due to frequent job disconnects

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

We're seeing frequent disconnects in big spurts. All trees closed.

https://tbpl.mozilla.org/php/getParsedLog.php?id=31446488&tree=Mozilla-B2g26-v1.2

exceptions.TypeError: '_sre.SRE_Pattern' object is not iterable

Comment 1

[Armen [:armenzg]]

I wonder if "I disabled the SPI protection for those tunnels and we're monitoring them. Everything now should stay stable." is at fault (Yesterday at 14:43:52 PST):
https://bugzilla.mozilla.org/show_bug.cgi?id=940356#c32

Comment 2

[Justin Wood (:Callek)]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #1)
> I wonder if "I disabled the SPI protection for those tunnels and we're
> monitoring them. Everything now should stay stable." is at fault (Yesterday
> at 14:43:52 PST):
> https://bugzilla.mozilla.org/show_bug.cgi?id=940356#c32

Unlikely, though there was a BGP flap just barely ahead of the reconfig we did today...

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #0)
> We're seeing frequent disconnects in big spurts. All trees closed.
> 
> https://tbpl.mozilla.org/php/getParsedLog.php?id=31446488&tree=Mozilla-B2g26-
> v1.2
> 
> exceptions.TypeError: '_sre.SRE_Pattern' object is not iterable

This was due to http://hg.mozilla.org/build/buildbotcustom/rev/2dc7842299bb which was backed out in http://hg.mozilla.org/build/buildbotcustom/rev/5c791695e95f -- a reconfig to fix is in progress.

Comment 3

[Armen [:armenzg]]

This has been fixed IIRC after the backout.