454023 - Sentry should automatically decrease weights on mirrors that get dropped and re-added

Status: RESOLVED FIXED

(Webtools :: Bouncer, defect)

Description

[Dave Miller [:justdave]]

One thing we noticed during the 3.0.1 Major Update push was several of our mirrors (even a couple of the really big ones - which caused some issues) would get overwhelmed with traffic and then get dropped for being unresponsive.  On the next pass, since they were no longer getting traffic and were responsive again, they'd get added back in, and immediately get dropped again on the next pass.  Rinse and repeat.

One of the mirrors this happened to during MU had a weight of 15000 (no, I'm not kidding) which was almost a quarter of the entire pool.  You can imagine every time that got pulled, the traffic went up a LOT for everyone else in the pool.

One way we thought of to possibly deal with this is to have sentry automatically decrease the weight on the mirror by a percentage (perhaps 10%) every time it gets re-added after being pulled for unresponsiveness.  The original starting weight should probably be recorded somewhere...  perhaps append to a comment field every time it gets dropped, or maybe just add a new column for the manually-set weight, and leave that one alone.

Comment 1

[Dave Miller [:justdave]]

I spent most of Wednesday (starting at around 3am when one of the primary mirrors started misbehaving and paged me) doing this by hand. :(  It was a never-ending battle.

Comment 2

[Dave Miller [:justdave]]

Attachment: patch v1

Comment 3

[Dave Miller [:justdave]]

I committed this to the production tag a while back.