552481 - connection between castro and mpt seems to be in rough shape

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

We've had a lot of jobs die in the past hour or so. Some due to steps failing, others due to slave lose. All of the failed builds happened on slaves which are located at Castro.

How's the Castro <-> MPT link? I heard last week that it was over capacity already.

Marking blocking because this is interfering with releases

Comment 1

[Derek Moore [:dmoore]]

We're working on this, at the moment. The link to MPT is currently at 100% capacity, and we're trying to gracefully throttle certain traffic.

Bug 551187 will also help alleviate this.

Comment 2

[Daniel Holbert [:dholbert]]

Note that this connectivity problem has caused serious tinderbox redness (bug 552506), and so the Firefox tree is currently closed to checkins until this is fixed enough to not affect tinderbox.

Comment 3

[matthew zeier [:mrz]]

Bug 551187 is tracking setting up a proxy @ Castro to reduce bandwidth needs.

Comment 4

[matthew zeier [:mrz]]

calling closed.  proxy helped, working on doubling bandwidth out of the office.