1060407 - High packet loss from usw2 causing frequent timeouts

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

All trees closed.

Comment 1

[Jordan Lund (:jlund)]

Attachment: bug_1060407_disable_usw2.patch

for landing: I think aws-manager's cloud tools updates via cron now

I would assume 0'ing this would have no effect after this patch: http://mxr.mozilla.org/build/source/cloud-tools/configs/watch_pending.cfg#3 ?

Comment 2

[Jordan Lund (:jlund)]

Comment on attachment 8481439 [details] [diff] [review]
bug_1060407_disable_usw2.patch

checked in: https://hg.mozilla.org/build/cloud-tools/rev/3d7663029c69

I think we will need to keep an eye on pending and limits.

These dials might need to be adjusted: http://mxr.mozilla.org/build/source/cloud-tools/configs/watch_pending.cfg#115

Comment 3

[Jordan Lund (:jlund)]

Attachment: bug_1060407_limit_usw2_to_750.patch

we have not experienced the packet loss since we stopped bringing up new usw2 slaves. We went from ~1100 before disabling to 781.

This patch will ensure that we stay around that limit (750) and proportion the slaves to what we currently consider as our 'usw2 limit capacity'. which was:

we can gradually increase this number as needed until we discover the root cause of network woes.

Comment 4

[Jordan Lund (:jlund)]

Comment on attachment 8481573 [details] [diff] [review]
bug_1060407_limit_usw2_to_750.patch

pushed: https://hg.mozilla.org/build/cloud-tools/rev/f0d66541f48c

Comment 5

[Justin Wood (:Callek)]

Backed out all the above changes with https://hg.mozilla.org/build/cloud-tools/rev/e47bf4c569dd

Reso for now.