All trees closed.
Created attachment 8481439 [details] [diff] [review] bug_1060407_disable_usw2.patch for landing: I think aws-manager's cloud tools updates via cron now I would assume 0'ing this would have no effect after this patch: http://mxr.mozilla.org/build/source/cloud-tools/configs/watch_pending.cfg#3 ?
Comment on attachment 8481439 [details] [diff] [review] bug_1060407_disable_usw2.patch checked in: https://hg.mozilla.org/build/cloud-tools/rev/3d7663029c69 I think we will need to keep an eye on pending and limits. These dials might need to be adjusted: http://mxr.mozilla.org/build/source/cloud-tools/configs/watch_pending.cfg#115
Created attachment 8481573 [details] [diff] [review] bug_1060407_limit_usw2_to_750.patch we have not experienced the packet loss since we stopped bringing up new usw2 slaves. We went from ~1100 before disabling to 781. This patch will ensure that we stay around that limit (750) and proportion the slaves to what we currently consider as our 'usw2 limit capacity'. which was: we can gradually increase this number as needed until we discover the root cause of network woes.
Comment on attachment 8481573 [details] [diff] [review] bug_1060407_limit_usw2_to_750.patch pushed: https://hg.mozilla.org/build/cloud-tools/rev/f0d66541f48c
Backed out all the above changes with https://hg.mozilla.org/build/cloud-tools/rev/e47bf4c569dd Reso for now.