1059637 - Packet loss between usw2 and scl3

Status: RESOLVED DUPLICATE of bug 1060416

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Smokeping to nagios1.private.releng.usw2

You can see from the graph that it's come and gone the last few days (my guess is that bug 1058073 would line up with a bad patch). Right now:

>                             My traceroute  [v0.75]
>bld-linux64-spot-444.build.releng.usw2.mozilla.com (0.0.Wed Aug 27 21:51:06 2014
>Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>                                          Packets               Pings
> Host                                   Loss%   Snt Drop   Avg  Best  Wrst StDev
> 1. 169.254.249.1                        0.0%   373    0   1.2   0.3  18.5   2.6
> 2. 169.254.249.25                       0.0%   373    0   0.8   0.4  31.8   2.2
> 3. 169.254.249.26                      22.5%   373   84  47.4  23.9 196.0  33.2
> 4. v-1033.border1.scl3.mozilla.net     16.9%   373   63  54.0  26.0 262.9  48.9
> 5. v-1029.fw1.scl3.mozilla.net         14.8%   372   55  53.5  28.4 280.6  48.6
> 6. 10.22.74.160                        17.0%   372   63  50.9  26.8 199.6  38.7

Packet loss shows up as slow transfer rates, which show up in retries and timeouts for tooltool, proxxy, and so on.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Note that smokeping is testing scl3 -> usw2 somewhere in netops-land, and mtr is testing from a spot instance back to runtime-binaries.pvt.build.mozilla.org (the releng web cluster).

https://hg.mozilla.org/build/cloud-tools/rev/b5be986133f7 to prefer us-east-1 over us-west-2, the smokeping for the former looks fine.

Comment 2

[Nigel Babu [:nigelb]]

I've closed all trees since we're getting consistent failures. If/When the first batch of retries come back green, I'll re-try the lot and re-open.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Pretty sure rail told me that sort of change doesn't have any effect on spot instances, which is backed up by 1050 running in usw2 vs 450 in use1, as of right now. Smokeping is much healthier for usw2 which is presumably why the integration trees are reopened again. They were closed between 2020 and 2326 Pacific.

Comment 4

[Jordan Lund (:jlund)]

marking this as a duplicate of 1060416. leaving 1060416 (releng side of things) and 1060407 (netops) open until reset our usw2 limits