964853 - bad ping times from srv.releng.scl3, test.releng.scl3, probably others, to default gateway

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Chris AtLee [:catlee]]

Packets               Pings              
 Host                                              Loss%   Snt   Last   Avg  Best  Wrst StDev 
 1. 10.26.48.1                                      0.4%   496  212.4 209.1 145.0 285.9  26.8 
 2. v-1034.border2.scl3.mozilla.net                 1.0%   495  416.7 417.5 299.3 554.6  52.5 
 3. v-1030.fw1.scl3.mozilla.net                     1.4%   495  422.1 418.5 300.2 563.2  51.8 
 4. buildbot-rw-vip.db.scl3.mozilla.com             1.4%   495  431.3 422.1 304.0 564.6  52.2

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

The suspicion is that this is due to overload on fw1.

Comment 2

[Amy Rich [:arr] [:arich]]

We had issues with WDS as well, and I asked adam to take a look at the firewall then, and he said it wasn't even breathing hard.  I wonder if it's all part of one larger issue.

Comment 3

[Arzhel Younsi [:XioNoX]]

We believe this is related to the IPsec/VPC issue as the module in charge of ipsec encryption/decryption (called SPU) is also the one that takes care of firewall traffic.
The firewall seems to return an incorrect load value for that module, thus why it looks fine at first sight.

Bug 957502 is about solving that issue.

Comment 4

[Hal Wine [:hwine] use NI!]

releng firewall upgrade and other measures have corrected the ping times