addons3.stage.db.phx1 high packet loss

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: team73, Assigned: pradcliffe+bugzilla)

Tracking

Details

(Reporter)

Description

3 years ago
Hello

The node started paging frequently due to high packet loss. I dont see anything wrong with the node, but could you please verify too?

Thank you

(12:37:37 PM) nagios-phx1: Wed 03:37:36 PST [1011] addons3.stage.db.phx1.mozilla.com:PING is WARNING: PING WARNING - Packet loss = 87%, RTA = 51.27 ms (http://m.mozilla.org/PING)
nagios-phx1 nagios-scl3 
(12:41:30 PM) jruszo: nagios-phx1: recheck 1011
(12:41:30 PM) nagios-phx1: jruszo: rechecking all services on host addons3.stage.db.phx1.mozilla.com
(12:41:43 PM) nagios-phx1: Wed 03:41:42 PST [1013] addons3.stage.db.phx1.mozilla.com:PING is OK: PING OK - Packet loss = 50%, RTA = 51.22 ms (http://m.mozilla.org/PING)
(12:56:43 PM) nagios-phx1: Wed 03:56:42 PST [1016] addons3.stage.db.phx1.mozilla.com:PING is WARNING: PING WARNING - Packet loss = 88%, RTA = 51.30 ms (http://m.mozilla.org/PING)
(12:56:43 PM) nagios-phx1: Wed 03:56:42 PST [1018] addons3.stage.db.phx1.mozilla.com:PING is OK: PING OK - Packet loss = 66%, RTA = 51.34 ms (http://m.mozilla.org/PING)
(1:07:45 PM) jruszo: this node is acting strange
(1:08:01 PM) nagios-scl3: Wed 04:08:00 PST [5046] buildbot4.db.scl3.mozilla.com:DB Swap High is CRITICAL: SWAP CRITICAL - 2% free (23 MB out of 2047 MB) (http://m.mozilla.org/DB+Swap+High)
(Assignee)

Comment 1

3 years ago
Please don't put IT things in the community IT section.
Component: Community IT → MOC: Problems
QA Contact: mzeier → lypulong
(Assignee)

Comment 2

3 years ago
Seems like a temporary event. Not sure what was happening there.

 <nagios-phx1:#sysadmins> Wed 04:24:24 PST [1022] 
  addons3.stage.db.phx1.mozilla.com:PING is OK: PING OK - Packet loss = 0%, RTA 
  = 51.08 ms (http://m.mozilla.org/PING)
(Assignee)

Comment 3

3 years ago
Looks like this was one of the migrated hosts, taken from phx1 to scl3 without (yet) being renamed. Still monitored from phx1. A blip in Internet connectivity between the sites can mean monitoring issues like that. This will happen occasionally until the host migrations are finished and they are monitored from scl3.

Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.8.75.1                         0.0%   140    0.6   0.6   0.5   0.8   0.0
 2. v-1040.border1.phx1.mozilla.net   0.0%   140    0.9   5.8   0.8  80.6  16.0
 3. ???
 4. v-1029.border1.scl3.mozilla.net   0.0%   139   25.6  26.5  25.5  58.2   4.2
 5. v-1029.fw1.scl3.mozilla.net       4.3%   139   26.0  26.1  26.0  26.3   0.0
 6. addons3.stage.db.phx1.mozilla.co  2.2%   139   51.2  51.1  51.0  51.3   0.0
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Assignee)

Updated

3 years ago
Assignee: nobody → pradcliffe+bugzilla
You need to log in before you can comment on or make changes to this bug.