804119 - high ftp load, sentry pulling CDNs, etc

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Peter Radcliffe [:pir]]

ftp1-4.dmz.scl3.mozilla.com all spiked in load (that's what alerted) and when checked iowait had spiked hard for no obvious reason.

Ganglia showed outbound network traffic somewhat increased (but not inbound). Small amount of sockets in TIME_WAIT all of a sudden which supports networkio issues.

03:40:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
03:50:01 AM     all      1.52      0.00      2.12      1.79      0.00     94.58
04:00:01 AM     all      0.25      0.00      1.53      1.64      0.00     96.59
04:10:01 AM     all      0.33      0.00      1.72      3.76      0.00     94.18
04:20:01 AM     all      1.37      0.00      1.95      2.09      0.00     94.59
04:30:01 AM     all      0.24      0.00      1.64      3.36      0.00     94.76
04:40:01 AM     all      0.33      0.00      1.81      3.45      0.00     94.41
04:50:01 AM     all      0.25      0.00      1.65     22.71      0.00     75.39
05:00:01 AM     all      1.40      0.00      2.20     43.92      0.00     52.48
05:10:01 AM     all      1.46      0.00      2.19     49.29      0.00     47.06
05:20:01 AM     all      0.51      0.00      2.07     56.31      0.00     41.11
05:30:01 AM     all      0.31      0.00      1.78     20.10      0.00     77.80

None of the problems mentioned in bug 752399 are currently there.

Load recovered, leaving this open for further debugging.

Comment 1

[Peter Radcliffe [:pir]]

https://ganglia-scl3.mozilla.org/ganglia/?r=week&cs=&ce=&m=cpu_wio&s=by+name&c=productdelivery-ftp&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4

Comment 2

[Peter Radcliffe [:pir]]

16:07 <@gcox:#systems> fox2mike: I killed off a long-running dedupe, which made 
  things clean up pretty well from where I see.  ftp2 load average was 1400, 
  now 2.

Comment 3

[Peter Radcliffe [:pir]]

sentry pulled the CDNs so the ftp servers were getting way more traffic than they should have. This DoSed the ftp servers, the zeus nodes (with various other things like hg being collateral damage) and scl3 in general.

sentry has been tweaked to Not Do That, the traffic has reduced and things have settled back to near normal.

Comment 4

[Aki Sasaki (not active)]

We're seeing some more 500 ise's on ftp; is this still happening, or is that something else?

Comment 5

[Rick Bryce [:rbryce]]

gcox is looking into the storage backend at the moment.

Comment 6

[Peter Radcliffe [:pir]]

This bug is still open because although the major symptoms are not present all the time the issue is not solved. Until bug 804413 is resolved, meaning the load on the filer is reduced to a level where it can support the ftp load when it goes above normal background levels, and further testing has been done to establish the problem has been fixed the bug remains open for further notes and debugging.

If you have problems that this bug is causing/exacerbating then please make those bugs dependent on this bug.

Comment 7

[Shyam Mani [:fox2mike]]

Peter, now that we know the root cause of the netapp issues, do we still need this open?