Closed Bug 1171383 Opened 11 years ago Closed 11 years ago

Outage from SCL3.

Categories

(Infrastructure & Operations Graveyard :: NetOps: DC Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
NetOps: DC Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rwatson, Assigned: arzhel)

Details

(Whiteboard: [nsm])

This morning I experienced a disconnect from IRC and then alerts from pingdom, nagios > PagerDuty from machines in SCL3. Reporting major sites (mozilla.org) as down. The outage lasted approx 8 minutes. Filing this bug for netops investigation. The few alerts I did get to IRC before D/C were: 10:07:48 <nagios-scl3> Thu 02:07:30 PDT [5022] zlb6.ops.scl3.mozilla.com:ZXTM - procs is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. (http://m.mozilla.org/ZXTM+-+procs) 10:08:06 <nagios-scl3> Thu 02:07:48 PDT [5023] admin1.pek1.mozilla.com (223.202.6.5) is DOWN :PING CRITICAL - Packet loss = 100%
From PagerDuty, here is a list of hosts that alerted and their corresponding #ID 9344 Jun 4, 2015 at 10:32 AM Host: switch3.r202-8.ops.releng.scl3.mozilla.net 9342 Jun 4, 2015 at 10:15 AM Host: nagios-zlb.vips.scl3.mozilla.com 9341 Jun 4, 2015 at 10:15 AM Host: video-origin-zlb.vips.scl3.mozilla.com 9340 Jun 4, 2015 at 10:15 AM Host: l10n-dashboard-zlb.vips.scl3.mozilla.com 9339 Jun 4, 2015 at 10:15 AM Host: fw1.tpe1.mozilla.net 9338 Jun 4, 2015 at 10:15 AM Host: www.mozilla.org 9337 Jun 4, 2015 at 10:15 AM Host: ftp2-zlb.vips.scl3.mozilla.com 9336 Jun 4, 2015 at 10:14 AM Host: developer2.webapp.scl3.mozilla.com 9335 Jun 4, 2015 at 10:14 AM Host: developer3.webapp.scl3.mozilla.com 9334 Jun 4, 2015 at 10:14 AM Host: zlb6.ops.scl3.mozilla.com 9333 Jun 4, 2015 at 10:13 AM Host: reviewboard.mozilla.org 9332 Jun 4, 2015 at 10:13 AM Host: mxr-zlb.vips.scl3.mozilla.com 9331 Jun 4, 2015 at 10:13 AM Host: bedrock-prod-zlb.vips.scl3.mozilla.com 9319 Jun 4, 2015 at 10:12 AM Host: ns1-zlb.dmz.scl3.mozilla.com 9317 Jun 4, 2015 at 10:12 AM Host: releases-zlb.vips.scl3.mozilla.com 9316 Jun 4, 2015 at 10:11 AM Host: zlb6.ops.scl3.mozilla.com 9315 Jun 4, 2015 at 10:11 AM Host: developer-zlb.vips.scl3.mozilla.com 9314 Jun 4, 2015 at 10:11 AM Host: nagios.mozilla.org 9313 Jun 4, 2015 at 10:11 AM Host: reviewboard-prod.vips.scl3.mozilla.com 9312 Jun 4, 2015 at 10:10 AM Host: ftp2-zlb.vips.scl3.mozilla.com 9311 Jun 4, 2015 at 10:10 AM Host: fw1.tpe1.mozilla.net 9310 Jun 4, 2015 at 10:10 AM Host: wap413.ops.tpe1.mozilla.net 9309 Jun 4, 2015 at 10:10 AM Host: nagios-zlb.vips.scl3.mozilla.com. 9308 Jun 4, 2015 at 10:10 AM Host: data.mozilla.com 9307 Jun 4, 2015 at 10:10 AM Host: wap407.ops.tpe1.mozilla.net 9305 Jun 4, 2015 at 10:09 AM Host: partnerbuilds-zlb.vips.scl3.mozilla.com 9304 Jun 4, 2015 at 10:08 AM Host: wap414.ops.tpe1.mozilla.net 9303 Jun 4, 2015 at 10:08 AM Host: fw1.tor1.mozilla.net 9302 Jun 4, 2015 at 10:08 AM Host: fw1.tier2.tor1.mozilla.net 9301 Jun 4, 2015 at 10:08 AM Host: fw1.tier1.tor1.mozilla.net 9300 Jun 4, 2015 at 10:07 AM Host: zlb6.ops.scl3.mozilla.com 9299 Jun 4, 2015 at 10:07 AM Host: admin1.pek1.mozilla.com 9298 Jun 4, 2015 at 10:07 AM Host: zlb6.ops.scl3.mozilla.com
and the ones from pingdom: 9343 Jun 4, 2015 at 10:16 AM Pingdom Alert: Incident #11769 for Netops - (Telia)xe-0-0-2.border1.sjc2 (62.115.8.162), has been assigned to you. Pingdom -- Resolved Details 9330 Jun 4, 2015 at 10:13 AM Pingdom Alert: Incident #11756 for Netops - LO0.fw1.sjc2 (63.245.219.37), has been assigned to you. Pingdom -- Resolved Details 9329 Jun 4, 2015 at 10:12 AM Pingdom Alert: incidents have been assigned to you Pingdom -- Resolved Details 9328 Jun 4, 2015 at 10:12 AM Pingdom Alert: incidents have been assigned to you Pingdom -- Resolved Details 9327 Jun 4, 2015 at 10:12 AM Pingdom Alert: incidents have been assigned to you Pingdom -- Resolved Details 9326 Jun 4, 2015 at 10:12 AM Pingdom Alert: incidents have been assigned to you Pingdom -- Resolved Details 9325 Jun 4, 2015 at 10:12 AM Pingdom Alert: incidents have been assigned to you Pingdom -- Resolved Details 9324 Jun 4, 2015 at 10:12 AM Pingdom Alert: incidents have been assigned to you Pingdom -- Resolved Details 9322 Jun 4, 2015 at 10:12 AM Pingdom Alert: incidents have been assigned to you Pingdom -- Resolved Details 9321 Jun 4, 2015 at 10:12 AM Pingdom Alert: incidents have been assigned to you Pingdom -- Resolved Details 9318 Jun 4, 2015 at 10:12 AM Pingdom Alert: incident #11749 for MTV2-Admin (admin1.mtv2.mozilla.com) has been reassigned Pingdom -- Resolved Details 9306 Jun 4, 2015 at 10:09 AM Pingdom Alert: Incident #11735 for vrouter5.mozilla.com (vrouter5.mozilla.com), has been assigned to you. Pingdom -- Resolved Details
Still investigating, but so far it seems like a brief DDOS attack There was a large inbound UDP and ICMP traffic spike from both our upstream providers: https://observium.private.scl3.mozilla.com/graphs/to=1433416666/id=896144/type=port_bits/from=1433395066/ https://observium.private.scl3.mozilla.com/graphs/to=1433416399/id=896054/type=port_bits/from=1433394799/ From ~500M to ~5G on one link and from ~100M to ~10G on the other
Assignee: network-operations → arzhel
Arzhel, do we know what was the attack source and pattern? Let me know if I can help here.
I did some investigation today. It was a DDoS attack on bedrock, using DNS and NTP reflection and some random UDP packets as well. Number of flow records (which might or might not correspond to connections) During the attack (30 minute window) 763875 pao1.all 265331 pao1.tcp 494349 pao1.udp A day before on the same time 1201 pao1.icmp 156 pao1.minus-1.icmp 219531 pao1.minus-1.tcp 2272 pao1.minus-1.udp Two days before on the same time 210 pao1.minus-2.icmp 242429 pao1.minus-2.tcp 2172 pao1.minus-2.udp You can clearly see a large number of UDP flows. Let's dive in some more. UDP flows to 63.245.215.20 - 489445 UDP flows NOT to 63.245.215.20 - 4904 20.215.245.63.in-addr.arpa. 2934 IN PTR bedrock-prod-zlb.vips.scl3.mozilla.com. That means other UDP traffic accounted for 1% of flows. 99% of flows were directed to bedrock. Aggregation by destination IP 2015-06-04 02:05:12.863 1486.056 UDP 63.245.215.20 1.7 M 803.3 M 4.3 M 479 489445 Bedrock clearly stands out with 489445 connections. Example record when aggregating by source ip 2015-06-04 02:09:14.553 0.000 UDP ^A 87.156.86.240 1 190 0 190 1 A quick look at the logs shows that packets had 190 bytes in size. The number of NTP+DNS packets sent to us does not add to the total amount of packets during the attack, and browsing through logs I can see random source/destination ports used. Based on the above it looks like we were a target of a DDoS from some large botnet - 353389 source IP addresses.
Can we drop all UDP traffic on the edge and allow only the ones we use, after some policing? I don't see a reason we allow unlimited stream of UDP on some random ports without any rate limiting. This limit for 'everything else' that is not DNS and NTP (which we already police) could be set high, but present. Like 100Mbit or even 1Gbit.
Whiteboard: [nsm]
NSM data are also interesting. nsm15 :: bro/logs/2015-06-04 » zcat conn.09:00:00-09:15:37.log.gz | grep udp | grep -E '63.245.215.20' | wc -l 130 ↵ 7268933 nsm15 :: bro/logs/2015-06-04 » zcat conn.09:00:00-09:15:37.log.gz | grep udp | grep -v -E '63.245.215.20' | wc -l 4357 Also, NSM noticed that over 99% of connection used destination port 20480, packet size 190 bytes.
NSM stayed online during the attack without any problems. Some workers crashed but recovered 20 seconds later.
(In reply to Michal Purzynski [:michal`] (use NEEDINFO) from comment #6) > Can we drop all UDP traffic on the edge and allow only the ones we use, > after some policing? I don't see a reason we allow unlimited stream of UDP > on some random ports without any rate limiting. > > This limit for 'everything else' that is not DNS and NTP (which we already > police) could be set high, but present. Like 100Mbit or even 1Gbit. This isn't a solution NetOps can implement with confidence. We have anticipated UDP traffic such as Vidyo and perhaps SIP. We'll look at per-prefix options as a remedial measure. We've also identified a DoS solution that should provide timely information that may be of benefit.
James, several folks mentioned wansight as a proactive tool for mitigating these sorts of attacks in the future. I understand it has been purchased, but I'm not sure of the operational state. Does your team have a implementation plan for it?
This is the procurement process, I expect it to be purchased shortly. The primary purpose is network visibility (think netflows) with an eye to use the DDoS capabilities. I'd expect this to be implemented by late Q3. Question for you ... is Bro able to detect this type of event?
Can Bro make you a coffee? Yes. Can it manage the ICS in your house? Yes. Should it? Not really ;-) Touring complete language, with events, schedulers, first class data types for everything you see in networks. And executing external applications. Asynchronously. This is a brave idea, and I would say, using Bro to detect DDoS is like using GROM as a regular troops. Let's use wansight's analyzers purposely built for detecting DDoS and providing you with some data to react, and when the attack comes, we will use Bro data to further understand the DDoS.
(In reply to Michal Purzynski [:michal`] (use NEEDINFO) from comment #12) > Can Bro make you a coffee? > > Yes. > > Can it manage the ICS in your house? > > Yes. > > Should it? > > Not really ;-) > > Touring complete language, with events, schedulers, first class data types > for everything you see in networks. And executing external applications. > Asynchronously. > > This is a brave idea, and I would say, using Bro to detect DDoS is like > using GROM as a regular troops. > > Let's use wansight's analyzers purposely built for detecting DDoS and > providing you with some data to react, and when the attack comes, we will > use Bro data to further understand the DDoS. I like my coffee black ... how do I get it dispense that?
(In reply to James Barnell [:jbarnell] from comment #13) > (In reply to Michal Purzynski [:michal`] (use NEEDINFO) from comment #12) > > Can Bro make you a coffee? > > > > Yes. > > > > I like my coffee black ... how do I get it dispense that? Warning: black coffee dispensed from a robot may be indistinguishable from used hydraulic fluid!
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.