1126410 - Hello is failing to respond to some requests

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Mark Banner (:standard8)]

We're seeing various people report that Hello isn't working. This started around 18:25 UTC and is still going on for some people.

One report was:

"Loop hawkRequest error:" Object { error: Exception, message: null, code: null, errno: null, toString: this.HawkClient.prototype._constructError/errorObj.toString() }
"RoomList error" Error: NS_ERROR_NET_RESET

Other people have not been able to get the room lists.

We're also not seeing any failure information on the status monitoring at https://status.services.mozilla.com/

Comment 1

[Bob Micheletto [:bobm]]

This problem was the result of a load test being run against the production environment.  It exposed a number of shortfalls that should be addressed in separate bugs.  I shall list them here, and close out this bug after creating the bugs for those individual issues.

Monitoring: The lack of anomaly alerting led to this being a user reported issue.  Anomalous fluctuations in traffic should be detected, and alerted through normal escalation channels.  Also, periods of anomalous traffic activity should be noted on the status dashboard.

Auto-scaling: During this issue, the AWS Auto Scaling Group for the loop-server application servers should have spawned new instances to deal with the increased load.  It did not.

IP Throttling: A sensible rate limiting policy on Production would have done much to filtering out this traffic.  See: https://github.com/mozilla/videur

Feature request for Loads V2: :benbangert mentioned that it might make good sense to add an IP blacklisting, or other type of failsafe in Loads V2 to discourage the load testing of production environments.

Comment 2

[Bob Micheletto [:bobm]]

Fixing QA contact.  Bugzilla is _helpful_ sometimes.

Comment 3

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #1)

> Monitoring: The lack of anomaly alerting led to this being a user reported
> issue.  Anomalous fluctuations in traffic should be detected, and alerted
> through normal escalation channels.  Also, periods of anomalous traffic
> activity should be noted on the status dashboard.

Filed under Bug 1126589.

Comment 4

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #1)

> Auto-scaling: During this issue, the AWS Auto Scaling Group for the
> loop-server application servers should have spawned new instances to deal
> with the increased load.  It did not.

Filed in Bug 1126605.

Comment 5

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #1)
> IP Throttling: A sensible rate limiting policy on Production would have done
> much to filtering out this traffic.  See: https://github.com/mozilla/videur

Filed in Bug 1126611.

Comment 6

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #1)
 
> Feature request for Loads V2: :benbangert mentioned that it might make good
> sense to add an IP blacklisting, or other type of failsafe in Loads V2 to
> discourage the load testing of production environments.

Added feature request here: https://github.com/loads/loads-broker/issues/19

Comment 7

[Bob Micheletto [:bobm]]

Closing this bug.

Comment 9

[Alexis Metaireau (:alexis)]

Thanks for filling these bugs Bob,

Another one I see (but I don't know if that's covered already or not), is the monitoring of our response times, and alerting based on that.

Are we currently doing that through heka? If not, I believe we should!

Comment 10

[Bob Micheletto [:bobm]]

Alexis, there isn't a specific monitor for it.  However, we are logging the request_time for users and we can add a summary to the Kibana dashboard to get that information.

Comment 11

[Bob Micheletto [:bobm]]

(In reply to Alexis Metaireau (:alexis) from comment #9)

I should mention, we used response time anomaly detection in Sync 1.5 for awhile, and found it hard filter out the actionable alerts.  I think we should establish a baseline model here for what they look like over time before turn those on.