1024927 - Loop server - reporting performance

Status: VERIFIED FIXED

(Hello (Loop) :: Server, defect, P1)

Comment 1

[Romain Testard [:RT]]

As mentioned by Adam a better metric of server load is the maximum number of events per second and concurrent connections (note that the concurrent connections are not a reflection of the concurrent call). 
We also want to know the nature of the user agent in order to differentiate load related to FFOS client or FF client

Comment 2

[James Bonacci [:jbonacci]]

Adding OPs here...

Comment 3

[Tarek Ziadé (:tarek)]

Can we manage these via stackdriver Benson ? or do we need a custom dashboard

Comment 4

[Tarek Ziadé (:tarek)]

^

Comment 5

[Benson Wong [:mostlygeek]]

Heka is likely a better fit for this. Stackdriver will give us server stats and some ELB req/second stats. We'll probably need to instrument loop-server to output these metrics periodically.

Comment 6

[Romain Testard [:RT]]

Katie, as discussed on Friday could you please give us an overview of what standard monitoring you typically have in place to monitor server performance so that I can validate if it fulfills the partner needs here? As discussed the user story may be too detailed and the overall ask is to have a view of how the server performs.

Comment 7

[Romain Testard [:RT]]

I removed the user story given that this is actually not a requirement from a product/partner standpoint.
I suggest this bug tracks any standard monitoring required to monitor the server load for capacity management purposes.

Comment 8

[Katie Parlante]

Standard Heka/Kibana setup we do for most servers:
- HTTP Status, from nginx logs. Monitor 2xx/4xx/5xx rates. Send email alerts from heka when anomalies are detected. From Kibana dashboards, can segment by User Agent, or by specific instances.
- HTTP Response times, from nginx logs. Monitor response times. Send email alerts from heka on anomalies. View mean/max in Kibana (adjustable time periods).

Common app specific logging/alerts/dashboards:
- Active daily user counts (uses heka's hyperloglog to count/estimate unique ids per day)
- app specific error rates
- counts and/or anomaly detection for specific endpoints (for fxa, create, login & email resend)
- mysql slow queries

In Stackdriver:
- Elastic Load Balancer req/s, 4xx/5xx, average latency
- cpu, memory usage (avg & per instance)
- network inbound, outbound traffic (kb/s)
- DB IOPS, latency

If there is any metric related to load not captured by the ELB or nginx endpoint logging, it should be logged by the loop server specifically, and we can monitor/graph that w/heka.

As discussed here: https://bugzilla.mozilla.org/show_bug.cgi?id=1003174#c15, I recommend a summary log line for each "request" that adds any app specific information (like uid, any app specific error codes, etc.)

Comment 9

[Alexis Metaireau (:alexis)]

https://github.com/mozilla-services/loop-server/commit/61da2e98999f74eefe29b36d6d45c61eeb678631

Comment 10

[u279076]

James, looks like this slipped off our radar. Can you please test this when you get back, or mark qa- if testing is no longer needed? Thanks.

Comment 11

[James Bonacci [:jbonacci]]

Quick code check. All looks good.