1236838 - (primarily 404) monitoring of mobile.cdn.mozilla.net and downloadable content (DLC)

Status: RESOLVED INACTIVE

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Sebastian Kaspari (:sebastian; :pocmo)]

With "Downloadable Content (DLC)" we are starting to download default fonts from mobile.cdn.mozilla.net (meta bug 1194338) instead of shipping them with Fennec (To reduce the app size and eventually update content independently from releases). The implementation is now ready for a test in Nightly (bug 1233799).

Before we go ahead I was wondering:
* Do we monitor mobile.cdn.mozilla.net in any way?
* Do we get notified if the clients would do unexpected things? (For example lots of 404s)
* Is there some way to see usage in general? If I could see Nightly clients successfully download the fonts then that would be nice; But that's not critical in any way.

Comment 1

[:Margaret Leibovic]

Shyam, can you help answer these questions? We don't have access to the kanban board, so we can't see the status of any work there. Thanks!

Comment 2

[:Atoll]

(In reply to :Margaret Leibovic from comment #1)
> Shyam, can you help answer these questions? We don't have access to the
> kanban board, so we can't see the status of any work there. Thanks!

Our kanban board is set up to mirror of Bugzilla, and has no "status of work" information other than what's in Bugzilla.

Comment 3

[:Atoll]

(In reply to Sebastian Kaspari (:sebastian) from comment #0)
> With "Downloadable Content (DLC)" we are starting to download default fonts
> from mobile.cdn.mozilla.net (meta bug 1194338) instead of shipping them with
> Fennec (To reduce the app size and eventually update content independently
> from releases). The implementation is now ready for a test in Nightly (bug
> 1233799).
> 
> Before we go ahead I was wondering:
> * Do we monitor mobile.cdn.mozilla.net in any way?
> * Do we get notified if the clients would do unexpected things? (For example
> lots of 404s)
> * Is there some way to see usage in general? If I could see Nightly clients
> successfully download the fonts then that would be nice; But that's not
> critical in any way.

MOC handles monitoring, so I'm pushing this request over to them; they can help you set up whatever monitoring we have available.

Metrics would have the usage statistics for this domain, if any are available. They may need time to set up the metrics, so reach out to them if you need them and we'll make it happen.

Comment 4

[Richard Newman [:rnewman]]

:bobm, this is the monitoring bug.

To clarify, assuming that we can trust our CDN provider(s), and assuming that we have stable steps for deploying and QAing new content, the most likely error here is about missing files or a catalog/content mismatch between Kinto and S3.

CCing :tarek to think about that some.

Comment 5

[Bob Micheletto [:bobm]]

Assigning this bug to JP.

Comment 6

[JP Schneider [:jp]]

We've got pretty reliable telemetry from AWS Cloudwatch on:
    Requests – Number of requests for all HTTP methods and for both HTTP and HTTPS requests
    BytesDownloaded – Number of bytes downloaded by viewers for GET, HEAD, and OPTIONS requests
    BytesUploaded – Number of bytes uploaded to your origin with CloudFront using POST and PUT requests
    TotalErrorRate – Percentage of all requests for which the HTTP status code is 4xx or 5xx
    4xxErrorRate – Percentage of all requests for which the HTTP status code is 4xx
    5xxErrorRate – Percentage of all requests for which the HTTP status code is 5xx

We can route that to datadog, newrelic, etc...whatever the audience needs and we usually do.  
From there, we can hang alerts and notifications as needed.

Comment 7

[Sebastian Kaspari (:sebastian; :pocmo)]

(In reply to JP Schneider [:jp] from comment #6)
> We've got pretty reliable telemetry from AWS Cloudwatch on:
>     Requests – Number of requests for all HTTP methods and for both HTTP and
> HTTPS requests
>     BytesDownloaded – Number of bytes downloaded by viewers for GET, HEAD,
> and OPTIONS requests
>     BytesUploaded – Number of bytes uploaded to your origin with CloudFront
> using POST and PUT requests
>     TotalErrorRate – Percentage of all requests for which the HTTP status
> code is 4xx or 5xx
>     4xxErrorRate – Percentage of all requests for which the HTTP status code
> is 4xx
>     5xxErrorRate – Percentage of all requests for which the HTTP status code
> is 5xx

Do we have all this now after the CDN move or do we need to specifically configure those? I'm wondering what has to happen before I can enable the font downloader in Nightly (bug 1233799).

Comment 8

[JP Schneider [:jp]]

Good morning and happy Friday!

We currently are collecting metrics, but not yet alerting on anything.  

Depending on the audience receiving the monitor/alert on 4xx/5xx rates, we can get the metrics and alerts where they need to go...so where is that?  :)

We can hook Datadog into this metric very easily, or provide email/pagerduty notifications and alerts straight from AWS,  etc...   Is that going to MOC, or someone else?

Comment 9

[Richard Newman [:rnewman]]

(In reply to JP Schneider [:jp] from comment #8)

> We can hook Datadog into this metric very easily, or provide email/pagerduty
> notifications and alerts straight from AWS,  etc...   Is that going to MOC,
> or someone else?

I guess there are two sets here: stuff that got screwed up operationally (e.g., accidentally changing permissions or deleting a bucket), for which the MOC or someone in product-ops should be aware, and stuff that's a mismatch between client and server.

Betting that the latter is our first port of call, an email alert to [e-mail] when we bump over a noise threshold is probably a good starting point. Email is likely to be an adequate channel.

Let's figure out what that noise threshold is after a week or two on Nightly.

Thanks, JP!