1275010 - MESSAGE_MANAGER_MESSAGE_SIZE histogram should not be aggregated

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Roberto Agostino Vitillo (:rvitillo)]

MESSAGE_MANAGER_MESSAGE_SIZE has over 10K unique keys for a single submission date, like:

"ublock0:sb:3471"
"ublock0:sb:2618"
"ublock0:sb:3792"
...

This is causing the aggregation service, that supports t.m.o, to take a *very* long time to compute aggregates and render the service unavailable in some cases.

The aggregation service was never designed to support histograms with that many keys and the histogram should be filtered out entirely by the aggregation pipeline. 

Bill, I am assuming that you are not using the aggregates from t.m.o., am I right?

Comment 1

[Roberto Agostino Vitillo (:rvitillo)]

Apparently some days have over 100K unique keys...

Comment 2

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

I'm not sure what you mean by "aggregates". We definitely use the data on telemetry.mozilla.org to look at the MESSAGE_MANAGER_MESSAGE_SIZE histogram (or we did when it first landed; it seems to take forever now).

I filed bug 1275032 just now with an idea to prevent these ublock keys from polluting the data. In the short term, could you throw away any keys that contain digits? We don't care about them and it would eliminate all these ublock keys. Doing this retroactively would be fine too.

Sorry for the trouble. I wasn't aware ublock was doing this when I landed the change.

Comment 3

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Bill McCloskey (:billm) from comment #2)
> I'm not sure what you mean by "aggregates". We definitely use the data on
> telemetry.mozilla.org to look at the MESSAGE_MANAGER_MESSAGE_SIZE histogram
> (or we did when it first landed; it seems to take forever now).

Right, t.m.o isn't useful right now for MESSAGE_MANAGER_MESSAGE_SIZE. 

> I filed bug 1275032 just now with an idea to prevent these ublock keys from
> polluting the data. In the short term, could you throw away any keys that
> contain digits? We don't care about them and it would eliminate all these
> ublock keys. Doing this retroactively would be fine too.

The aggregation job is going to ignore the histogram entirely for the time being. I would like to avoid  special casing the aggregation job in order to fix issues with histograms that should be dealt with on the client side. Even if we landed such change, it would require to retroactively apply it on the database as well as t.m.o will happily continue to fetch aggregates built in the past month, which will render the service unavailable. This is maybe something Chris could look into?

For now, you can compute the aggregates by running your own Spark job.

Comment 4

[Roberto Agostino Vitillo (:rvitillo)]

Ref: https://github.com/mozilla/python_mozaggregator/pull/15

Comment 5

[Andrew McCreight [:mccr8] (PTO until 12-2)]

Once I've fixed the issues with the key spam (which I'm going to do in bug 1275707), should I just rename the telemetry from MESSAGE_MANAGER_MESSAGE_SIZE to something else? I'll get bsmedberg to review it. MESSAGE_MANAGER_MESSAGE_SIZE is on beta, and we're pretty late in beta, so I'm not sure I'm going to be able to fix it there before it goes to release.

Comment 6

[Andrew McCreight [:mccr8] (PTO until 12-2)]

Sorry, I should have fixed this earlier on the client side. I'd noticed all of these keys a little while ago, and that the site was super slow sometimes when looking at them.

Comment 7

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Andrew McCreight [:mccr8] from comment #5)
> Once I've fixed the issues with the key spam (which I'm going to do in bug
> 1275707), should I just rename the telemetry from
> MESSAGE_MANAGER_MESSAGE_SIZE to something else? 

Yes, please rename the probe.

Comment 8

[Andrew McCreight [:mccr8] (PTO until 12-2)]

The new version has landed in bug 1275707, as MESSAGE_MANAGER_MESSAGE_SIZE2. No ublock data yet, but I'll keep an eye on it to double check that it isn't spammy.