1388025 - Get rid of crash_aggregates in favor of error_aggregates

Status: RESOLVED FIXED

(Data Platform and Tools :: General, task, P2)

Description

[Mauro Doglio [:mdoglio]]

The Mission Control dataset (error_aggregates) schema is a superset of what crash_aggregates offers. Also, its latency is <1h vs up to 24h. We should get rid of crash_aggregates and save both machine and human time.

Comment 1

[David Durst [:ddurst]]

Is that (error_aggregates) coming from main pings, or is it using crash pings? I'm wondering if we should focus on using error_aggregates for everything dealing with crashes or if we should be creating a new crash_aggregates based on crash pings.

Comment 2

[Mauro Doglio [:mdoglio]]

It's using both, like crash_aggregates does. I like the idea of having a crash ping-only aggregates dataset. I would be happy to work on that if it becomes a priority.

Comment 3

[David Durst [:ddurst]]

With 55 in release, we finally have the full capability to correlate crash reports with crash pings (if applicable). So if we could get this dataset up and running, we could start looking at ping-only data on all of release, even segmented by process type. I don't know who would determine the priority of this, but it would be very useful to get dashboards hooked up to this data as we lead up to 57.

Comment 4

[Frank Bertsch [:frank]]

(In reply to Mauro Doglio [:mdoglio] from comment #2)
> It's using both, like crash_aggregates does. I like the idea of having a
> crash ping-only aggregates dataset. I would be happy to work on that if it
> becomes a priority.

We could use direct-to-parquet for this dataset. It doesn't require parsing the entire payload, you can have it output a subset of fields to parquet, which in this case would probably just be that dimensional information (version, app, build, etc.).

Comment 5

[Mauro Doglio [:mdoglio]]

It sounds like we can:
1- get rid of crash_aggregates
2- use crash_summary (which is already in parquet) to derive new datasets (like the one for crash reports correlations that :ddurst is talking about).

Comment 6

[Firefox Bug Husbandry Bot]

Moved, per https://bugzilla.mozilla.org/show_bug.cgi?id=1453996

Comment 7

[William Lachance (:wlach)]

I'll take this.

Comment 8

[William Lachance (:wlach)]

First step is going through redash to find queries still scheduled against this dataset. There were quite a few, but most were obviously out of date / not relevant any more. Except for some that were scheduled/run by [e-mail] and [e-mail], [e-mail].

Romain, Harald, Calixte: can you let me know if it's ok to stop scheduling runs of these queries? If the data is still useful, can I help you migrate to error_aggregates?

https://docs.telemetry.mozilla.org/datasets/streaming/error_aggregates/reference.html

The schema is fairly similar so it shouldn't take long to update your queries if needed.

[e-mail]:

Release_OK 32 VS 64-bit Crash Rates: https://sql.telemetry.mozilla.org/queries/14107/source
ESR 32 VS 64-bit Crash Rates: https://sql.telemetry.mozilla.org/queries/51957/source
Release 32 VS 64-bit Crash Rates: https://sql.telemetry.mozilla.org/queries/4655/source

[e-mail]:

Firefox Beta Crash Rates By Version/Build, v2 https://sql.telemetry.mozilla.org/queries/2856/source
Usage Hour ratio e10s/non-e10s https://sql.telemetry.mozilla.org/queries/972/source
Release Crash Rate by Activity Date https://sql.telemetry.mozilla.org/queries/331/source
Firefox Aurora Crash Rates By Version/Build https://sql.telemetry.mozilla.org/queries/999/source
Firefox Crash Rates by Release/Date https://sql.telemetry.mozilla.org/queries/184/source
Firefox Watershed Beta 44.0b1 https://sql.telemetry.mozilla.org/queries/469/source
[deprecated] Firefox Beta Crash Rates By Version/Build, v1 https://sql.telemetry.mozilla.org/queries/207/source
Beta Usage Hour ratio e10s/non-e10s https://sql.telemetry.mozilla.org/queries/1415/source

[e-mail]:

Daily usage khours for Firefox release, beta, aurora and nightly: https://sql.telemetry.mozilla.org/queries/346/source

Comment 9

[Romain Testard [:RT]]

OK for my queries, the ones mentioned above won't run anymore.

Comment 10

[Calixte Denizet (:calixte)]

I just killed mine.

Comment 11

[:Harald Kirschner :digitarald]

OK for mine, health.graphics doesn't use them anymore.

Comment 12

[William Lachance (:wlach)]

Thanks all! Harald: I took the liberty of unscheduling your queries for you, since you weren't using them anymore. :)

I will go ahead with the subsequent steps in the deprecation process.

Comment 13

[William Lachance (:wlach)]

This dataset has been unscheduled from airflow and removed from hive metastore, next step per our deprecation policy is to wait a week and make sure nothing breaks.

Comment 14

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-batch-view/pull/530

Comment 15

[William Lachance (:wlach)]

All done!