Closed Bug 1388025 Opened 4 years ago Closed 2 years ago

Get rid of crash_aggregates in favor of error_aggregates


(Data Platform and Tools :: Datasets: General, task, P2)



(Not tracked)



(Reporter: mdoglio, Assigned: wlach)



(Whiteboard: [DataPlatform])


(1 file)

The Mission Control dataset (error_aggregates) schema is a superset of what crash_aggregates offers. Also, its latency is <1h vs up to 24h. We should get rid of crash_aggregates and save both machine and human time.
Is that (error_aggregates) coming from main pings, or is it using crash pings? I'm wondering if we should focus on using error_aggregates for everything dealing with crashes or if we should be creating a new crash_aggregates based on crash pings.
It's using both, like crash_aggregates does. I like the idea of having a crash ping-only aggregates dataset. I would be happy to work on that if it becomes a priority.
With 55 in release, we finally have the full capability to correlate crash reports with crash pings (if applicable). So if we could get this dataset up and running, we could start looking at ping-only data on all of release, even segmented by process type. I don't know who would determine the priority of this, but it would be very useful to get dashboards hooked up to this data as we lead up to 57.
(In reply to Mauro Doglio [:mdoglio] from comment #2)
> It's using both, like crash_aggregates does. I like the idea of having a
> crash ping-only aggregates dataset. I would be happy to work on that if it
> becomes a priority.

We could use direct-to-parquet for this dataset. It doesn't require parsing the entire payload, you can have it output a subset of fields to parquet, which in this case would probably just be that dimensional information (version, app, build, etc.).
It sounds like we can:
1- get rid of crash_aggregates
2- use crash_summary (which is already in parquet) to derive new datasets (like the one for crash reports correlations that :ddurst is talking about).
Depends on: 1382236
Priority: -- → P3
Moved, per
Component: Datasets: Crash Aggregates → Datasets: General
Assignee: nobody → akomarzewski
Priority: P3 → P2
Whiteboard: [DataPlatform]

I'll take this.

Assignee: akomarzewski → wlachance

First step is going through redash to find queries still scheduled against this dataset. There were quite a few, but most were obviously out of date / not relevant any more. Except for some that were scheduled/run by and,

Romain, Harald, Calixte: can you let me know if it's ok to stop scheduling runs of these queries? If the data is still useful, can I help you migrate to error_aggregates?

The schema is fairly similar so it shouldn't take long to update your queries if needed.

Release_OK 32 VS 64-bit Crash Rates:
ESR 32 VS 64-bit Crash Rates:
Release 32 VS 64-bit Crash Rates:

Firefox Beta Crash Rates By Version/Build, v2
Usage Hour ratio e10s/non-e10s
Release Crash Rate by Activity Date
Firefox Aurora Crash Rates By Version/Build
Firefox Crash Rates by Release/Date
Firefox Watershed Beta 44.0b1
[deprecated] Firefox Beta Crash Rates By Version/Build, v1
Beta Usage Hour ratio e10s/non-e10s

Daily usage khours for Firefox release, beta, aurora and nightly:

Flags: needinfo?(rtestard)
Flags: needinfo?(hkirschner)
Flags: needinfo?(cdenizet)

OK for my queries, the ones mentioned above won't run anymore.

Flags: needinfo?(rtestard)

I just killed mine.

Flags: needinfo?(cdenizet)

OK for mine, doesn't use them anymore.

Flags: needinfo?(hkirschner)

Thanks all! Harald: I took the liberty of unscheduling your queries for you, since you weren't using them anymore. :)

I will go ahead with the subsequent steps in the deprecation process.

Type: enhancement → task
Depends on: 1541148

This dataset has been unscheduled from airflow and removed from hive metastore, next step per our deprecation policy is to wait a week and make sure nothing breaks.

All done!

Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.