1619352 - Investigate validation errors in `telemetry.crash` for `#/environment`

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, defect, P4)

Description

[William Lachance (:wlach)]

https://datastudio.google.com/u/0/reporting/1MKUu478GKR7myUUe5eItvyCpxP5TBIqm/page/QDQ4

Validation errors in telemetry.crash for #/environment reached 309k in the last 28 days.

I will look into this.

Comment 1

[William Lachance (:wlach)]

Did up a query in redash, a cursory glance indicates that the error is usually that there is no build metadata. These pings seem highly useless (almost no information):

https://sql.telemetry.mozilla.org/queries/68798/source

:gsvelto, do you know how we might be generating such pings?

Comment 2

[Gabriele Svelto [:gsvelto]]

I've poked through the data a bit and it seems to me that those are all main crash pings (i.e. processType is set to main) which means they were sent by the crashreporter client directly. In that case if the ping is almost empty it's because the .extra file was borked. This can happen especially during OOM crashes. That being said I can see pings where there's quite a bit of data in the ping but the environment field is empty.

Since the TelemetryEnvironment annotation should contain JSON we parse it and use it to populate that field. If we fail parsing the JSON data in that annotation then we won't populate the field at all.

At the crash reporter level this is a typical case of JSON-within-JSON which makes things complicated. Do we have a feeling of how many of these crashes are missing that field compared to the total amount of main-process crash pings we receive?

Comment 3

[Gabriele Svelto [:gsvelto]]

I think I know how I could figure out what's going wrong. Is it possible to pull out of this data a list of all the minidumpSha256Hash values for these pings? Armed with the list I could look up corresponding crashes on Socorro and see what's in the environment. The reason why I'm asking for all of them is that we usually have two order of magnitude less crash reports than crash pings so searching a handful by hand won't cut it.

Comment 4

[William Lachance (:wlach)]

(In reply to Gabriele Svelto [:gsvelto] from comment #3)

I think I know how I could figure out what's going wrong. Is it possible to pull out of this data a list of all the minidumpSha256Hash values for these pings? Armed with the list I could look up corresponding crashes on Socorro and see what's in the environment. The reason why I'm asking for all of them is that we usually have two order of magnitude less crash reports than crash pings so searching a handful by hand won't cut it.

Hi there, sorry for the delay, I think this query should cover your needs:

https://sql.telemetry.mozilla.org/queries/68985/source

I used this technique to extract the minidumpSha256Hash values: https://wlach.github.io/blog/2019/10/using-bigquery-javascript-udfs-to-analyze-firefox-telemetry-for-fun-profit/

Comment 5

[Gabriele Svelto [:gsvelto]]

Excellent thanks, I got hang of a number of crashes that have corresponding entries on Socorro to analyze, this is one for example.