1579702 - Nightly crash reports missing install times

Status: NEW

(Toolkit :: Crash Reporting, defect, P2)

Description

[Andrew McCreight [:mccr8]]

A number of us have noticed many Nightly crash reports are missing install times. If nothing else, this breaks de-duplication. Maybe this is just some kind of report we weren't getting at all before (super early in startup or something?) but I figured it was at least worth filing.

Comment 1

[Gabriele Svelto [:gsvelto] (PTO)]

These crashes come from the patch I landed in bug 1566855. They are orphaned minidumps that lack an .extra file which is why they don't have metadata save for the bare minimum attached by the browser so that they won't be rejected by crash-stats. Previously users wouldn't be able to submit those crashes at all. As you noticed they lack the install time field because we don't know which version of Firefox generated them. In theory we shouldn't generate any more of them because the fix for bug 1282776 ensure that all minidumps have an .extra file. So if you encounter them they should be coming from older versions of Firefox. If they don't go away soon then we might have another problem with crash generation.

Note that since bug 1566855 is riding the trains then beta users should have started sending those in the past few days and we'll get more when we reach release.

Comment 2

[Andrew McCreight [:mccr8]]

(In reply to Gabriele Svelto [:gsvelto] from comment #1)

As you noticed they lack the install time field because we don't know which version of Firefox generated them. In theory we shouldn't generate any more of them because the fix for bug 1282776 ensure that all minidumps have an .extra file. So if you encounter them they should be coming from older versions of Firefox. If they don't go away soon then we might have another problem with crash generation.

Thanks for the explanation. FWIW, the ones I've seen do have a build id and version field. For instance: bp-ef9a970f-9ab1-464f-9ef2-0e21f0190906 I've only noticed these in the last few weeks while doing crash triage, while it looks like bug 1282776 landed a while ago. Is that expected?

Comment 3

[Gabriele Svelto [:gsvelto] (PTO)]

(In reply to Andrew McCreight [:mccr8] from comment #2)

Thanks for the explanation. FWIW, the ones I've seen do have a build id and version field. For instance: bp-ef9a970f-9ab1-464f-9ef2-0e21f0190906

Yes, we include the build id because otherwise crash-stats would reject them. However it's the build-id of the version submitting them, not the one that caused them. Unfortunately there's no way to tell the correct build id.

I've only noticed these in the last few weeks while doing crash triage, while it looks like bug 1282776 landed a while ago. Is that expected?

Yes, bug 1282776 prevented further of those from being generated, but users who already had them on their machines couldn't submit them before I landed bug 1566855. So they should have started appearing right after bug 1566855 landed.

Comment 4

[Julian Seward [:jseward]]

I just looked at perhaps eight different crash sigs for the Windows nightlies of 2019 09 12 (Thursday) and none of them had installation times. I also looked at a couple of OSX signatures, and those did have installation times. Is there some Windows-specific breakage here?

Comment 5

[Gabriele Svelto [:gsvelto] (PTO)]

Can you point me to them? I think I can augment the information in synthesized crashes with the crash time and - if it's coherent with the current installation time - add both so that it's possible to tell if the orphaned crash really belongs to the version that submitted it or not. On a side note I find disturbing that we're still getting those crashes, we were possibly missing a lot of them before, more than I had anticipated.

Comment 6

[Julian Seward [:jseward]]

Hmm, now that I look again at them, there are a few that have install times. When I looked yesterday it seemed like all the install times were missing. Anyway, mostly they still are missing. URLs below.

https://crash-stats.mozilla.com/signature/?build_id=20190912215412&release_channel=nightly&product=Firefox&platform=Windows&signature=OOM%20%7C%20unknown%20%7C%20NS_ABORT_OOM%20%7C%20mozilla%3A%3AArenaAllocator%3CT%3E%3A%3AAllocate&date=%3E%3D2019-09-12T00%3A00%3A00.000Z&date=%3C2019-09-16T08%3A25%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

https://crash-stats.mozilla.com/signature/?build_id=20190912215412&release_channel=nightly&product=Firefox&platform=Windows&signature=RtlAnsiStringToUnicodeString%20%7C%20MsgWaitForMultipleObjectsEx%20%7C%20MsgWaitForMultipleObjects%20%7C%20mozilla%3A%3Aipc%3A%3AMessageChannel%3A%3AWaitForSyncNotify%20%7C%20mozilla%3A%3Aipc%3A%3AMessageChannel%3A%3ASend%20%7C%20mozilla%3A%3Aplugins%3A%3APPluginInstanceChild%3A%3ASendShowDirectDXGISurface&date=%3E%3D2019-09-12T00%3A00%3A00.000Z&date=%3C2019-09-16T08%3A25%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

https://crash-stats.mozilla.com/signature/?build_id=20190912215412&release_channel=nightly&product=Firefox&platform=Windows&signature=OOM%20%7C%20unknown%20%7C%20mozalloc_abort%20%7C%20mozalloc_handle_oom%20%7C%20moz_xmalloc%20%7C%20mozilla%3A%3Anet%3A%3AnsStandardURL%3A%3ATemplatedMutator%3CT%3E%3A%3AInit&date=%3E%3D2019-09-12T00%3A00%3A00.000Z&date=%3C2019-09-16T08%3A25%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

https://crash-stats.mozilla.com/signature/?build_id=20190912094122&release_channel=nightly&product=Firefox&platform=Windows&signature=OOM%20%7C%20unknown%20%7C%20NS_ABORT_OOM%20%7C%20nsStringInputStream%3A%3ASerializeInternal%3CT%3E&date=%3E%3D2019-09-12T00%3A00%3A00.000Z&date=%3C2019-09-16T08%3A25%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

https://crash-stats.mozilla.com/signature/?build_id=20190912094122&release_channel=nightly&product=Firefox&platform=Windows&signature=OOM%20%7C%20unknown%20%7C%20js%3A%3AAutoEnterOOMUnsafeRegion%3A%3Acrash%20%7C%20js%3A%3AAutoEnterOOMUnsafeRegion%3A%3Acrash%20%7C%20js%3A%3ATenuringTracer%3A%3Atraverse%3CT%3E&date=%3E%3D2019-09-12T00%3A00%3A00.000Z&date=%3C2019-09-16T08%3A25%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

https://crash-stats.mozilla.com/signature/?build_id=20190912094122&release_channel=nightly&product=Firefox&platform=Windows&signature=mozilla%3A%3Adetail%3A%3AHashTable%3CT%3E%3A%3AchangeTableSize%20%7C%20js%3A%3ASavedStacks%3A%3AgetOrCreateSavedFrame&date=%3E%3D2019-09-12T00%3A00%3A00.000Z&date=%3C2019-09-16T08%3A25%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

https://crash-stats.mozilla.com/signature/?build_id=20190912094122&release_channel=nightly&product=Firefox&platform=Windows&signature=js%3A%3Adetail%3A%3AOrderedHashTable%3CT%3E%3A%3Aremove&date=%3E%3D2019-09-12T00%3A00%3A00.000Z&date=%3C2019-09-16T08%3A25%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

Comment 7

[Marcia Knous [:marcia]]

One macOS that is missing the install time is https://bit.ly/2khMzzC.

Comment 8

[Gabriele Svelto [:gsvelto] (PTO)]

Thanks, I went through all the crashes and here's what I gathered from them:

• They're coming from a handful of machines, some of the OOMs under different signatures are most likely coming from the same one (the CPU info is identical for example)

• A lot of them have the main thread stack messed up pretty badly. Since they all come from two machines there's a chance that's down to flaky hardware which might explain why the .extra file has not being generated: it's possible that Firefox crashed while writing out the crash report

• One is a flash process crash and we had issues with the .extra file not being written already so maybe that's a valid one

• The mac crash is happening very early during content process startup so it should have an .extra file because that's one of the cases I explicitly addressed. I will try to reproduce it

I'm also trying to think how to augment the .extra file. Maybe I could just add the install time field using whatever is in the current Firefox version so that these crashes are deduplicated correctly. After all it's still pretty easy to tell that the .extra file is missing.

Comment 9

[Gabriele Svelto [:gsvelto] (PTO)]

After a lull I've encountered more occurrences of these. All OOM or plugin crashes. I'm especially worried about the latter because the former might be inevitable.