Closed Bug 1558817 Opened 3 years ago Closed 3 years ago

[10.15] Many single install crashes seem to be missing symbols

Categories

(Tecken :: General, task)

Unspecified
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: marcia, Unassigned)

References

(Blocks 1 open bug)

Details

https://bit.ly/2XGLQXg shows a set of crash reports that seem to be missing symbols. This might have caused me to bucket a bunch of crashes under Bug 1556846 that maybe don't belong there since further down in the stack they seem to have plugin related stacks.

I reprocessed a few but they came out the same. Marco - any ideas?

Flags: needinfo?(mcastelluccio)
Blocks: catalina

Volume of these types of crashes continues, especially since Apple released the first public beta of Catalina yesterday. Here is an example query from nightly crash triage - https://bit.ly/2ZGMdSa.

As soon as we get symbols, I can reprocess in large batches. Just let me know.

We are getting the symbols from the Apple update packages, and there seems to be no update packages for Catalina available yet.
I can only see SafariTechPreviewCatalina and Voice Update - Catarina (not sure why "r" and not "l").

Flags: needinfo?(mcastelluccio)

Hello Marco - Apple just pushed another beta yesterday - can we check to see if there are any symbols? Thanks.

Flags: needinfo?(mcastelluccio)

Still nothing :(

Did we ever get symbols for beta versions of the OS?

Flags: needinfo?(mcastelluccio)

(In reply to Marco Castelluccio [:marco] from comment #5)

Still nothing :(

Did we ever get symbols for beta versions of the OS?

Pretty sure we did for 10.14 - see Bug 1480101. I think we were scraping from our machines and then uploading.

OK, I think we have to follow the same manual process until the update packages are available.

Adding a ni on spohl or haik to see if either of them can help with the symbols here. I think there also may be some kind of issue going on with Python as well on 10.15?

As a side note, do we think there is a possible issue with the crash reporter itself generating multiple crash reports versus some of these people crashing 44 times or more - see https://bit.ly/2YIZzgP for one example? The other day I spotted a macOS 10.15 crash that had 1 install/84 crashes.

Flags: needinfo?(spohl.mozilla.bugs)
Flags: needinfo?(haftandilian)

I've inspected a sample of the crashes and they all look valid. There's usually only a handful of seconds between one crash and the next so they look like auto-submitted content crashes of a process that keeps crashing and keeps being re-spawned to crash again. What's odd is that they all seem to be from the rdd process i.e. the remote data decoder.

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #8)

As a side note, do we think there is a possible issue with the crash reporter itself generating multiple crash reports versus some of these people crashing 44 times or more - see https://bit.ly/2YIZzgP for one example? The other day I spotted a macOS 10.15 crash that had 1 install/84 crashes.

I was just testing the Widevine crash with Firefox started from the terminal and saw the following

dyld: Library not loaded: @executable_path/libmozglue.dylib
  Referenced from: /Applications/Firefox Nightly.app/Contents/MacOS/crashreporter.app/Contents/MacOS/minidump-analyzer
  Reason: image not found

From Console.app, I can see that the minidump-analyzer process is crashing. Could that explain any of the results you're seeing?

I'll file a new bug on this.

Flags: needinfo?(haftandilian)

Filed bug 1564573 "[macOS 10.15] minidump-analyzer crashing on 10.15 while handling crash".

See Also: → 1564573

Just to add an update here - the lions share of single installation crashes I was seeing seem to have stopped after the fix for bug 1556846 landed. We will still need symbols for 10.15. Maybe I should just close this bug out for now? Or leave it open for the scraping symbols part?

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #12)

Just to add an update here - the lions share of single installation crashes I was seeing seem to have stopped after the fix for bug 1556846 landed. We will still need symbols for 10.15. Maybe I should just close this bug out for now? Or leave it open for the scraping symbols part?

I don't mind scraping symbols again. Let me know if it makes sense to do this for a Beta of 10.15, or if we should wait until the official release of 10.15.

Flags: needinfo?(spohl.mozilla.bugs) → needinfo?(mozillamarcia.knous)

I would expect the automatic symbol uploader to work when 10.15 is official (as the updates will be available normally through Apple's update server), until then it would be nice to have the symbols for the Beta.

(In reply to Stephen A Pohl [:spohl] from comment #13)

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #12)

Just to add an update here - the lions share of single installation crashes I was seeing seem to have stopped after the fix for bug 1556846 landed. We will still need symbols for 10.15. Maybe I should just close this bug out for now? Or leave it open for the scraping symbols part?

I don't mind scraping symbols again. Let me know if it makes sense to do this for a Beta of 10.15, or if we should wait until the official release of 10.15.

I think it makes sense. We have done it before in the Mojave timeframe and it seemed to work well.

Flags: needinfo?(mozillamarcia.knous)

I have scraped the symbols for beta 3 (19A501i). I'm now updating to the latest beta and will scrape the symbols there as well. What's the process these days to get these symbols uploaded? I'm happy to upload to a Google Drive again so that someone with access can upload them from there.

Flags: needinfo?(mozillamarcia.knous)

Marco should be able to help here.

Flags: needinfo?(mozillamarcia.knous) → needinfo?(mcastelluccio)

Yep, I have access and can upload them. Share them with me once you've uploaded them.

(In reply to Marco Castelluccio [:marco] from comment #18)

Yep, I have access and can upload them. Share them with me once you've uploaded them.

Shared symbols with you via Google Drive. Thanks!

Flags: needinfo?(mcastelluccio)
Flags: needinfo?(mcastelluccio)

Uploaded, thanks!

Marcia, could you reprocess a few crashes to see if they are symbolicated now?

Flags: needinfo?(mcastelluccio)

(In reply to Marco Castelluccio [:marco] from comment #20)

Uploaded, thanks!

Marcia, could you reprocess a few crashes to see if they are symbolicated now?

I reprocessed a few:

Great, looks like it worked.
Should we do a mass reprocess to move all old crashes to the new signatures?

(In reply to Marco Castelluccio [:marco] from comment #22)

Great, looks like it worked.
Should we do a mass reprocess to move all old crashes to the new signatures?

We could do that. There may not be that many signatures. I believe Will will need specific queries. Adding ni on him for that question - although it appears he is on PTO.

Flags: needinfo?(willkg)

I reprocessed all the crash reports for the last month with "dylib" in them. That's a massive superset, but I don't see a way to know which version of MacOS the crash report came from.

I reprocessed 31,028 crash reports. Now the search returns 30,518. So at least 500 or so changed signatures such that "dylib" doesn't show up.

Did that help?

Flags: needinfo?(willkg)

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #24)

I reprocessed all the crash reports for the last month with "dylib" in them. That's a massive superset, but I don't see a way to know which version of MacOS the crash report came from.

I reprocessed 31,028 crash reports. Now the search returns 30,518. So at least 500 or so changed signatures such that "dylib" doesn't show up.

Did that help?

yes, I see many more bugs that at least have a useful top stack now. Thanks for doing that. I think we can resolve this bug now since the main issues are addressed. And as Marco notes in Comment 3, we should be getting the symbols from the Apple update package at some point.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED

Moving from Socorro product to Tecken product.

Component: Tecken → General
Product: Socorro → Tecken
Version: Trunk → unspecified
You need to log in before you can comment on or make changes to this bug.