1558817 - [10.15] Many single install crashes seem to be missing symbols

Status: RESOLVED FIXED

(Tecken :: General, task)

Description

[Marcia Knous [:marcia]]

https://bit.ly/2XGLQXg shows a set of crash reports that seem to be missing symbols. This might have caused me to bucket a bunch of crashes under Bug 1556846 that maybe don't belong there since further down in the stack they seem to have plugin related stacks.

I reprocessed a few but they came out the same. Marco - any ideas?

Comment 1

[Marcia Knous [:marcia]]

Volume of these types of crashes continues, especially since Apple released the first public beta of Catalina yesterday. Here is an example query from nightly crash triage - https://bit.ly/2ZGMdSa.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

As soon as we get symbols, I can reprocess in large batches. Just let me know.

Comment 3

[Marco Castelluccio [:marco]]

We are getting the symbols from the Apple update packages, and there seems to be no update packages for Catalina available yet.
I can only see SafariTechPreviewCatalina and Voice Update - Catarina (not sure why "r" and not "l").

Comment 4

[Marcia Knous [:marcia]]

Hello Marco - Apple just pushed another beta yesterday - can we check to see if there are any symbols? Thanks.

Comment 5

[Marco Castelluccio [:marco]]

Still nothing :(

Did we ever get symbols for beta versions of the OS?

Comment 6

[Marcia Knous [:marcia]]

(In reply to Marco Castelluccio [:marco] from comment #5)

Still nothing :(

Did we ever get symbols for beta versions of the OS?

Pretty sure we did for 10.14 - see Bug 1480101. I think we were scraping from our machines and then uploading.

Comment 7

[Marco Castelluccio [:marco]]

OK, I think we have to follow the same manual process until the update packages are available.

Comment 8

[Marcia Knous [:marcia]]

Adding a ni on spohl or haik to see if either of them can help with the symbols here. I think there also may be some kind of issue going on with Python as well on 10.15?

As a side note, do we think there is a possible issue with the crash reporter itself generating multiple crash reports versus some of these people crashing 44 times or more - see https://bit.ly/2YIZzgP for one example? The other day I spotted a macOS 10.15 crash that had 1 install/84 crashes.

Comment 9

[Gabriele Svelto [:gsvelto]]

I've inspected a sample of the crashes and they all look valid. There's usually only a handful of seconds between one crash and the next so they look like auto-submitted content crashes of a process that keeps crashing and keeps being re-spawned to crash again. What's odd is that they all seem to be from the rdd process i.e. the remote data decoder.

Comment 10

[Haik Aftandilian [:haik]]

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #8)

As a side note, do we think there is a possible issue with the crash reporter itself generating multiple crash reports versus some of these people crashing 44 times or more - see https://bit.ly/2YIZzgP for one example? The other day I spotted a macOS 10.15 crash that had 1 install/84 crashes.

I was just testing the Widevine crash with Firefox started from the terminal and saw the following

dyld: Library not loaded: @executable_path/libmozglue.dylib
  Referenced from: /Applications/Firefox Nightly.app/Contents/MacOS/crashreporter.app/Contents/MacOS/minidump-analyzer
  Reason: image not found

From Console.app, I can see that the minidump-analyzer process is crashing. Could that explain any of the results you're seeing?

I'll file a new bug on this.

Comment 11

[Haik Aftandilian [:haik]]

Filed bug 1564573 "[macOS 10.15] minidump-analyzer crashing on 10.15 while handling crash".

Comment 12

[Marcia Knous [:marcia]]

Just to add an update here - the lions share of single installation crashes I was seeing seem to have stopped after the fix for bug 1556846 landed. We will still need symbols for 10.15. Maybe I should just close this bug out for now? Or leave it open for the scraping symbols part?

Comment 13

[Stephen A Pohl [:spohl]]

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #12)

Just to add an update here - the lions share of single installation crashes I was seeing seem to have stopped after the fix for bug 1556846 landed. We will still need symbols for 10.15. Maybe I should just close this bug out for now? Or leave it open for the scraping symbols part?

I don't mind scraping symbols again. Let me know if it makes sense to do this for a Beta of 10.15, or if we should wait until the official release of 10.15.

Comment 14

[Marco Castelluccio [:marco]]

I would expect the automatic symbol uploader to work when 10.15 is official (as the updates will be available normally through Apple's update server), until then it would be nice to have the symbols for the Beta.

Comment 15

[Marcia Knous [:marcia]]

(In reply to Stephen A Pohl [:spohl] from comment #13)

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #12)

Just to add an update here - the lions share of single installation crashes I was seeing seem to have stopped after the fix for bug 1556846 landed. We will still need symbols for 10.15. Maybe I should just close this bug out for now? Or leave it open for the scraping symbols part?

I don't mind scraping symbols again. Let me know if it makes sense to do this for a Beta of 10.15, or if we should wait until the official release of 10.15.

I think it makes sense. We have done it before in the Mojave timeframe and it seemed to work well.

Comment 16

[Stephen A Pohl [:spohl]]

I have scraped the symbols for beta 3 (19A501i). I'm now updating to the latest beta and will scrape the symbols there as well. What's the process these days to get these symbols uploaded? I'm happy to upload to a Google Drive again so that someone with access can upload them from there.

Comment 17

[Marcia Knous [:marcia]]

Marco should be able to help here.

Comment 18

[Marco Castelluccio [:marco]]

Yep, I have access and can upload them. Share them with me once you've uploaded them.

Comment 19

[Stephen A Pohl [:spohl]]

(In reply to Marco Castelluccio [:marco] from comment #18)

Yep, I have access and can upload them. Share them with me once you've uploaded them.

Shared symbols with you via Google Drive. Thanks!

Comment 20

[Marco Castelluccio [:marco]]

Uploaded, thanks!

Marcia, could you reprocess a few crashes to see if they are symbolicated now?

Comment 21

[Marcia Knous [:marcia]]

(In reply to Marco Castelluccio [:marco] from comment #20)

Uploaded, thanks!

Marcia, could you reprocess a few crashes to see if they are symbolicated now?

I reprocessed a few:

• [@ libnetwork.dylib@0x23e2af] became [@ tcp_connection_fallback_handle_change_on_queue ] - Reference crash report https://crash-stats.mozilla.com/report/index/f2202ad9-3e72-4901-be10-dcc380190717
• [@ libsqlite3.dylib@0xefda6] became [@ sqlite3demo_superlock ] - Reference crash report https://crash-stats.mozilla.com/report/index/a005bfe0-a544-4c81-98d2-7e7960190717
• [@ libsystem_kernel.dylib@0x23da6] became [@ __kernelVersionNumber ] - Reference crash report https://crash-stats.mozilla.com/report/index/20ce3492-132f-4064-a118-814090190717

Comment 22

[Marco Castelluccio [:marco]]

Great, looks like it worked.
Should we do a mass reprocess to move all old crashes to the new signatures?

Comment 23

[Marcia Knous [:marcia]]

(In reply to Marco Castelluccio [:marco] from comment #22)

Great, looks like it worked.
Should we do a mass reprocess to move all old crashes to the new signatures?

We could do that. There may not be that many signatures. I believe Will will need specific queries. Adding ni on him for that question - although it appears he is on PTO.

Comment 24

[Will Kahn-Greene [:willkg] ET needinfo? me]

I reprocessed all the crash reports for the last month with "dylib" in them. That's a massive superset, but I don't see a way to know which version of MacOS the crash report came from.

I reprocessed 31,028 crash reports. Now the search returns 30,518. So at least 500 or so changed signatures such that "dylib" doesn't show up.

Did that help?

Comment 25

[Marcia Knous [:marcia]]

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #24)

I reprocessed all the crash reports for the last month with "dylib" in them. That's a massive superset, but I don't see a way to know which version of MacOS the crash report came from.

I reprocessed 31,028 crash reports. Now the search returns 30,518. So at least 500 or so changed signatures such that "dylib" doesn't show up.

Did that help?

yes, I see many more bugs that at least have a useful top stack now. Thanks for doing that. I think we can resolve this bug now since the main issues are addressed. And as Marco notes in Comment 3, we should be getting the symbols from the Apple update package at some point.

Comment 26

[Will Kahn-Greene [:willkg] ET needinfo? me]

Moving from Socorro product to Tecken product.