Closed Bug 1588843 Opened 5 years ago Closed 5 years ago

[10.15] Need symbols for 10.15 to be uploaded

Categories

(Tecken :: General, task)

Product:
Tecken
Component:
General
Platform:
Unspecified
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: smichaud, Unassigned)

References

Details

macOS Catalina (10.15) has just been released. We need symbols uploaded to ensure that native system calls are symbolicated in crash stacks (instead of just rendered as an offset in a system library or framework).

[Tracking Requested - why for this release]:

Since Catalina has already been released, we should be able to symbolicate Catalina system calls as soon as possible. It may be too late to track this for FF 70, in which case we should track it for FF 71.

tracking-firefox70: --- → ?
See Also: → 1480101

Marco: Bug 1587904 got closed out. Did symbols for 10.15 get uploaded?

Flags: needinfo?(mcastelluccio)

Bug 1587904 only dealt with Firefox symbols, not macOS ones.

RC2 is tomorrow, if there's something we need to land for 70, maybe Marco and Pascal (or Julien) can take care of it so that we'll be ready for the RC2 build Wed. morning Pacific.

status-firefox70: --- → affected
tracking-firefox70: ? → +
Flags: needinfo?(pascalc)

Once symbols are uploaded, I can reprocess crash reports in socorro. I'll keep tabs on this.

I noticed that Marco's job has failed twice - see https://github.com/marco-c/breakpad-mac-update-symbols/blob/master/fetch-task.json. His job is scraping symbols from Apple system updates. In the past when this failed we have had to revert to manually scraping symbols (I think).

Marcia, is there a record of the failed tasks that I can look at to learn more about the failures? I wouldn't be surprised if the script needs to be updated for Catalina. I can test it locally if need be, and try to update it myself.

Flags: needinfo?(mozillamarcia.knous)

Until two days ago, it was not failing but it was also not downloading any symbols for 10.15.
Since two days ago, it started downloading symbols for 10.15 (yay!) and it started failing (boo!), so it's probably related.

The failure though doesn't seem to be happening while trying to dump 10.15 symbols, but 10.13. See https://tools.taskcluster.net/groups/cihDfh_IQTybeissRzPsHA/tasks/IJKiEH5QRlO6DDIeETqJVw/runs/0/logs/public%2Flogs%2Flive.log:

2019-10-16 04:08:41,557 - root - INFO - Dumping symbols from package: /opt/data-reposado/html/content/downloads/30/00/041-90784-A_10MABRT921/cb5f3z5st81lc7dtogbtns139k4ikgxq8n/macOSUpdCombo10.13.2Auto.pkg
Error while extracting archive:(Payload): archived-checksum message digest hash values do not match (No such file or directory)
Traceback (most recent call last):
  File "/home/worker/breakpad-mac-update-symbols/PackageSymbolDumper.py", line 271, in <module>
    main()
  File "/home/worker/breakpad-mac-update-symbols/PackageSymbolDumper.py", line 267, in main
    process_packages(finder, args.to, args.tracking_file, args.dump_syms)
  File "/home/worker/breakpad-mac-update-symbols/PackageSymbolDumper.py", line 236, in process_packages
    dump_symbols_from_package(executor, dump_syms, pkg, to)
  File "/home/worker/breakpad-mac-update-symbols/PackageSymbolDumper.py", line 199, in dump_symbols_from_package
    expand_pkg(pkg, temp_dir)
  File "/home/worker/breakpad-mac-update-symbols/PackageSymbolDumper.py", line 44, in expand_pkg
    subprocess.check_call('cd "{dest}" && xar -x -f "{src}"'.format(src=pkg_path, dest=out_path), shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'cd "/tmp/tmp5STU32" && xar -x -f "/opt/data-reposado/html/content/downloads/30/00/041-90784-A_10MABRT921/cb5f3z5st81lc7dtogbtns139k4ikgxq8n/macOSUpdCombo10.13.2Auto.pkg"' returned non-zero exit status 1

I haven't had time to look into it yet as I was at a training, but I could probably look into it very soon.
The source code is at https://github.com/marco-c/breakpad-mac-update-symbols.

(In reply to Liz Henry (:lizzard) from comment #4)

RC2 is tomorrow, if there's something we need to land for 70, maybe Marco and Pascal (or Julien) can take care of it so that we'll be ready for the RC2 build Wed. morning Pacific.

Luckily we don't need to land anything as this is external. I guess we still want to track as it is important.

(In reply to comment #9)

This a real headscratcher. The file cb5f3z5st81lc7dtogbtns139k4ikgxq8n/macOSUpdCombo10.13.2Auto.pkg seems to be missing on the attempt to dump symbols from it (the error is "No such file or directory"). But above the file is recorded as having been successfully downloaded. Might the file not be missing, but instead corrupt? If so, it would seem it was already corrupt on Apple's http://swcdn.apple.com/ server. I wonder what would happen if you just skipped this file.

(In reply to Steven Michaud [:smichaud] (Retired) from comment #10)

(In reply to comment #9)

This a real headscratcher. The file cb5f3z5st81lc7dtogbtns139k4ikgxq8n/macOSUpdCombo10.13.2Auto.pkg seems to be missing on the attempt to dump symbols from it (the error is "No such file or directory"). But above the file is recorded as having been successfully downloaded. Might the file not be missing, but instead corrupt?

I've downloaded it locally, and it seems to be indeed corrupt.

If so, it would seem it was already corrupt on Apple's http://swcdn.apple.com/ server. I wonder what would happen if you just skipped this file.

I've just done this a couple hours ago, and I've retriggered the job (which is still running, https://tools.taskcluster.net/groups/UTS7Y58KTAqjLKN1ms8cVQ/tasks/G-HLkHDCQ9eLb62f2C5-wA/runs/0/artifacts). Let's see how it goes.

The latest run was successful, with only macOSUpdCombo10.13.2Auto.pkg being corrupted: https://tools.taskcluster.net/groups/O4BXmJGLTYmPgARtFIv30g.
Quite a few symbols were uploaded (https://symbols.mozilla.org/uploads/upload/96244).

Will, could you reprocess crashes? Let's start with a few just to make sure the job uploaded the right symbols, and if it works then we can reprocess a larger set.

Flags: needinfo?(willkg)
Flags: needinfo?(pascalc)
Flags: needinfo?(mcastelluccio)

I reprocessed the following crash reports because they seemed like candidates to me:

The signatures for all of those stayed the same.

This crash has some frames that weren't symbolicated before, but now are:

I didn't notice newly symbolicated frames in the other three crash reports.

Is that what we'd expect here? Are there other crash reports we should be testing reprocessing with?

Flags: needinfo?(willkg)

I don't know much about how Reposado works (used by Marco's script), but it seems to download only updates. As far as I can tell, only one update for macOS 10.15 has yet been released (which didn't bump the minor version but updated the build number to 19A602). If we have no other source of symbols, we may need to wait until 10.15.1 is released to have decent coverage of Catalina symbols.

Here are the two 10.15 updates I could find in the log for https://tools.taskcluster.net/groups/O4BXmJGLTYmPgARtFIv30g (from comment #12 above). Neither is very large.

macOSUpd10.15.RecoveryHDUpdate.pkg (500626708 bytes)
macOSUpd10.15.pkg (132059514 bytes)

bp-fcec295b-81dc-4900-9891-59c3a0191011

This seems to be corrupt, or to contain entries pointing to the stack (in which case it's a stack overflow).

(In reply to Steven Michaud [:smichaud] (Retired) from comment #14)

I don't know much about how Reposado works (used by Marco's script), but it seems to download only updates. As far as I can tell, only one update for macOS 10.15 has yet been released (which didn't bump the minor version but updated the build number to 19A602). If we have no other source of symbols, we may need to wait until 10.15.1 is released to have decent coverage of Catalina symbols.

Yes, that's exactly how it works. I was assuming there would be update packages from 10.14 -> 10.15 too, isn't that the case?
If that isn't the case, we probably need to upload them manually like we did for the beta symbols.

I was assuming there would be update packages from 10.14 -> 10.15 too

Major version updates work differently. Basically you download an installer which contains a bootable image which boots and installs the OS. Minor updates use *.pkg files which contain just the files being updated. Reposado appears to work only with *.pkg files.

If you do upload symbols manually, please include those for the following files (if possible):

All dylibs in /usr/lib and /usr/lib/system
All frameworks in /System/Library/Frameworks and /System/Library/PrivateFrameworks

Having complete (or near complete) coverage of system calls is enormously helpful for someone who (like me) works at the interface between the app and the OS, and often deals with Apple bugs.

Another thing:

A lot of system libraries on recent versions of macOS are universal files that have binaries for two different 64-bit architectures -- x86_64 and x86_64h (the 'h' stands for Haswell). Each has its own set of symbols. /usr/lib/libobjc.dylib on 10.15 and 10.14 is an example. I assume our scripts and Socorro can deal with this. Am I right?

Many of the non-symbolicated crash stacks that I've looked at recently (particularly for 10.15) contain addresses that must be in the Haswell binaries (the addresses only make sense there).

(Following up comment #18)

https://github.com/firefox-devtools/Gecko-Profiler-Addon/issues/29 would seem to indicate that we know how to get symbols for x86_64h (Haswell) binaries. But are we doing this in Socorro?

(In reply to Steven Michaud [:smichaud] (Retired) from comment #18)

A lot of system libraries on recent versions of macOS are universal files that have binaries for two different 64-bit architectures -- x86_64 and x86_64h (the 'h' stands for Haswell). Each has its own set of symbols. /usr/lib/libobjc.dylib on 10.15 and 10.14 is an example. I assume our scripts and Socorro can deal with this. Am I right?

I think my script is, https://github.com/luser/breakpad-scrape-system-symbols/blob/a6ac5ce438135411f1434329d78becf7fe033d7a/scrapesymbols/gathersymbols.py#L150-L151, where get_archs is https://github.com/luser/breakpad-scrape-system-symbols/blob/master/scrapesymbols/gathersymbols.py#L39-L46. I assume get_archs is returning the correct architectures, but I can't test as I don't have a Mac.

(In reply to Steven Michaud [:smichaud] (Retired) from comment #16)

I was assuming there would be update packages from 10.14 -> 10.15 too

Major version updates work differently. Basically you download an installer which contains a bootable image which boots and installs the OS. Minor updates use *.pkg files which contain just the files being updated. Reposado appears to work only with *.pkg files.

(In reply to Steven Michaud [:smichaud] (Retired) from comment #17)

If you do upload symbols manually, please include those for the following files (if possible):

All dylibs in /usr/lib and /usr/lib/system
All frameworks in /System/Library/Frameworks and /System/Library/PrivateFrameworks

Having complete (or near complete) coverage of system calls is enormously helpful for someone who (like me) works at the interface between the app and the OS, and often deals with Apple bugs.

Stephen, could you grab the symbols and send them to me like you did for bug 1558817?

Flags: needinfo?(spohl.mozilla.bugs)

Stephen, I'm not sure what "manually" means to you. Do you have a script to extract the symbols, or will you literally have to run a command on each dylib or framework? If the latter, I'm sure we can work out a list of the most important dylibs and frameworks. How about the following, to start with?

/usr/lib/libSystem.dylib
/usr/lib/libc++abi.dylib
/usr/lib/libc.dylib
/usr/lib/libffi.dylib
/usr/lib/libobjc.dylib
/usr/lib/libstdc++.dylib
/usr/lib/system/libdyld.dylib
/usr/lib/system/libsystem_*.dylib
/System/Library/Frameworks/AppKit.framework/AppKit
/System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/HIToolbox
/System/Library/Frameworks/Cocoa.framework/Cocoa
/System/Library/Frameworks/CoreAudio.framework/CoreAudio
/System/Library/Frameworks/CoreData.framework/CoreData
/System/Library/Frameworks/CoreDisplay.framework/CoreDisplay
/System/Library/Frameworks/CoreGraphics.framework/CoreGraphics
/System/Library/Frameworks/CoreImage.framework/CoreImage
/System/Library/Frameworks/CoreMedia.framework/CoreMedia
/System/Library/Frameworks/CoreText.framework/CoreText
/System/Library/Frameworks/CoreVideo.framework/CoreVideo
/System/Library/Frameworks/Foundation.framework/Foundation
/System/Library/Frameworks/ImageIO.framework/ImageIO
/System/Library/Frameworks/OpenCL.framework/OpenCL
/System/Library/Frameworks/OpenGL.framework/OpenGL
/System/Library/Frameworks/QuartzCore.framework/QuartzCore
/System/Library/PrivateFrameworks/FinderKit.framework/FinderKit
/System/Library/PrivateFrameworks/SkyLight.framework/SkyLight

(In reply to comment #20)

I assume get_archs is returning the correct architectures, but I can't test as I don't have a Mac.

I just tried lipo -info /usr/lib/libobjc.dylib on 10.15, and it does return both 64-bit architectures:

    Architectures in the fat file: libobjc.dylib are: x86_64 x86_64h i386

(Following up comment #21)

Additions to list, from the results of otool -Lv XUL:

/System/Library/Frameworks/AudioToolbox.framework/AudioToolbox
/System/Library/Frameworks/ExceptionHandling.framework/ExceptionHandling
/System/Library/Frameworks/IOKit.framework/IOKit
/System/Library/Frameworks/VideoToolbox.framework/VideoToolbox
/System/Library/Frameworks/AVFoundation.framework/AVFoundation
/System/Library/Frameworks/LocalAuthentication.framework/LocalAuthentication
/System/Library/Frameworks/Security.framework/Security
/System/Library/Frameworks/SystemConfiguration.framework/SystemConfiguration
/System/Library/PrivateFrameworks/CoreUI.framework/CoreUI
/System/Library/PrivateFrameworks/CoreSymbolication.framework/CoreSymbolication
/usr/lib/libcups.dylib
/System/Library/Frameworks/CoreLocation.framework/CoreLocation
/System/Library/Frameworks/AudioUnit.framework/AudioUnit
/System/Library/Frameworks/AddressBook.framework/AddressBook

I didn't know we were using CoreSymbolication :-)

(Following up comment #23)

And yet some more:

/System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/AE
/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/LaunchServices
/System/Library/Frameworks/ApplicationServices.framework/Frameworks/ATS.framework/ATS
/System/Library/Frameworks/ApplicationServices.framework/Frameworks/ATSUI.framework/ATSUI
/System/Library/Frameworks/ApplicationServices.framework/Frameworks/HIServices.framework/HIServices
/System/Library/Frameworks/ApplicationServices.framework/Frameworks/PrintCore.framework/PrintCore

There are steps in bug 1480101 comment 6 and subsequent comments. I will forward the email with the script to you. I will not be able to scrape these symbols myself for a while and appreciate anyone else scraping/uploading them. Thank you!

Flags: needinfo?(spohl.mozilla.bugs)

Stephen, I've got a couple of questions about the gathersymbols.py script:

Does it identify which OS version the symbols are coming from, and if so how? We wouldn't want to get this wrong on the symbol server.

It looks as if only the top level directories in SYSTEM_DIRS are processed. Is this correct? If so, one could easily fix the problem by adding selected subdirectories to SYSTEM_DIRS (for example /usr/lib/system in addition to /usr/lib).

Once I've got answers to these two questions, I'll take a crack at scraping symbols on my 10.15 system. I'm not a Mozilla employee, though, so I may have trouble getting the required permissions.

Flags: needinfo?(spohl.mozilla.bugs)

The website https://crash-analysis.mozilla.com/ seems to have disappeared, or been moved. Anyone have any information?

This website is used by the gathersymbols.py script.

The crash-analysis hasn't been around in years. The Symbols server (https://symbols.mozilla.org/) tracks missing symbols now. I don't know offhand how to update this script to use the Symbols server to fetch missing symbols. Maybe it's better to just pass in `--all``?

(In reply to Steven Michaud [:smichaud] (Retired) from comment #27)

The website https://crash-analysis.mozilla.com/ seems to have disappeared, or been moved. Anyone have any information?

This website is used by the gathersymbols.py script.

Steven, this is called out and discussed in the comment I linked to. Bug 1480101 comment 6.

Flags: needinfo?(spohl.mozilla.bugs)

Kinda looks like I'm going to have to write my own script to scrape symbols. Before I do that I'll try to understand how to interact with the symbol server by reading the docs on https://symbols.mozilla.org/.

Will, do you have any idea how to handle the problem of making sure the symbol server knows which macOS version the symbols I upload come from?

It may take me some time to get to the bottom of this. Right now I'm also trying to update https://github.com/steven-michaud/HookCase to be compatible with macOS 10.15. I think that takes priority, and I estimate it'll take me another week or so. Then it will take me at least a few days to write my own script (scavenging as much as I can from Ted Mielczarek's script). So it may be two or three weeks before we have complete coverage of Catalina symbols. In the meantime we already have a fair amount of coverage from macOSUpd10.15.pkg. I looked at it and found that it contains many of the most important frameworks, though unfortunately none of the dylibs in /usr/lib or /usr/lib/system.

I'm happy for someone else to take responsibility for this. But for the moment (and in the absence of Ted Mielczarek) it looks like I'm the best qualified to deal with it.

(In reply to comment #29)

Thanks. I'd missed that. But I still think I should write my own script, after I learn how to deal with the problem of making sure the symbol server knows that my symbols come from macOS 10.15.0.

Here are the files in macOSUpd10.15.pkg which are also in my lists from comment #21, comment #23 and comment #24 above (including CoreFoundation, which wasn't but should have been):

/System/Library/Frameworks/AVFoundation.framework/AVFoundation
/System/Library/Frameworks/AppKit.framework/AppKit
/System/Library/Frameworks/AudioToolbox.framework/AudioToolbox
/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation
/System/Library/Frameworks/CoreGraphics.framework/CoreGraphics
/System/Library/Frameworks/Foundation.framework/Foundation
/System/Library/Frameworks/IOKit.framework/IOKit

Flags: needinfo?(willkg)

I'll also ask Marco:

Do you have any idea how to handle the problem of making sure the symbol server knows which macOS version the symbols I upload come from?

Flags: needinfo?(mcastelluccio)

It doesn't seem worth it writing your own script, if we have to wait three weeks for scraping symbols. Why not scraping them now with Stephen's script, and then, if you want, write your own script for the next time we need this?
I don't want to force you though, Calixte can do it on his machine if you can't.

(In reply to Steven Michaud [:smichaud] (Retired) from comment #33)

Do you have any idea how to handle the problem of making sure the symbol server knows which macOS version the symbols I upload come from?

The symbol server doesn't have to know, it'll match the symbols from their ID.

Flags: needinfo?(willkg)
Flags: needinfo?(mcastelluccio)

it'll match the symbols from their ID.

What's the "ID"? :-)

(In reply to Steven Michaud [:smichaud] (Retired) from comment #35)

it'll match the symbols from their ID.

What's the "ID"? :-)

Is it the debug_id in gathersymbols.py?

Flags: needinfo?(mcastelluccio)

The Symbols server doesn't know where the symbols came from or operating system version. Pretty much everything it knows is in the path.

The debug id is generated when the compiled file is generated. That way the extracted symbols are linked to the binary they were extracted from. I haven't read gathersymbols.py, but I think you're right in that it is the debug id.

The debug id is generated when the compiled file is generated.

You mean when dump_syms generates its output?

When I run dump_syms -a x86_64 /System/Library/Frameworks/AppKit.framework/AppKit, the first line of output is:

    MODULE mac x86_64 EFB74848E23F3FC3B167BA1F960996CC0 AppKit

I take it the "debug id" is EFB74848E23F3FC3B167BA1F960996CC0. Am I right?

The succeeding lines (after the first) are symbols and their addresses.

Note that the debug id is EFB74848E23F3FC3B167BA1F960996CC0 for AppKit on the latest version of macOS 10.14.6. On the latest version of 10.15 it's 1F4FBEF8F3AC31C2BF04381BC72E36D10.

For those of you who are paying close attention :-)

As best I can tell, some effort was put into ensuring that the debug id is unique per file. Though I still haven't been able to find out how it's generated on the Mac (on Windows and Linux it's fairly straightforward).

I've been using the latest version of Google's dump_syms from https://github.com/google/breakpad/tree/master/src/tools/mac/dump_syms. It wouldn't build on 10.15. But I managed to build it on 10.14.6, then copy it over. I figured it was worth my time to get the latest and greatest. If anyone knows there's a problem with this, please speak up.

So it now looks like I should be able to run Ted Mielczarek's gathersymbols.py script sometime tomorrow or the day after. I'll alter it to include missing directories, like /usr/lib/system. I'll wait until Monday to try uploading the results to the symbol server. I'll need to get the appropriate permissions, so I'll need for Mozilla employees to be around to help me, if need be.

I suggest that you run the script as-is. Let's try not to "fix things that ain't broke" since it risks causing other problems, for example with the symbol server. Once you have the output we'll make sure that you can upload it to a Google Drive like I've been doing. It will be uploaded to the symbol server from there.

I'm happy to gather symbols going forward. At the risk of sounding harsh, gathering these symbols is a straightforward thing and shouldn't have required the numerous need-info's here. Curiosity about the scripts and how they work is great, but this bug here was a simple request to run an existing script using existing steps and get these symbols uploaded.

My system is about to be updated to the latest seed and I can gather these symbols myself, if still necessary.

Flags: needinfo?(mcastelluccio)

(In reply to comment #42)

The symbol server has clearly changed since Ted wrote his script. How was I to know that it hadn't changed since you last scraped symbols manually, a year ago? I just wanted to make sure I "did no harm". To be sure of that I needed to ask some simple questions. As it turned out I was mostly able to piece together the answers myself, which showed me that it's very unlikely for the symbol server to have changed in such a way that it couldn't deal with the straight output of dump_syms.

I'm happy to gather symbols going forward.

Please do. I have no desire to have access to the symbol server.

I strongly suggest that the following directories be added to SYSTEM_DIRS in gathersymbols.py:

/usr/lib/system
/System/Library/Application.services/Frameworks
/System/Library/Carbon.framework/Frameworks
/System/Library/CoreServices.framework/Frameworks

I probably shouldn't do that on my own authority. But without them, important symbols will be missing until they're gradually filled in by Apple updates (via Reposado).

Just to make it clear, I don't plan to upload any symbols to the symbol server.

I'll keep playing with Ted's script, to learn how it works. But that's now as far as I'm going to take it.

Steven, there's no need to be offended. We just need these symbols quickly, improvements to the scripts are very welcome, but at a later stage.
If you can scrape them and send them to me, I'll upload them to the symbol server. Even Stephen doesn't have access to the symbol server, he just sends the symbols to me and I upload them.
If you don't want to anymore, I'll ask somebody else or wait for Stephen.

OK, I'll give it a try. I should have the scraped symbols ready today or tomorrow. You'll need to tell me how I should get them to you. Most convenient would be a server URL to which I can upload them. They'll be far too big for email.

Thanks! You can use https://send.firefox.com/ or any other upload service of your choice.

I just ran gathersymbols.py and it worked fine. It even recursed subdirectories, so it won't be necessary to add the directories to SYSTEM_DIR that I listed in comment #42. I checked, and found that symbols.zip contains symbols for all architectures (including x86_64h). I just finished uploading the file.

https://send.firefox.com/download/cce51b92cafcc236/#3oUPpWD4_4EUSmX9ZOw4vw

(In reply to Steven Michaud [:smichaud] (Retired) from comment #48)

I just ran gathersymbols.py and it worked fine. It even recursed subdirectories, so it won't be necessary to add the directories to SYSTEM_DIR that I listed in comment #42. I checked, and found that symbols.zip contains symbols for all architectures (including x86_64h). I just finished uploading the file.

https://send.firefox.com/download/cce51b92cafcc236/#3oUPpWD4_4EUSmX9ZOw4vw

Thanks! Could you re-upload it? (The link has expired)
Maybe set a higher limit of downloads, since it's not a sensitive file.

Maybe set a higher limit of downloads

Sigh, I'm having trouble setting the limit to more than one download. If I try, send.firefox.com insists I log in, and I don't have an account. Any other upload service you can suggest?

Or better yet, tell me which password I'd need to use at send.mozilla.com. I tried both my bugzilla and LDAP passwords. Neither worked.

Do I need a sync account? I've quite deliberately chosen not to have one. (I'd prefer not to sync all my bookmarks and tabs to each machine.)

Turns out I already had a sync account without realizing it. I reset the password and uploaded symbols.zip again, this time with a limit of 20 downloads.

https://send.firefox.com/download/7c7e39a6c2c6cb1b/#if8W3pImMhqiXJqEmTid-g

Thanks! I've uploaded them to the symbol server.

Will, could you, as before, reprocess crashes? Let's start again with a few just to make sure the right symbols were uploaded, and if it works then we can reprocess a larger set.

Flags: needinfo?(willkg)

For what it's worth, here's another symbol.zip file. Even with all the symbols gathersymbols.py collects, we're still missing some important ones -- those from user-mode drivers in /System/Library/Extensions. Examples on 10.15 include AMDRadeonX4000GLDriver, AppleIntelSKLGraphicsGLDriver and GeForceGLDriver (all of which show up in our crash statistics unsymbolicated). Most of the macho files in /System/Library/Extensions are kernel-mode files (for which we'll never need symbols, because they don't run in user mode). But there are also some user-mode macho files, all of which seem to be in "extensions" ending in *.bundle. As best I can tell, all the macho files in a "bundle" are user-mode.

I gathered these symbols using the following command:

    OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES python /usr/local/src/breakpad-scrape-system-symbols/breakpad-scrape-system-symbols-master/scrapesymbols/gathersymbols.py -v --all /usr/local/bin/dump_syms /System/Library/Extensions/*.bundle

https://send.firefox.com/download/7342ee22ee824f3e/#xfsbKVREA5cf_kZXejqVjA

This time I set the download limit to 100.

I reprocessed the reports from comment 13 and there are now symbols for some of them:

These didn't change:

bp-3134c829-aeb0-4d62-a2bd-636a30191014
bp-558f1f0a-65c3-4855-a50d-3d25d0191011 (from comment #13)

These two are from a pre-release build of Catalina (19A582a). The release was build 19A583.

bp-fcec295b-81dc-4900-9891-59c3a0191011 (from comment #13)

As I said above in comment #14, this one is either corrupt or contains addresses pointing to the stack. Having symbols shouldn't have made any difference.

All the successfully symbolicated crash stacks are from build 19A583.

It's puzzling that symbolication of system calls should have completely failed on build 19A582a, which must have been one of the GM builds, and very close to build 19A583. But there you have it.

Here are a bunch of crashes on the 19A602 build (the current 10.15.0 build) reported in the last two hours:

https://crash-stats.mozilla.com/search/?platform_version=~19A602&date=%3E%3D2019-10-21T14%3A17%3A00.000Z&date=%3C2019-10-21T16%3A17%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Aside from addresses that (I think) point to the stack, and with two exceptions, all the system calls appear to be properly symbolicated.

The exceptions are libpkcs11-dnietif.so (bug 1533422) and FlashPlayer-10.6. Both are third party modules.

There's also one puzzling report where system calls are symbolicated but XUL calls aren't:
bp-c977fa06-ffac-459d-8873-2bb690191021

Aside from addresses that (I think) point to the stack

Even it they don't point to the stack, the module information seems to be missing. You need to know which module an address is in before you can symbolicate it.

Catalina 10.15.1 was just released. Did its update package get downloaded by Reposado?

The combo update hasn't yet appeared at https://support.apple.com/downloads/macos. I'm not sure when it will, or even that it will. It's conceivable that OS updates will now only be available via the Software Update preference panel. I'll keep an eye on this, and check back in a week or so.

The combo update hasn't yet appeared at https://support.apple.com/downloads/macos. I'm not sure when it will, or even that it will. It's conceivable that OS updates will now only be available via the Software Update preference panel. I'll keep an eye on this, and check back in a week or so.

Never mind. It just appeared:

https://support.apple.com/kb/DL2022?viewlocale=en_US&locale=en_US

Symbols for macOS 10.15.1 have started to show up in crash stacks at https://crash-stats.mozilla.com/. So the macOSUpd10.15.1.dmg update must have been downloaded and processed by Marco's breakpad-mac-update-symbols scripts. But when I re-ran gathersymbols.py on the computer where I ran it originally (since updated from 10.15 to 10.15.1), I noticed that some symbols were missing. A significant number of files were missing only their x86_64h symbols (the x86_64 symbols were already present). So I think there must be something wrong with how the breakpad-mac-update-symbols scripts process x86_64h (Haswell) symbols.

Here's a file containing the missing symbols. I've set it to 100 downloads and 7 days (the maximum values possible).

https://send.firefox.com/download/d497b65f5e2ba01f/#jYSEEG5qhiXmRy_KUhJRkA

Flags: needinfo?(mcastelluccio)

For what it's worth, I grepped the entire breakpad-mac-update-symbols distro on "x86_64" and "x86_64h" and got no hits. Maybe the problem isn't in these scripts but somewhere else. Are there any scripts that prune symbols on the symbol server?

I think I may have it. I got no hits in any script file, but I did get one (for "x86_64") on the ELF-based 'lipo' executable. "strings -" on lipo shows a bunch of architectures, including x86_64, but not including x86_64h. The lipo binary is (I assume) the one that runs on AWS's Linux machines (the ones Mozilla uses to run the reposado script). It must not be able to find the x86_64h architecture in a "fat" binary.

In comment #22 I ran the macOS 'lipo', not quite realizing what was up.

I found a currently maintained port of Apple's cctools (including lipo) for Linux and other non-Apple OSs. It explicitly includes x86_64h among its "supported target architectures".

https://github.com/tpoechtrager/cctools-port

Does anyone know where the lipo binary in https://github.com/marco-c/breakpad-mac-update-symbols comes from? If there isn't an updated version, maybe Mozilla should switch to using Thomas Pöchtrager's version.

For the record, here are all the Apple cctools source distros on opensource.apple.com:

https://opensource.apple.com/source/cctools/

See Also: → 1562953

(Following up comment #65)

I've downloaded the 'lipo' binary from https://github.com/marco-c/breakpad-mac-update-symbols to a Linux system and confirmed that it doesn't see the x86_64h architecture in a 3-architecture "fat" binary. I tested (using lipo -info) on the /usr/lib/libobjc.A.dylib file downloaded from a macOS 10.12 (Sierra) system. The Linux system's own 'file' command does see the x86_64h architecture.

At some point I'll build a "new" lipo binary and generate a pull request on it for https://github.com/marco-c/breakpad-mac-update-symbols.

Let's close this, as it served its purpose, and open a new bug to update lipo.

Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(mcastelluccio)
Resolution: --- → FIXED
See Also: → 1593460

Haswell symbolication still isn't working, even though Haswell symbols are no longer missing from the symbol server. I'm pursuing this further at bug 1371390.

Moving from Socorro product to Tecken product.

status-firefox70: fixed → ---
tracking-firefox70: + → ---
Component: Tecken → General
Product: Socorro → Tecken
You need to log in before you can comment on or make changes to this bug.