1480101 - [10.14] Need symbols for 10.14 to be uploaded

Status: RESOLVED FIXED

(Tecken :: General, task)

Description

[Marcia Knous [:marcia]]

[Tracking Requested - why for this release]: Nominating for tracking since we are tracking 10.14 shipping during the timeframe of our 63 release.

Ted and I have been talking about getting symbols for 10.14 in Bug 1467230 using the instructions here: https://github.com/luser/breakpad-scrape-system-symbols/.

Although I have a 10.14 install, I haven't had any success being able to use scrapesymbols on my machine even with a tweak he emailed me.

Since we need these uploaded prior to 10.14 shipping, filing this to not lose track of it.

Comment 1

[Marcia Knous [:marcia]]

Hello Stephen - Do you think you can try to run the script if you get a chance? If so, I can email you the other tweak that Ted sent to me via email. Thanks.

Comment 2

[Stephen A Pohl [:spohl]]

Yes, I can give this a shot.

Comment 3

[Marcia Knous [:marcia]]

(In reply to Stephen A Pohl [:spohl] from comment #2)
> Yes, I can give this a shot.

Hello Stephen - Just checking in to see if you have had time to give this a try?

Comment 4

[Stephen A Pohl [:spohl]]

My apologies, I forgot to leave the n-i set and this fell off my radar. Will give this a shot before end of day tomorrow.

Comment 5

[Stephen A Pohl [:spohl]]

Quick update: I have been able to get the script to run on macOS 10.14. However, there appears to be some kind of change to the system libraries that dump_syms is unable to handle. I may have to build dump_syms from source with `configure && make` in the breakpad repo to debug further. Leaving n-i set until I have more info.

Comment 6

[Stephen A Pohl [:spohl]]

I have been able to get the script to run and scrape the symbols. However, because crash-analysis.mozilla.com is no longer accessible, I had to scrape all symbols instead of just the missing ones. The symbols.zip file is too large to be attached to this bug. Marcia, are you set up to upload the symbols? Where should I upload the symbols.zip file? I have not tried running uploadsymbols.py as it requires an authentication token.

A few things that were important:

1. In setup.sh, I commented out line 6:
> # wget https://people.mozilla.org/~tmielczarek/`uname -s`/dump_syms
2. Copy dump_syms into the breakpad-scrape-system-symbols-master directory.
3. When running gathersymbols.py, make sure to pass the full path to dump_syms. Otherwise, the script will fail with a large number of "file not found" errors.
4. Apple appears to have changed the way fork() works on 10.13+ and this can cause the following crashes with Python fork():
> objc[6763]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
To work around this, set the following env variable: OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
The following command ended up working successfully:
> $ OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES python scrapesymbols/gathersymbols.py -v --all /Users/spohl/Documents/breakpad-scrape-system-symbols-master/dump_syms

Comment 7

[Marcia Knous [:marcia]]

Stephen: I don't have permissions on https://symbols.mozilla.org/ at the moment. It looks as a bug can be filed to request permissions or maybe Will can help? You could probably upload the file to Google Drive and then share it.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

Peter: Can you help Marcia?

Comment 9

[Peter Bengtsson [:peterbe]]

Hi Stephen and Marcia,
To upload symbols on symbols.mozilla.org you need too 

a) sign in at least once so your email can be found
b) be vouched by someone, e.g. Ted

Once you have permission to upload symbols you can do it in one of three ways:

1) Use the REST API [0]
2) Upload it as a regular file upload (the .zip) through the web frontend
3) Upload the symbols.zip somewhere such that you have a public URL to it and then specify this URL in the REST API or the web frontend.  


[0] https://tecken.readthedocs.io/en/latest/upload.html#how-it-works

Comment 10

[(not currently active) Ted Mielczarek]

spohl: if you can provide me with the symbols.zip somehow I will upload it for you. Perhaps on Google Drive?

If we can work around any of those issues in the code in the GitHub repo I'd be happy to merge a PR.

Comment 11

[Stephen A Pohl [:spohl]]

(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #10)
> spohl: if you can provide me with the symbols.zip somehow I will upload it
> for you. Perhaps on Google Drive?

I just shared this with your LDAP via Google Drive. Please let me know if this didn't work.

> If we can work around any of those issues in the code in the GitHub repo I'd
> be happy to merge a PR.

I will check to see how much of this we can incorporate into the script directly. At the very least I will update the docs/readme for the script.

Comment 12

[(not currently active) Ted Mielczarek]

I uploaded those symbols:
https://symbols.mozilla.org/uploads/upload/28068

Comment 13

[Stephen A Pohl [:spohl]]

Thanks, Ted! Is there anything left to do here? Does anything need to be verified? Or can this bug be closed at this time?

Comment 14

[(not currently active) Ted Mielczarek]

I think the only thing left to do would be to reprocess crashes that are missing symbols and ensure that everything looks OK. If there are individual crashes they can be reprocessed from the crash-stats UI directly. If there are whole signatures that want reprocessing we have a commandline tool to do that: https://github.com/peterbe/reprocess-supersearch

Comment 15

[Will Kahn-Greene [:willkg] ET needinfo? me]

I can reprocess. We have good tools for batch reprocessing in Socorro now. It's similar to reprocess-supersearch, but probably a little faster and works in a Socorro local dev environment container.

https://socorro.readthedocs.io/en/latest/howto.html?highlight=reprocess#reprocess-crashes

If someone can throw together a few super search urls, I can do it from that.

Comment 16

[Marcia Knous [:marcia]]

I was able to individually reprocess https://crash-stats.mozilla.com/report/index/d7b45233-8637-4d88-bd75-7e61a0180830 and filed a bug from that stack. I think our 10.14 population is pretty small, so we haven't generated a huge volume of crashes. The worst crash, Bug 1467230, doesn't appear to be happening on any seeds after 10.14.0 18A293u so I think we are out of the woods on that one.

The latest seed 18A377a was pushed out at the beginning of this week - https://bit.ly/2PTMld6 is the search that would capture the Firefox crashes for that version. There are so few on that list I think we can deal with them individually for now.

Comment 17

[Will Kahn-Greene [:willkg] ET needinfo? me]

Marcia: I reprocessed those just now.

Comment 18

[Stephen A Pohl [:spohl]]

Based on the recent comments I'm going to close this bug.

Comment 19

[Stephen A Pohl [:spohl]]

To close the loop: A PR for breakpad-scrape-system-symbols has been merged today that will avoid problems due to a lack of an absolute path to dump_syms.

I have opened two issues in GitHub for the following:
1. dump_syms no longer available from people.mozilla.org
2. Setting the OBJC_DISABLE_INITIALIZE_FORK_SAFETY env variable when running gathersymbols.py.

I will defer to someone else on where to host dump_syms and I haven't had time to evaluate (and wouldn't know off-hand) how much work it would be to use virtualenvwrapper to address issue 2 automatically.

Comment 20

[Jeff Muizelaar [:jrmuizel]]

Can the scripts be run for 10.14.0 18A389?

Comment 21

[Stephen A Pohl [:spohl]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #20)
> Can the scripts be run for 10.14.0 18A389?

Yes, they can. Unfortunately, my system has already been updated to macOS 10.14.1 18B75 and there doesn't seem to be an installer for 10.14.0 18A389 on the Apple Developer download site. I'm happy to walk someone through the steps if we can find a system with 10.14.0 18A389 installed.

Comment 22

[Kartikaya Gupta (email:[e-mail])]

I have 18A391 on one machine, if you need that.

Comment 23

[Stephen A Pohl [:spohl]]

:jrmuizel, I'm guessing you were asking about this specific build because we have crash reports that need to be symbolized?

:kate, if you could go through comment 6 and see if you can scrape the symbols, that would be great. I will also send you an email with the dump_syms binary. Thanks!

Comment 24

[Stephen A Pohl [:spohl]]

(In reply to Stephen A Pohl [:spohl] from comment #23)
> :kate

s/kate/kats/, sorry (phone autocorrect).

Comment 25

[Kartikaya Gupta (email:[e-mail])]

I used the command from comment 6 with the dump_syms in the email and generated the symbols.zip. It is here: https://drive.google.com/file/d/1Aa2991m9GxpajusSN4d89rRHb3iFcZEQ/view?usp=sharing

Comment 26

[(not currently active) Ted Mielczarek]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #25)
> I used the command from comment 6 with the dump_syms in the email and
> generated the symbols.zip. It is here:
> https://drive.google.com/file/d/1Aa2991m9GxpajusSN4d89rRHb3iFcZEQ/
> view?usp=sharing

I uploaded these symbols to the symbol server.

Comment 27

[Peter Bengtsson [:peterbe]]

(In reply to Stephen A Pohl [:spohl] from comment #6)
> I have been able to get the script to run and scrape the symbols. However,
> because crash-analysis.mozilla.com is no longer accessible, I had to scrape
> all symbols instead of just the missing ones. 

The new symbols.mozilla.org records all missing symbols. I think it was years ago that we recorded missing symbols on crash-analysis.mozilla.com

Basically, every request for a symbol on symbols.mozilla.org that leads to the conclusion that it doesn't exist in S3 is logged persistently. A couple of caveats...:

1) We only store the missing symbol *once* per every 24h. 
2) Some requests are completely ignored [0].
3) You can download a CSV file (it's actually dynamic) of all recent (yesterday or today) missing symbols but it's limited to .pdb symbols with .sym filenames. [1]

The CSV report is https://symbols.mozilla.org//missingsymbols.csv but you can see ALL missing symbols here: https://symbols.mozilla.org/downloads/missing


[0] https://github.com/mozilla-services/tecken/blob/8f1716458d68d539ea4b30d8c4ed3b72971da8de/tecken/download/views.py#L49
[1] https://github.com/mozilla-services/tecken/blob/8f1716458d68d539ea4b30d8c4ed3b72971da8de/tecken/download/views.py#L329-L330

Comment 28

[Jeff Muizelaar [:jrmuizel]]

It looks like we're missing symbols for 10.14.1 18B75 as well:
https://crash-stats.mozilla.com/report/index/28bde9a4-ce32-4ff0-b6db-0ef2b0181119#tab-details

Ted, shouldn't we be getting symbols for updates automatically?

Comment 29

[(not currently active) Ted Mielczarek]

Uploads from the symbol cron jobs are currently broken: bug 1487141. I discussed handing over responsibility for those cron jobs to RelMan last week, you should follow up with sylvestre or marco if you need more info.

Comment 30

[Will Kahn-Greene [:willkg] ET needinfo? me]

Moving from Socorro product to Tecken product.