1371017 - Crash in XUL@0x36b83cd | XUL@0x36bcb1b | XUL@0x36c02fe | XUL@0x36b9b38 | XUL@0x36b947c | XUL@0x36b8dd9 | XUL@0x36b8977 | XUL@0x36b935c | XUL@0xa07de | XUL@0x123b921 | XUL@0xda7a5c | XUL@0x113da47 | XUL@0x1267279 | XUL@0x1ce2c5c | XUL@0xda482e | XUL@0xd...

Status: RESOLVED DUPLICATE of bug 1380381

(Toolkit :: Crash Reporting, defect)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-d0804931-cc57-481c-9fd8-3d3ff0170607.
=============================================================

Today's Uptime report shows a bunch of Mac signatures that looks to have symbol issues - see http://bit.ly/2sgSWoX

Ted - Any ideas what might be going on? In the run up to merge we really need good crash data.

Comment 1

[[:philipp]]

the comment from bp-3a3fee82-dd35-4ea6-b5f7-eb8dd0170607 says: "Clicking the menu icon crashes every time"

Comment 2

[Julien Cristau [:jcristau]]

marking as blocker for 55 as we'll be in a very bad place to roll out to beta without symbols for mac

Comment 3

[(not currently active) Ted Mielczarek]

Exciting, llvm-dsymutil is crashing. Selected log excerpts from https://public-artifacts.taskcluster.net/TWx923mfQYmQi6duHFFJFQ/0/public/build/log_raw.log :

 /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/_virtualenv/bin/python -m mozbuild.action.dumpsymbols /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/library/XUL /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/library/XUL_syms.track
 Starting Mac pre-processing on file: /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/library/XUL
 Running Mac pre-processing on file: /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/library/XUL
 /builds/slave/m-cen-m64-ntly-000000000000000/build/src/clang/bin/llvm-dsymutil --arch=x86_64 /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/library/XUL

 Stack dump:
 0.	Program arguments: /builds/slave/m-cen-m64-ntly-000000000000000/build/src/clang/bin/llvm-dsymutil --arch=x86_64 /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/library/XUL
 Error running dsymutil: Command '['/builds/slave/m-cen-m64-ntly-000000000000000/build/src/clang/bin/llvm-dsymutil', '--arch=x86_64', '/builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/library/XUL']' returned non-zero exit status -11
 Finished processing /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/library/XUL in 190.83s

Comment 4

[Marcia Knous [:marcia]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #3)
> Exciting, llvm-dsymutil is crashing. Selected log excerpts from
> https://public-artifacts.taskcluster.net/TWx923mfQYmQi6duHFFJFQ/0/public/
> build/log_raw.log :
> 
>  /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/
> _virtualenv/bin/python -m mozbuild.action.dumpsymbols
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL_syms.track
>  Starting Mac pre-processing on file:
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL
>  Running Mac pre-processing on file:
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL
>  /builds/slave/m-cen-m64-ntly-000000000000000/build/src/clang/bin/llvm-
> dsymutil --arch=x86_64
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL
> 
>  Stack dump:
>  0.	Program arguments:
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/clang/bin/llvm-
> dsymutil --arch=x86_64
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL
>  Error running dsymutil: Command
> '['/builds/slave/m-cen-m64-ntly-000000000000000/build/src/clang/bin/llvm-
> dsymutil', '--arch=x86_64',
> '/builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL']' returned non-zero exit status -11
>  Finished processing
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL in 190.83s

Ted, are you the right person to fix this - or who should I ping? This has been deemed a blocker for the merge next week.

Comment 5

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Coop, Laura, this is a blocking bug for 55.0 beta1 next week. Could you please help find an owner who can fix this asap? Thanks!

Comment 6

[(not currently active) Ted Mielczarek]

I'm still looking into this. I'm going to try to reproduce with a local build. If that doesn't work I'll have to get a loaner build machine to try it out on.

Comment 7

[Phil Ringnalda (:philor)]

STR to get my crash like the comment 1 crash is "new profile, install https://addons.mozilla.org/en-US/firefox/addon/copy-link-text-webextension/ (or one of a half-dozen others I found by searching for 'webextension' and installing whatever came up) and click the hamburger." According to aswan, who got it in a debugger with those steps, the actual crash from doing that is bug 1370263.

(And if you like extremely twisted workarounds, found while installing random things, after you've installed Copy Link Text and crashed a few times install https://addons.mozilla.org/en-US/firefox/addon/media-conversion-tool/ and the crash goes away.)

Comment 8

[Chris Cooper [:coop] (he/him)]

Ted has picked this up.

Comment 9

[(not currently active) Ted Mielczarek]

In an email, Marcia said she first noticed it in crash-stats starting June 7th. I'll try looking at build logs to see if I can narrow down when it first appeared.

I could not reproduce locally on my Mac with the same llvm-dsymutil binary we use in CI, but building with the clang that shipped with my XCode.

Comment 10

[(not currently active) Ted Mielczarek]

Just for sanity, there weren't any changes to the toolchain in use in that timeframe:
https://hg.mozilla.org/mozilla-central/filelog/e61060be36424240058f8bef4c5597f401bc8b7e/browser/config/tooltool-manifests/macosx64/releng.manifest

Comment 11

[Marcia Knous [:marcia]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> In an email, Marcia said she first noticed it in crash-stats starting June
> 7th. I'll try looking at build logs to see if I can narrow down when it
> first appeared.
> 
> I could not reproduce locally on my Mac with the same llvm-dsymutil binary
> we use in CI, but building with the clang that shipped with my XCode.

When I was doing Uptime triage on Monday the 5th I did not see it - https://wiki.mozilla.org/Platform/Uptime/NightlyCrashTriage/2017Q2#2017-06-05_-_marcia (and I always check the Mac builds). The next 2 triagers did not note it, and Andrew noted it when he was doing the triage for the build on the 7th.

Comment 12

[(not currently active) Ted Mielczarek]

I'm grepping the logs for "Error running dsymutil".

It failed on Thursday's nightly:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=OS%20X%2010.7%20opt&selectedJob=105777885

and the OS X opt build from the same changeset (which should be the exact same build type):
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=OS%20X%2010.7%20opt&selectedJob=105689534

but passed on the next central OS X opt build:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=OS%20X%2010.7%20opt&selectedJob=105786977

Failed on Tuesday's *second* nightly:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=OS%20X%2010.7%20opt&fromchange=4dd1d17ba22660b8f5869a707f2e4e9f9dd5be5b&selectedJob=105141438

Passed on Monday's nightly:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=OS%20X%2010.7%20opt&fromchange=98f1390029f9bd558de991a53c92342ac0addfc4&selectedJob=104479655

Passed on Tuesday's *first* nightly:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=OS%20X%2010.7%20opt&fromchange=98f1390029f9bd558de991a53c92342ac0addfc4&selectedJob=104834002

So it originally regressed on central in this push, which is a merge from autoland:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=fd04166b7114949ce63783e10a069b98d76df573&filter-searchStr=OS%20X%2010.7%20opt

There's a whole pile of Servo changesets in there, it wouldn't surprise me if one of them was to blame.

Comment 13

[(not currently active) Ted Mielczarek]

In light of the fact that it passed again on a newer changeset I'm going to update to one of the changesets where it failed in CI and build that locally to see if I can reproduce.

Comment 14

[(not currently active) Ted Mielczarek]

Specifically I'm going to try a local build of fd04166b7114949ce63783e10a069b98d76df573.

Comment 15

[(not currently active) Ted Mielczarek]

FWIW, the crashes in the signature field are almost certainly not all the same *crash*. They're all missing symbols due to this llvm-dsymutil, but they're all probably different crashes.

Comment 16

[Steven Michaud [:smichaud] (Retired)]

Could the crashes have something to do with how much RAM is installed on the machine where they're happening? :-)

Comment 17

[Steven Michaud [:smichaud] (Retired)]

The crashes in dsymutil, that is.

Comment 18

[(not currently active) Ted Mielczarek]

So interestingly, I looked at builds on autoland for the changesets in that merge, and it doesn't seem to be failing there. This is the last autoland changeset in that merge, and it's not failing:
https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=fad0b0be486a2c44198a025983f4ad138d02ec8b&filter-searchStr=os%20x%2010.7%20opt&selectedJob=104940320

Comment 19

[(not currently active) Ted Mielczarek]

Wait, I confused myself, it passed in that merge:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&fromchange=fd04166b7114949ce63783e10a069b98d76df573&filter-searchStr=os%20x%2010.7%20opt&selectedJob=105017865

but failed in the next merge (from inbound):
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&fromchange=fd04166b7114949ce63783e10a069b98d76df573&filter-searchStr=os%20x%2010.7%20opt&selectedJob=105023665

Comment 20

[(not currently active) Ted Mielczarek]

...but the last inbound changeset in that merge ran llvm-dsymutil successfully in its build on inbound:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&tochange=b6e861dcd4c25b9dd98bb820e7cad8a475f8e953&filter-searchStr=os%20x%2010.7%20opt&selectedJob=105011162

(In reply to Steven Michaud [:smichaud] (Retired) from comment #16)
> Could the crashes have something to do with how much RAM is installed on the
> machine where they're happening? :-)

I was going to look into the machine specs, but looking at the logs I just noticed that the inbound build I linked at the top of this comment (which succeeded) and the central build I linked at the end of comment 19 (which failed) both built on the exact same build machine!

slave: bld-lion-r5-051

Comment 21

[Steven Michaud [:smichaud] (Retired)]

dsymutil is part of the clang distro.  The source code for Apple's clang distros (at least for those whose source Apple has released to the present date) is at https://opensource.apple.com/source/clang/.  The source code for Apple's most recently released dsymutil is (as best I can tell) here https://opensource.apple.com/source/clang/clang-800.0.42.1/src/tools/dsymutil/dsymutil.cpp.auto.html.

From that it seems that '-11' isn't a legal return value. exitDsymtuil() seems only ever to be called with a parameter of '0' or '1'.

Comment 22

[Steven Michaud [:smichaud] (Retired)]

In any case, though, dsymutil seems to write error messages to stderr whenever it fails. I wonder why those messages aren't showing up in the logs.

Comment 23

[(not currently active) Ted Mielczarek]

We're not using Apple's clang for our builds anymore, we use our own builds from upstream. We're using the llvm-dsymutil that gets built as part of that clang+llvm build.

And yes, `-11` is SIGSEGV :)

Comment 24

[(not currently active) Ted Mielczarek]

I'm wondering if this is tickled by dep vs. clobber builds. I pushed that one failing changeset to try to see if llvm-dsymutil fails there:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=41664204d9c5eadea3c725fbb5fad6268384f5f4

Comment 25

[(not currently active) Ted Mielczarek]

(In reply to Steven Michaud [:smichaud] (Retired) from comment #22)
> In any case, though, dsymutil seems to write error messages to stderr
> whenever it fails. I wonder why those messages aren't showing up in the logs.

It's a little buried, but in comment 3 you can see:
 Stack dump:
 0.	Program arguments: /builds/slave/m-cen-m64-ntly-000000000000000/build/src/clang/bin/llvm-dsymutil --arch=x86_64 /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/library/XUL

Unfortunately that's all we get. :-(

Comment 26

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #19)
> but failed in the next merge (from inbound):
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> central&fromchange=fd04166b7114949ce63783e10a069b98d76df573&filter-
> searchStr=os%20x%2010.7%20opt&selectedJob=105023665

Apparently I can't copy+paste successfully today either. The failing merge is:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=5801aa478de12a62b2b2982659e787fcc4268d67&filter-searchStr=os%20x%2010.7%20opt

I built 5801aa478de12a62b2b2982659e787fcc4268d67 locally and llvm-dsymutil does not crash on the resulting XUL binary.

Comment 27

[Steven Michaud [:smichaud] (Retired)]

> And yes, `-11` is SIGSEGV :)

So dsymutil really is crashing, and you weren't using the word "crash" as a figure of speech :-) That would explain why it doesn't write anything to stderr.

In any case I have another semi-wild guess.  Searching the web on "dsymutil crash", it seems that it can crash because of malformed input files, including ones other than those specified on the command line (for example Info.plist).  So I wonder if, in some cases, trash is being left (not cleaned up) from previous build cycles.

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #20)
> slave: bld-lion-r5-051

There doesn't seem to be a slave correlation here. The 20170609030207 nightly, which is rev f4262773c433 on bld-lion-r5-070, also fails to extract XUL symbols. Same for 20170608030205 on bld-lion-r5-070. These are older xserves with 8G of RAM. Loaner ?

Also, I think we should make failure to extract symbols at least turn the build orange, maybe red, so this is more visible in future.

Comment 29

[Mike Conley (:mconley) (:⚙️)]

Hey folks (and hi smichaud!!). chmanchester pointed me towards this bug because I can't seem to build symbols locally.

Using ./mach build buildsymbols -j1 -v, I also get:

 0:11.85 Starting Mac pre-processing on file: /Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/XUL
 0:11.85 Running Mac pre-processing on file: /Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/XUL
 0:11.85 /usr/bin/dsymutil --arch=x86_64 /Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/XUL
 1:58.18 Error running dsymutil: Command '['/usr/bin/dsymutil', '--arch=x86_64', '/Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/XUL']' returned non-zero exit status -11
 1:58.18 No symbols found in file: /Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/XUL

So I can hit this bug pretty easily locally.

ted, is that useful at all? If you have stuff you want me to try, let me know.

Comment 30

[Mike Conley (:mconley) (:⚙️)]

ni? for comment 29.

Comment 31

[Mike Conley (:mconley) (:⚙️)]

I ran /usr/bin/dsymutil --arch=x86_64 /Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/XUL --verbose > output.dump

Here are the last 100 lines of output.dump:

MacBook-Pro-104:mozilla-central mikeconley$ tail -n 100 output.dump
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/rust/<panic macros>")
                      DW_AT_decl_line [DW_FORM_data1]	(5)
                      DW_AT_location [DW_FORM_block1]	(<0x09> 03 38 3e 11 00 00 00 00 00 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x00000f20] = "_ZN18webrender_bindings8bindings17wr_dp_push_border10_FILE_LINEE")
Found valid debug map entry: __ZN18webrender_bindings8bindings23wr_dp_push_border_image10_FILE_LINE17h5c908b1f72eca6c5E 	0000000000113e20 => 0000000004ea0480
0x00001d36:         DW_TAG_variable [4]
                      DW_AT_name [DW_FORM_strp]	( .debug_str[0x00000339] = "_FILE_LINE")
                      DW_AT_type [DW_FORM_ref4]	(cu + 0x1e83c => {0x0001ff55})
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/rust/<panic macros>")
                      DW_AT_decl_line [DW_FORM_data1]	(5)
                      DW_AT_location [DW_FORM_block1]	(<0x09> 03 20 3e 11 00 00 00 00 00 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x00000f79] = "_ZN18webrender_bindings8bindings23wr_dp_push_border_image10_FILE_LINEE")
Found valid debug map entry: __ZN18webrender_bindings8bindings26wr_dp_push_border_gradient10_FILE_LINE17hb3a868c6cc194d68E 	0000000000113e08 => 0000000004ea0468
0x00001d58:         DW_TAG_variable [4]
                      DW_AT_name [DW_FORM_strp]	( .debug_str[0x00000339] = "_FILE_LINE")
                      DW_AT_type [DW_FORM_ref4]	(cu + 0x1e83c => {0x0001ff55})
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/rust/<panic macros>")
                      DW_AT_decl_line [DW_FORM_data1]	(5)
                      DW_AT_location [DW_FORM_block1]	(<0x09> 03 08 3e 11 00 00 00 00 00 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x00000fdb] = "_ZN18webrender_bindings8bindings26wr_dp_push_border_gradient10_FILE_LINEE")
Found valid debug map entry: __ZN18webrender_bindings8bindings33wr_dp_push_border_radial_gradient10_FILE_LINE17ha563d0bcc79c8374E 	0000000000113df0 => 0000000004ea0450
0x00001dad:         DW_TAG_variable [4]
                      DW_AT_name [DW_FORM_strp]	( .debug_str[0x00000339] = "_FILE_LINE")
                      DW_AT_type [DW_FORM_ref4]	(cu + 0x1e83c => {0x0001ff55})
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/rust/<panic macros>")
                      DW_AT_decl_line [DW_FORM_data1]	(5)
                      DW_AT_location [DW_FORM_block1]	(<0x09> 03 f0 3d 11 00 00 00 00 00 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x00001047] = "_ZN18webrender_bindings8bindings33wr_dp_push_border_radial_gradient10_FILE_LINEE")
Found valid debug map entry: __ZN18webrender_bindings8bindings26wr_dp_push_linear_gradient10_FILE_LINE17hf8bf25ee1e7ed546E 	0000000000113dd8 => 0000000004ea0438
0x00001e02:         DW_TAG_variable [4]
                      DW_AT_name [DW_FORM_strp]	( .debug_str[0x00000339] = "_FILE_LINE")
                      DW_AT_type [DW_FORM_ref4]	(cu + 0x1e83c => {0x0001ff55})
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/rust/<panic macros>")
                      DW_AT_decl_line [DW_FORM_data1]	(5)
                      DW_AT_location [DW_FORM_block1]	(<0x09> 03 d8 3d 11 00 00 00 00 00 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x000010b3] = "_ZN18webrender_bindings8bindings26wr_dp_push_linear_gradient10_FILE_LINEE")
Found valid debug map entry: __ZN18webrender_bindings8bindings26wr_dp_push_radial_gradient10_FILE_LINE17h4aa0dfbdea6fa2feE 	0000000000113dc0 => 0000000004ea0420
0x00001e57:         DW_TAG_variable [4]
                      DW_AT_name [DW_FORM_strp]	( .debug_str[0x00000339] = "_FILE_LINE")
                      DW_AT_type [DW_FORM_ref4]	(cu + 0x1e83c => {0x0001ff55})
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/rust/<panic macros>")
                      DW_AT_decl_line [DW_FORM_data1]	(5)
                      DW_AT_location [DW_FORM_block1]	(<0x09> 03 c0 3d 11 00 00 00 00 00 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x00001118] = "_ZN18webrender_bindings8bindings26wr_dp_push_radial_gradient10_FILE_LINEE")
Found valid debug map entry: __ZN18webrender_bindings8bindings21wr_dp_push_box_shadow10_FILE_LINE17hb0a0e97c42192b8eE 	0000000000113da8 => 0000000004ea0408
0x00001eac:         DW_TAG_variable [4]
                      DW_AT_name [DW_FORM_strp]	( .debug_str[0x00000339] = "_FILE_LINE")
                      DW_AT_type [DW_FORM_ref4]	(cu + 0x1e83c => {0x0001ff55})
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/obj-x86_64-apple-darwin14.5.0/toolkit/library/rust/<panic macros>")
                      DW_AT_decl_line [DW_FORM_data1]	(5)
                      DW_AT_location [DW_FORM_block1]	(<0x09> 03 a8 3d 11 00 00 00 00 00 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x00001178] = "_ZN18webrender_bindings8bindings21wr_dp_push_box_shadow10_FILE_LINEE")
Found valid debug map entry: __ZN114_$LT$webrender_bindings..bindings..WrExternalImageHandler$u20$as$u20$webrender..renderer..ExternalImageHandler$GT$4lock17h61265c1721597b9aE 	00000000000003e0 => 0000000003b924b0
0x00001fd1:         DW_TAG_subprogram [52] *
                      DW_AT_low_pc [DW_FORM_addr]	(0x00000000000003e0)
                      DW_AT_high_pc [DW_FORM_addr]	(0x000000000000046b)
                      DW_AT_frame_base [DW_FORM_block1]	(<0x01> 56 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x0000f56b] = "_ZN18webrender_bindings8bindings8{{impl}}4lockE")
                      DW_AT_name [DW_FORM_strp]	( .debug_str[0x00009667] = "lock")
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/gfx/webrender_bindings/src/bindings.rs")
                      DW_AT_decl_line [DW_FORM_data2]	(591)
                      DW_AT_type [DW_FORM_ref4]	(cu + 0x47669 => {0x00048d82})
                      DW_AT_external [DW_FORM_flag]	(0x01)
                      DW_AT_APPLE_optimized [DW_FORM_flag]	(0x01)
Found valid debug map entry: __ZN114_$LT$webrender_bindings..bindings..WrExternalImageHandler$u20$as$u20$webrender..renderer..ExternalImageHandler$GT$6unlock17hcfa61f6918f3c67dE 	0000000000000470 => 0000000003b92540
0x00002108:         DW_TAG_subprogram [67] *
                      DW_AT_low_pc [DW_FORM_addr]	(0x0000000000000470)
                      DW_AT_high_pc [DW_FORM_addr]	(0x0000000000000481)
                      DW_AT_frame_base [DW_FORM_block1]	(<0x01> 56 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x0000f59b] = "_ZN18webrender_bindings8bindings8{{impl}}6unlockE")
                      DW_AT_name [DW_FORM_strp]	( .debug_str[0x00009486] = "unlock")
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/gfx/webrender_bindings/src/bindings.rs")
                      DW_AT_decl_line [DW_FORM_data2]	(619)
                      DW_AT_external [DW_FORM_flag]	(0x01)
                      DW_AT_APPLE_optimized [DW_FORM_flag]	(0x01)
Found valid debug map entry: _wr_vec_u8_free 	0000000000000490 => 0000000003b92560
0x000021a1:         DW_TAG_subprogram [67] *
                      DW_AT_low_pc [DW_FORM_addr]	(0x0000000000000490)
                      DW_AT_high_pc [DW_FORM_addr]	(0x00000000000004aa)
                      DW_AT_frame_base [DW_FORM_block1]	(<0x01> 56 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x0000f647] = "_ZN18webrender_bindings8bindings14wr_vec_u8_freeE")
                      DW_AT_name [DW_FORM_strp]	( .debug_str[0x0000f638] = "wr_vec_u8_free")
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/gfx/webrender_bindings/src/bindings.rs")
                      DW_AT_decl_line [DW_FORM_data2]	(683)
                      DW_AT_external [DW_FORM_flag]	(0x01)
                      DW_AT_APPLE_optimized [DW_FORM_flag]	(0x01)
Found valid debug map entry: __ZN99_$LT$webrender_bindings..bindings..CppNotifier$u20$as$u20$webrender_traits..api..RenderNotifier$GT$15new_frame_ready17hc3ff1d302f8c60d9E 	00000000000004b0 => 0000000003b92580
0x000022bc:         DW_TAG_subprogram [67] *
                      DW_AT_low_pc [DW_FORM_addr]	(0x00000000000004b0)
                      DW_AT_high_pc [DW_FORM_addr]	(0x00000000000004bd)
                      DW_AT_frame_base [DW_FORM_block1]	(<0x01> 56 )
                      DW_AT_MIPS_linkage_name [DW_FORM_strp]	( .debug_str[0x0000f689] = "_ZN18webrender_bindings8bindings8{{impl}}15new_frame_readyE")
                      DW_AT_name [DW_FORM_strp]	( .debug_str[0x0000f679] = "new_frame_ready")
                      DW_AT_decl_file [DW_FORM_data1]	("/Users/mikeconley/Projects/mozilla-central/gfx/webrender_bindings/src/bindings.rs")
                      DW_AT_decl_line [DW_FORM_data2]	(747)
                      DW_AT_external [DW_FORM_flag]	(0x01)
                      DW_AT_APPLE_optimized [DW_FORM_flag]	(0x01)
Found valid debug map entry: __ZN99_$LT$webrender_bindings..bindings..CppNotifier$u20$as$u20$webrender_traits..api..RenderNotifier$GT$22new_scroll_frame_ready17hd4c664c73f74b211E 	00000000000004c0 => 0000000003b92590
0x0000230e:         DW_TAG_subprogram [67] *

Comment 32

[Mike Conley (:mconley) (:⚙️)]

Also, I just found bug 1301751 which seems somewhat similar... I'm going to try re-applying the fix from bug 1301751 (which was un-applied in bug 1305731).

Comment 33

[Mike Conley (:mconley) (:⚙️)]

Well, I got myself unstuck, although I don't know if it's a help for the folks in this bug.

Instead of applying the workaround fix in bug 1301751, I ran rustup update, and updated from 

rustc 1.17.0 (56124baa9 2017-04-24)

to 

rustc 1.18.0 (03fc9d622 2017-06-06)

This appears to have fixed my issue.

Comment 34

[(not currently active) Ted Mielczarek]

(In reply to Steven Michaud [:smichaud] (Retired) from comment #27)
> In any case I have another semi-wild guess.  Searching the web on "dsymutil
> crash", it seems that it can crash because of malformed input files,
> including ones other than those specified on the command line (for example
> Info.plist).  So I wonder if, in some cases, trash is being left (not
> cleaned up) from previous build cycles.

I thought about this, but someone pointed out that it's failing on nightly builds which are always clobbers. I suspect this is just a retread of the last time we hit this issue, bug 1301751, where rustc started emitting some DWARF that dsymutil didn't like.

(In reply to Nick Thomas [:nthomas] from comment #28)
> Also, I think we should make failure to extract symbols at least turn the
> build orange, maybe red, so this is more visible in future.

I tried to land a patch like that in bug 1304042, but it broke the build for some other issues that we had been wallpapering over in the past. It's worth revisiting post bug 1337986, since that changed the way a lot of symbol dumping works.

mconley: thanks for the info! I notice your log from comment 31 has webrender stuff in it. That's a useful sign that this may once again be a rustc issue.

Comment 35

[Andrew McCreight [:mccr8]]

As Marcia pointed out in her Uptime crash triage, this problem went away again in the 6/10 Nightly, and continues to not be present on the 6/11 and 6/12 Nightlies.

Comment 36

[Marcia Knous [:marcia]]

Would be great to know at some point why this happens, and why it goes away (maybe that is explained in the latter part of Comment 34).

Comment 37

[Marcia Knous [:marcia]]

Removing "blocking" per IRC discussion in release management channel, but adding a plus to keep this on the radar.

Comment 38

[Mike Conley (:mconley) (:⚙️)]

(In reply to Marcia Knous [:marcia - use ni] from comment #36)
> Would be great to know at some point why this happens, and why it goes away
> (maybe that is explained in the latter part of Comment 34).

Welp, if it's any help, the problem is back for me. :(

I guess I'll see if I can get WebRender to not produce debug stuff.

Comment 39

[Mike Conley (:mconley) (:⚙️)]

FWIW, adding ac_add_options --disable-webrender allowed me to build the XUL symbols again on OS X.

Comment 40

[Liz Henry (:lizzard) (relman/hg->git project)]

Ted, anything left to do here for 55? Or can we close this bug?

Comment 41

[nader.zubaid]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #3)
> Exciting, llvm-dsymutil is crashing. Selected log excerpts from
> https://public-artifacts.taskcluster.net/TWx923mfQYmQi6duHFFJFQ/0/public/
> build/log_raw.log :
> 
>  /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/
> _virtualenv/bin/python -m mozbuild.action.dumpsymbols
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL_syms.track
>  Starting Mac pre-processing on file:
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL
>  Running Mac pre-processing on file:
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL
>  /builds/slave/m-cen-m64-ntly-000000000000000/build/src/clang/bin/llvm-
> dsymutil --arch=x86_64
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL
> 
>  Stack dump:
>  0.	Program arguments:
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/clang/bin/llvm-
> dsymutil --arch=x86_64
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL
>  Error running dsymutil: Command
> '['/builds/slave/m-cen-m64-ntly-000000000000000/build/src/clang/bin/llvm-
> dsymutil', '--arch=x86_64',
> '/builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL']' returned non-zero exit status -11
>  Finished processing
> /builds/slave/m-cen-m64-ntly-000000000000000/build/src/obj-firefox/toolkit/
> library/XUL in 190.83s

Comment 42

[(not currently active) Ted Mielczarek]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #40)
> Ted, anything left to do here for 55? Or can we close this bug?

Sorry. This isn't *currently* breaking anything, but this issue is sort of silently lurking, which is not great. We need a fix for bug 1304042 to at least make it so this fails the build.

Actually fixing this will require someone to be able to reproduce it reliably, and probably building a local copy of clang with debug symbols and figuring out what part of llvm is crashing.

Comment 43

[Liz Henry (:lizzard) (relman/hg->git project)]

OK, thanks, sounds like this is work in progress, not currently a blocker for release, and release management can stop tracking this bug.

Comment 45

[(not currently active) Ted Mielczarek]

This was worked around in bug 1380381, clearing tracking flags.