1510698 - Missing symbols for libxul in recent builds

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Jason Kratzer [:jkratzer]]

At some point between build 20181127215721 and 20181128095655 it appears that symbols for libxul are no longer packaged with debug builds on linux.  Windows builds appear to be fine.

Relevant output from minidump_stackwalk:
2018-11-28 17:24:31: http_symbol_supplier.cc:553: INFO: HTTPSymbolSupplier: querying http://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/firefox-bin/8E77AE31EB177A206FE897CD2CE80DE00/firefox-bin.sym
2018-11-28 17:24:31: http_symbol_supplier.cc:467: INFO: HTTPSymbolSupplier: HTTP response code: 404
2018-11-28 17:24:31: stackwalker.cc:97: INFO: Couldn't load symbols for: /home/ubuntu/firefox/firefox-bin|8E77AE31EB177A206FE897CD2CE80DE00
2018-11-28 17:24:31: simple_symbol_supplier.cc:196: INFO: No symbol file at /tmp/libxul.so/F02840F0EB4E0C96C7C562D8008EFE590/libxul.so.sym
2018-11-28 17:24:31: http_symbol_supplier.cc:553: INFO: HTTPSymbolSupplier: querying http://s3-us-west-2.amazonaws.com/org.mozilla.crash-stats.symbols-public/v1/libxul.so/F02840F0EB4E0C96C7C562D8008EFE590/libxul.so.sym
2018-11-28 17:24:31: http_symbol_supplier.cc:467: INFO: HTTPSymbolSupplier: HTTP response code: 404
2018-11-28 17:24:31: stackwalker.cc:97: INFO: Couldn't load symbols for: /home/ubuntu/firefox/libxul.so|F02840F0EB4E0C96C7C562D8008EFE590
2018-11-28 17:24:31: basic_code_modules.cc:110: INFO: No module at 0x0
2018-11-28 17:24:31: basic_code_modules.cc:110: INFO: No module at 0x7fffb8a0d090
2018-11-28 17:24:31: basic_code_modules.cc:110: INFO: No module at 0x0

Comment 1

[Jesse Schwartzentruber (:truber)]

Ted, any idea what might be going on here?

Comment 2

[(not currently active) Ted Mielczarek]

Just some informational bits, it looks like these are the two nightlies you're talking about:
https://hg.mozilla.org/mozilla-central/rev/204cda7581188cfc8c8ef11dce4680dadf2b43bb
https://hg.mozilla.org/mozilla-central/rev/5c66354bff282452a6f1a3c911fa8756b6e752af

So the regression range would be:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=204cda7581188cfc8c8ef11dce4680dadf2b43bb&tochange=5c66354bff282452a6f1a3c911fa8756b6e752af

Nothing immediately jumps out. I'll dig in a little more.

Comment 3

[(not currently active) Ted Mielczarek]

This seems to be limited to debug builds. I realized that we get the num_static_constructors metric from the dump_syms output so I looked at the graphs for that metric and it was obvious where this broke:
https://treeherder.mozilla.org/perf.html#/graphs?series=autoland,1468482,1,2&highlightedRevisions=bba74d02a136

This points to bug 1501796 breaking things. Presumably nasm must be emitting some debug info that dump_syms can't deal with.

Comment 4

[Thomas Daede [:TD-Linux]]

We could try explicitly specifying a different debug format for nasm (or no debug info at all). https://www.nasm.us/doc/nasmdoc2.html#section-2.1.12

Is there a way to reproduce this on try?

Comment 6

[(not currently active) Ted Mielczarek]

(In reply to Thomas Daede [:TD-Linux] from comment #4)
> Is there a way to reproduce this on try?

It seems to reproduce on all Linux debug builds, the logs just don't show anything informative.

Comment 7

[Tyson Smith [:tsmith]]

Everything seemed to back to normal late last week until this morning. Any idea what changed?

I will disable fuzzing on debug builds again for now.

Comment 8

[Thomas Daede [:TD-Linux]]

Yes, instead of building the whole binary with nasm, we switched back to yasm. Then we turned on dav1d assembly, which builds only the dav1d .o files with nasm. I guess putting *any* object files generated with nasm breaks symbols.

Comment 9

[Gabriele Svelto [:gsvelto]]

I'll see if I can repro this locally and eventually fix dump_syms to accommodate for whatever DWARF debuginfo nasm is spitting out.

Comment 10

[Gabriele Svelto [:gsvelto]]

I reproduced this with my local build. dump_syms is failing with an abort() while parsing the DWARF data here:

https://searchfox.org/mozilla-central/rev/6c784c93cfbd5119ed07773a170b59fbce1377ea/toolkit/crashreporter/google-breakpad/src/common/dwarf/dwarf2reader.cc#158

I'll try and figure out what's going on.

Comment 11

[Gabriele Svelto [:gsvelto]]

The DWARF abbreviations generated by nasm 2.13.01 are not terminated causing the DWARF parser to fail:

https://bugzilla.nasm.us/show_bug.cgi?id=3392417

This was already fixed in nasm 2.13.02 so we'll need to bump the version we use.

Comment 12

[Tyson Smith [:tsmith]]

Gabriele any updates? Do you know when bug 1520163 might be resolved?

Comment 13

[Gabriele Svelto [:gsvelto]]

I don't know unfortunately, if there's no other way we might roll out a workaround in the meantime.

Comment 14

[Gabriele Svelto [:gsvelto]]

Is this still a problem? Bug 1520163 should have fixed it.

Comment 15

[Jason Kratzer [:jkratzer]]

Nope. Symbols are working once again.

Comment 16

[Gabriele Svelto [:gsvelto]]

Good, closing as fixed.