818903 - Investigate why debug info files have crc mismatches

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

When getting symbols files from the symbols server for nightlies, gdb complains about most of the files (except, interestingly, libxul.so, and sometimes a few others)

I can't reproduce locally, it failed to reproduce on a staging machine (bug 816102), and I failed to reproduce on try with PGO both on and off. So I guess instrumenting a nightly build is about the only way to have some feedback about what's going on.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Temporarily add some info about binaries and debug info files

Comment 2

[Mike Hommey [:glandium]]

http://hg.mozilla.org/mozilla-central/rev/3bb2d53dde34

Comment 3

[Mike Hommey [:glandium]]

So, if I take a random file from last nightly's log, the output added for it is:
Contents of section .gnu_debuglink:
 0000 66697265 666f782e 64626700 d5208b6a  firefox.dbg.. .j
./dist/bin/firefox.dbg crc: 6a8b20d5

At this point (so, right after objcopy), the crc matches (d5208b6a == 6a8b20d5, the value in .gnu_debuglink being a dumped little endian value).

(log is https://tbpl.mozilla.org/php/getParsedLog.php?id=17708635&tree=Firefox )

Its full path is firefox/1DC5137538355C0921B17717453612A50/firefox.dbg.gz
If I download this nightly, and check the firefox binary, I can see the .gnu_debuglink section has not changed:
Contents of section .gnu_debuglink:
 0000 66697265 666f782e 64626700 d5208b6a  firefox.dbg.. .j

However, if I download http://symbols.mozilla.org/firefox/firefox/1DC5137538355C0921B17717453612A50/firefox.dbg.gz, gunzip -lv gives me:
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla 74ce102f Nov 14 21:21              196552              523844  62.5% firefox.dbg

Something is either modifying firefox.dbg between the objcopy and the symbol full archive creation, or something is modifying firefox.dbg on the server.

Comment 4

[Mike Hommey [:glandium]]

In fact, the gunzip -lv output makes it striking obvious: the firefox.dbg is old. Which means its id,
1DC5137538355C0921B17717453612A50, matches several different firefoxes.

Comment 5

[Mike Hommey [:glandium]]

That could also explain some funky stack traces on Android...

Comment 6

[Mike Hommey [:glandium]]

Attachment: Link with --build-id when available

Let's just switch to build ids. Breakpad supports it, and I think our toolchains now do, too (thanks to mock) ; the NDK should support it too.

https://tbpl.mozilla.org/?tree=Try&rev=71643686641b

Comment 7

[(not currently active) Ted Mielczarek]

Comment on attachment 690067 [details] [diff] [review]
Link with --build-id when available

Review of attachment 690067 [details] [diff] [review]:
-----------------------------------------------------------------

That makes a lot of sense. With the hash of the .text section you'd get identical debug IDs whenever the code compiles the same.

Comment 8

[Mike Hommey [:glandium]]

http://hg.mozilla.org/mozilla-central/rev/d62b457df9a7
http://hg.mozilla.org/mozilla-central/rev/4e83d0987a31

Comment 9

[Mike Hommey [:glandium]]

This doesn't apply to nspr and nss :-/

Comment 10

[Ed Morley [:emorley]]

(In reply to Mike Hommey [:glandium] from comment #5)
> That could also explain some funky stack traces on Android...

Do you have any examples of these? (Just wondering if it affected any of the stacks seen on TBPL).

Comment 11

[Mike Hommey [:glandium]]

(In reply to Ed Morley [UTC+0; email:edmorley@moco] from comment #10)
> (In reply to Mike Hommey [:glandium] from comment #5)
> > That could also explain some funky stack traces on Android...
> 
> Do you have any examples of these? (Just wondering if it affected any of the
> stacks seen on TBPL).

I don't.

Comment 12

[Mike Hommey [:glandium]]

Attachment: Also link NSPR and NSS with --build-id when available

Comment 13

[(not currently active) Ted Mielczarek]

(In reply to Ed Morley [UTC+0; email:edmorley@moco] from comment #10)
> Do you have any examples of these? (Just wondering if it affected any of the
> stacks seen on TBPL).

This should have no impact on TBPL, since stack traces there are generated using Breakpad symbols which are matched with the build being used for testing.

Comment 14

[(not currently active) Ted Mielczarek]

Comment on attachment 690331 [details] [diff] [review]
Also link NSPR and NSS with --build-id when available

Review of attachment 690331 [details] [diff] [review]:
-----------------------------------------------------------------

We could also port this configure test to NSPR if you wanted.

Comment 15

[Mike Hommey [:glandium]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #14)
> Comment on attachment 690331 [details] [diff] [review]
> Also link NSPR and NSS with --build-id when available
> 
> Review of attachment 690331 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> We could also port this configure test to NSPR if you wanted.

That would be more hassle to uplift.

Comment 16

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/28807b41f92e

Comment 17

[Ed Morley [:emorley]]

(In reply to Mike Hommey [:glandium] from comment #16)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/28807b41f92e

Backed out for configure failures:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a6414d343875

Comment 18

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3cb1250425c0

Comment 19

[Mike Hommey [:glandium]]

And a fixup
https://hg.mozilla.org/integration/mozilla-inbound/rev/508d1f5c60b0

Comment 20

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/3cb1250425c0
https://hg.mozilla.org/mozilla-central/rev/508d1f5c60b0