Investigate why debug info files have crc mismatches

RESOLVED FIXED in mozilla20

Status

Firefox Build System
General
RESOLVED FIXED
5 years ago
2 months ago

People

(Reporter: glandium, Assigned: glandium)

Tracking

Trunk
mozilla20
x86_64
Linux

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

(Assignee)

Description

5 years ago
When getting symbols files from the symbols server for nightlies, gdb complains about most of the files (except, interestingly, libxul.so, and sometimes a few others)

I can't reproduce locally, it failed to reproduce on a staging machine (bug 816102), and I failed to reproduce on try with PGO both on and off. So I guess instrumenting a nightly build is about the only way to have some feedback about what's going on.
(Assignee)

Comment 1

5 years ago
Created attachment 689215 [details] [diff] [review]
Temporarily add some info about binaries and debug info files
Attachment #689215 - Flags: review?(ted)
(Assignee)

Updated

5 years ago
Assignee: nobody → mh+mozilla
Attachment #689215 - Flags: review?(ted) → review+
(Assignee)

Comment 3

5 years ago
So, if I take a random file from last nightly's log, the output added for it is:
Contents of section .gnu_debuglink:
 0000 66697265 666f782e 64626700 d5208b6a  firefox.dbg.. .j
./dist/bin/firefox.dbg crc: 6a8b20d5

At this point (so, right after objcopy), the crc matches (d5208b6a == 6a8b20d5, the value in .gnu_debuglink being a dumped little endian value).

(log is https://tbpl.mozilla.org/php/getParsedLog.php?id=17708635&tree=Firefox )

Its full path is firefox/1DC5137538355C0921B17717453612A50/firefox.dbg.gz
If I download this nightly, and check the firefox binary, I can see the .gnu_debuglink section has not changed:
Contents of section .gnu_debuglink:
 0000 66697265 666f782e 64626700 d5208b6a  firefox.dbg.. .j

However, if I download http://symbols.mozilla.org/firefox/firefox/1DC5137538355C0921B17717453612A50/firefox.dbg.gz, gunzip -lv gives me:
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla 74ce102f Nov 14 21:21              196552              523844  62.5% firefox.dbg

Something is either modifying firefox.dbg between the objcopy and the symbol full archive creation, or something is modifying firefox.dbg on the server.
(Assignee)

Comment 4

5 years ago
In fact, the gunzip -lv output makes it striking obvious: the firefox.dbg is old. Which means its id,
1DC5137538355C0921B17717453612A50, matches several different firefoxes.
(Assignee)

Comment 5

5 years ago
That could also explain some funky stack traces on Android...
(Assignee)

Comment 6

5 years ago
Created attachment 690067 [details] [diff] [review]
Link with --build-id when available

Let's just switch to build ids. Breakpad supports it, and I think our toolchains now do, too (thanks to mock) ; the NDK should support it too.

https://tbpl.mozilla.org/?tree=Try&rev=71643686641b
Attachment #690067 - Flags: review?(ted)
(Assignee)

Updated

5 years ago
Attachment #689215 - Attachment is obsolete: true
Comment on attachment 690067 [details] [diff] [review]
Link with --build-id when available

Review of attachment 690067 [details] [diff] [review]:
-----------------------------------------------------------------

That makes a lot of sense. With the hash of the .text section you'd get identical debug IDs whenever the code compiles the same.
Attachment #690067 - Flags: review?(ted) → review+
(Assignee)

Comment 8

5 years ago
http://hg.mozilla.org/mozilla-central/rev/d62b457df9a7
http://hg.mozilla.org/mozilla-central/rev/4e83d0987a31
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Whiteboard: [leave open]
Target Milestone: --- → mozilla20
(Assignee)

Comment 9

5 years ago
This doesn't apply to nspr and nss :-/
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Mike Hommey [:glandium] from comment #5)
> That could also explain some funky stack traces on Android...

Do you have any examples of these? (Just wondering if it affected any of the stacks seen on TBPL).
(Assignee)

Comment 11

5 years ago
(In reply to Ed Morley [UTC+0; email:edmorley@moco] from comment #10)
> (In reply to Mike Hommey [:glandium] from comment #5)
> > That could also explain some funky stack traces on Android...
> 
> Do you have any examples of these? (Just wondering if it affected any of the
> stacks seen on TBPL).

I don't.
(Assignee)

Comment 12

5 years ago
Created attachment 690331 [details] [diff] [review]
Also link NSPR and NSS with --build-id when available
Attachment #690331 - Flags: review?(ted)
(In reply to Ed Morley [UTC+0; email:edmorley@moco] from comment #10)
> Do you have any examples of these? (Just wondering if it affected any of the
> stacks seen on TBPL).

This should have no impact on TBPL, since stack traces there are generated using Breakpad symbols which are matched with the build being used for testing.
Comment on attachment 690331 [details] [diff] [review]
Also link NSPR and NSS with --build-id when available

Review of attachment 690331 [details] [diff] [review]:
-----------------------------------------------------------------

We could also port this configure test to NSPR if you wanted.
Attachment #690331 - Flags: review?(ted) → review+
(Assignee)

Comment 15

5 years ago
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #14)
> Comment on attachment 690331 [details] [diff] [review]
> Also link NSPR and NSS with --build-id when available
> 
> Review of attachment 690331 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> We could also port this configure test to NSPR if you wanted.

That would be more hassle to uplift.
https://hg.mozilla.org/mozilla-central/rev/3cb1250425c0
https://hg.mozilla.org/mozilla-central/rev/508d1f5c60b0
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED

Updated

2 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.