1870823 - Add product name, version, channel and build id to the symbol files

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, enhancement)

Description

[Gabriele Svelto [:gsvelto]]

In order to make it simpler to filter symbol files we want to add more information in the form of INFO lines. The end goal is to have symbols that look like this:

MODULE Linux x86_64 1F246A128A9B4CBE2CF6768834D701A90 firefox
INFO CODE_ID 126A241F9B8ABE4C2CF6768834D701A9
INFO VENDOR Mozilla
INFO PRODUCTNAME Firefox
INFO RELEASECHANNEL default
INFO VERSION 122.0a1
INFO BUILDID 20231219143824
INFO GENERATOR mozilla/dump_syms 2.2.2
FILE 0 hg:hg.mozilla.org/mozilla-central:toolkit/xre/CmdLineAndEnvUtils.h:55cd203304b8d5343dd941d772085dc25fd278c4
FILE 1 hg:hg.mozilla.org/mozilla-central:mfbt/Assertions.h:55cd203304b8d5343dd941d772085dc25fd278c4
...

The VENDOR, PRODUCTNAME, RELEASECHANNEL, VERSION and BUILDID will match those of the built the symbol file comes from.

Comment 1

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1870823 - Add product name, version, release channel and build id to the symbol files r=gerard-majax

Comment 2

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1870823 - Fix remaining linting issues r=gerard-majax

Depends on D196838

Comment 3

[Gabriele Svelto [:gsvelto]]

Will, Yannis, does this look like enough info for addressing bug 1861359 and similar issues that involve knowing to which build a certain .sym file belongs to?

Comment 4

[Gabriele Svelto [:gsvelto]]

Note that this is a WIP because I haven't cut a new release of dump_syms with the required functionality yet.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think this is what we need.

This covers symbols files for Mozilla products. I think we'd also want to add this information to symbols files generated by the symbols scrapers. Given that these fields are Mozilla-centric, what would the header for something like ntdll.dll look like? Maybe something like this where it only adds the VERSION bit?

MODULE windows x86_64 4E2FE9F5853741CE92B4B76674057EEE1 ntdll.pdb
INFO CODE_ID 5CB9356E19F000 ntdll.dll
INFO VERSION 6.1.7601.24441
INFO GENERATOR mozilla/dump_syms 2.2.0
PUBLIC 1020 0 NtDllUserStubs
PUBLIC 12f0 0 _mask@@NegDouble@

Comment 6

[Gabriele Svelto [:gsvelto]]

Yes, that's definitely doable in the Windows symbol scraper. macOS and Linux are a bit trickier but I think we can do something there too. To avoid confusion we might even use a different field such as DLLVERSION given that Firefox DLL's could have both the regular product version and a DLL version.

Comment 7

[Yannis Juglaret [:yannis]]

I agree that it is important to have the "DLL" (also applies for firefox.exe) version in that list, and to have it for Mozilla binaries, and that it should be its own field. In bug 1861359 we'd like to check for coherence between the DLL version, the code ID, and the debug ID so we will be most successful with all three available. By adding this field to the list from comment 0 I agree we should have all we need.

Note that it has happened in the past that we released two different binaries with the same DLL version (thereby causing bug 1843977), so that will be something to remember when we write the code that should detect incoherence (DLL version should not be treated as unique - even though we’d like it to be).

Comment 8

[Gabriele Svelto [:gsvelto]]

Adding the DLL version is gonna take a bit more as I need to add some extra functionality to dump_syms. Let's land this as is with the VENDOR, PRODUCTNAME, RELEASECHANNEL, VERSION and BUILDID. I'll open a follow up for the DLL versions.

Comment 9

[Gabriele Svelto [:gsvelto]]

Stealing this back :)

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2722d316776f
Add product name, version, release channel and build id to the symbol files r=gerard-majax
https://hg.mozilla.org/integration/autoland/rev/20014fe22fc5
Fix remaining linting issues r=gerard-majax

Comment 11

[Narcis Beleuzu [:NarcisB]]

Backed out for toolchain bustages

Backout link: https://hg.mozilla.org/integration/autoland/rev/cfe658475285f9b5c867bcbd191d2e1332df885e
Log link: https://treeherder.mozilla.org/logviewer?job_id=449907275&repo=autoland&lineNumber=1091

Comment 12

[Gabriele Svelto [:gsvelto]]

The macOS and Windows build use an older rust toolchain compared to the Linux one, I need to update their task definitions.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e642f77696be
Add product name, version, release channel and build id to the symbol files r=gerard-majax
https://hg.mozilla.org/integration/autoland/rev/9b3ea3c9cf52
Fix remaining linting issues r=gerard-majax

Comment 14

[Natalia Csoregi [:nataliaCs]]

Backed out for causing spidermonkey bustage on symbolstore.py

• backout: https://hg.mozilla.org/integration/autoland/rev/343e945a502ea35f40687ffca3eab7543522fe79
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=ENA7NEkSRTa-mRIfyOYujQ.0&revision=f64e162799d9bd2165756c4d7690068eb14545af
• failure log: https://treeherder.mozilla.org/logviewer?job_id=450284856&repo=autoland&lineNumber=10412

[task 2024-03-11T09:05:24.917Z] Unexpected error: [Errno 2] No such file or directory: '/builds/worker/workspace/obj-spider/buildid.h'
[task 2024-03-11T09:05:24.917Z]     main()
[task 2024-03-11T09:05:24.917Z]   File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 1107, in main
[task 2024-03-11T09:05:24.917Z] Traceback (most recent call last):
[task 2024-03-11T09:05:24.917Z]   File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 1112, in <module>
[task 2024-03-11T09:05:24.917Z]     dumper.Process(args[2], options.count_ctors)
[task 2024-03-11T09:05:24.917Z]   File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 532, in Process
[task 2024-03-11T09:05:24.917Z]     main()
[task 2024-03-11T09:05:24.917Z]   File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 1107, in main
[task 2024-03-11T09:05:24.917Z]     self.ProcessFile(file_to_process, count_ctors=count_ctors)
[task 2024-03-11T09:05:24.917Z]   File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 546, in ProcessFile
[task 2024-03-11T09:05:24.917Z]     dumper.Process(args[2], options.count_ctors)
[task 2024-03-11T09:05:24.917Z]     self.ProcessFileWork(
[task 2024-03-11T09:05:24.917Z]   File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 584, in ProcessFileWork
[task 2024-03-11T09:05:24.917Z]   File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 532, in Process
[task 2024-03-11T09:05:24.917Z]     cmd = self.dump_syms_cmdline(file, arch, dsymbundle=dsymbundle)
[task 2024-03-11T09:05:24.917Z]      self.ProcessFile(file_to_process, count_ctors=count_ctors)
[task 2024-03-11T09:05:24.917Z]        ^^^^^^^^^  File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 546, in ProcessFile
[task 2024-03-11T09:05:24.917Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2024-03-11T09:05:24.917Z]   File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 555, in dump_syms_cmdline
[task 2024-03-11T09:05:24.917Z]     self.ProcessFileWork(
[task 2024-03-11T09:05:24.917Z]   File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 584, in ProcessFileWork
[task 2024-03-11T09:05:24.917Z]     buildid = io.open(path, "r", encoding="utf-8").read().split()[2]
[task 2024-03-11T09:05:24.917Z]               ^^^^^^^^^^^^^^^^^^^    ^cmd = self.dump_syms_cmdline(file, arch, dsymbundle=dsymbundle)^
[task 2024-03-11T09:05:24.917Z] ^^^^^^^^^^^^^^^
[task 2024-03-11T09:05:24.917Z] FileNotFoundError  :  [Errno 2] No such file or directory: '/builds/worker/workspace/obj-spider/buildid.h'
[task 2024-03-11T09:05:24.917Z]       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2024-03-11T09:05:24.917Z]   File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 555, in dump_syms_cmdline
[task 2024-03-11T09:05:24.917Z]     buildid = io.open(path, "r", encoding="utf-8").read().split()[2]
[task 2024-03-11T09:05:24.917Z]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2024-03-11T09:05:24.917Z] FileNotFoundError: [Errno 2] No such file or directory: '/builds/worker/workspace/obj-spider/buildid.h'

Comment 15

[Gabriele Svelto [:gsvelto]]

On it, there were some issues with the macOS builds too.

Comment 16

[Gabriele Svelto [:gsvelto]]

This try run has all the builds for all the platforms and I've manually verified all the symbol archives. I just need to double-check the SpiderMonkey builds before trying to land again.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bf23686d4852
Add product name, version, release channel and build id to the symbol files r=gerard-majax
https://hg.mozilla.org/integration/autoland/rev/8e71522c5b0c
Fix remaining linting issues r=gerard-majax

Comment 18

[Sandor Molnar[:smolnar]]

Backed out for causing build bustages

Backout link: https://hg.mozilla.org/integration/autoland/rev/703ecd7e3512e96d3ccecd50429d35f9599f8fb1

Push with failures

Failure log -> Unexpected error: sequence item 1: expected str instance, list found

Comment 19

[Gabriele Svelto [:gsvelto]]

I thought I had run all the possible build combinations but I probably didn't since this is an obvious failure. Here's a new try run to see if my fix works correctly.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ddf78baddede
Add product name, version, release channel and build id to the symbol files r=gerard-majax
https://hg.mozilla.org/integration/autoland/rev/7e8d750c1ade
Fix remaining linting issues r=gerard-majax

Comment 21

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/ddf78baddede
https://hg.mozilla.org/mozilla-central/rev/7e8d750c1ade