1717909 - System symbols scraped from Windows symbol servers are missing CFI information

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Gabriele Svelto [:gsvelto]]

I was looking at some recent symbols that were scraped from Microsoft's servers and noticed that they are missing CFI information:

• ntdll.sym
• kernelbase.sym
• combase.sym
• rpcrt4.sym

This suggests a bug in the symbol scraping task. I'll try to investigate today or tomorrow at the latest.

Comment 1

[Gabriele Svelto [:gsvelto]]

This is the task that generated the ntdll.sym symbols in comment 0. Now to figure out what went wrong...

Comment 2

[Gabriele Svelto [:gsvelto]]

This is super silly. We're making too many connections to the symbol server and silently failing, this is in the task log:

[task 2021-06-23T00:03:15.609Z] WARNING:root:Error with https://msdl.microsoft.com/download/symbols/MLogHook.pdb/AE890A7047524698BC2CC65A0AC24C631/MLogHook.pdb: retry
[task 2021-06-23T00:03:15.610Z] ERROR:root:Cannot connect to host msdl.microsoft.com:443 ssl:default [Connection reset by peer]
[task 2021-06-23T00:03:15.610Z] Traceback (most recent call last):
[task 2021-06-23T00:03:15.610Z]   File "/usr/local/lib/python3.7/dist-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
[task 2021-06-23T00:03:15.610Z]     return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
[task 2021-06-23T00:03:15.610Z]   File "/usr/lib/python3.7/asyncio/base_events.py", line 986, in create_connection
[task 2021-06-23T00:03:15.610Z]     ssl_handshake_timeout=ssl_handshake_timeout)
[task 2021-06-23T00:03:15.610Z]   File "/usr/lib/python3.7/asyncio/base_events.py", line 1014, in _create_connection_transport
[task 2021-06-23T00:03:15.610Z]     await waiter
[task 2021-06-23T00:03:15.610Z]   File "/usr/lib/python3.7/asyncio/selector_events.py", line 801, in _read_ready__data_received
[task 2021-06-23T00:03:15.610Z]     data = self._sock.recv(self.max_size)
[task 2021-06-23T00:03:15.610Z] ConnectionResetError: [Errno 104] Connection reset by peer

We have to further slow down how many symbols we fetch at the same time.

Comment 3

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1717909 - Fetch only one file at a time from Microsoft's symbol servers; r=marco

Comment 4

[Calixte Denizet (:calixte)]

I was expecting that 4 is not so bad and if we miss few pdbs then we can get them the day after.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5d32419d6b5b
Fetch only one file at a time from Microsoft's symbol servers; r=marco,calixte

Comment 6

[Calixte Denizet (:calixte)]

After thinking about that:

• I'm finally no sure that the fix here is correct:
  • we check that the file exists on one of the symbol server: https://searchfox.org/mozilla-central/source/tools/crashreporter/system-symbols/win/symsrv-fetch.py#79 and we retry several times in case network error (and in this case the error is logged).
  • once we know where the files are, we get them: https://searchfox.org/mozilla-central/source/tools/crashreporter/system-symbols/win/symsrv-fetch.py#108

if we don't manage to get a file we expect to get it in next run, so finally I think that reducing the limit_per_host will just increase the time spent in this task with no special benefit.
Anyway for x86_64 target, CFI info are in the dll:
https://searchfox.org/mozilla-central/source/tools/crashreporter/system-symbols/win/symsrv-fetch.py#297

so I'd say that we manage to get the pdb, for any reason we don't have the dll/exe for it and then we finally have partial data in .sym.
Or maybe for any reason the dll/exe doesn't contain CFI data (is it possible ??) and so we've pdb & dll but no CFI in the .sym.
Normally we don't produce the .sym when the dll is missing (only for x86_64 target).

Maybe it's possible to not have CFI info when exception stuff is disabled at compile time because in this case CFI are pretty useless, but maybe I'm wrong.
If I'm correct the fix could be to add an option in dump_syms to fail in case of missing CFI data, because it's useless for stack walker.

Comment 7

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/5d32419d6b5b

Comment 8

[Gabriele Svelto [:gsvelto]]

(In reply to Calixte Denizet (:calixte) from comment #6)

After thinking about that:

• I'm finally no sure that the fix here is correct:
  • we check that the file exists on one of the symbol server: https://searchfox.org/mozilla-central/source/tools/crashreporter/system-symbols/win/symsrv-fetch.py#79 and we retry several times in case network error (and in this case the error is logged).
  • once we know where the files are, we get them: https://searchfox.org/mozilla-central/source/tools/crashreporter/system-symbols/win/symsrv-fetch.py#108

if we don't manage to get a file we expect to get it in next run, so finally I think that reducing the limit_per_host will just increase the time spent in this task with no special benefit.
Anyway for x86_64 target, CFI info are in the dll:
https://searchfox.org/mozilla-central/source/tools/crashreporter/system-symbols/win/symsrv-fetch.py#297

I see, so I have misinterpreted those, they're just warnings, right?

so I'd say that we manage to get the pdb, for any reason we don't have the dll/exe for it and then we finally have partial data in .sym.
Or maybe for any reason the dll/exe doesn't contain CFI data (is it possible ??) and so we've pdb & dll but no CFI in the .sym.
Normally we don't produce the .sym when the dll is missing (only for x86_64 target).

Maybe it's possible to not have CFI info when exception stuff is disabled at compile time because in this case CFI are pretty useless, but maybe I'm wrong.
If I'm correct the fix could be to add an option in dump_syms to fail in case of missing CFI data, because it's useless for stack walker.

Well, there's definitely something odd going on. If I look at what happened with the ntdll.dll from comment 0 I find these two lines in the task output:

[task 2021-06-23T00:05:37.729Z] INFO:root:To dump: ntdll.pdb/7FCBBFA95B3DDBC61A90F7C3FD4A7F371, ntdll.dll/088BF6211f5000 and has_code = False
[task 2021-06-23T00:05:37.729Z] INFO:root:To dump: ntdll.pdb/7FCBBFA95B3DDBC61A90F7C3FD4A7F371, / and has_code = 

They're odd, they look like some kind of duplication. Additionally we seem to download the PDB twice:

[task 2021-06-23T00:05:37.816Z] DEBUG:root:Fetch url: https://msdl.microsoft.com/download/symbols/ntdll.pdb/7FCBBFA95B3DDBC61A90F7C3FD4A7F371/ntdll.pdb
[task 2021-06-23T00:05:37.816Z] DEBUG:root:Fetch url: https://msdl.microsoft.com/download/symbols/ntdll.pdb/7FCBBFA95B3DDBC61A90F7C3FD4A7F371/ntdll.pdb

Additionally the associated DLL is ntdll.dll with code id 088BF6211f5000 and it seems that it cannot be found. I tried to manually scrape it again today and it's not there either. So yeah, it seems like the correct fix is to fail if we only have the PDB, so that the scripts will try to re-scrape the file at a later time and hopefully will find the corresponding DLL. I'm re-opening this.

Comment 9

[Gabriele Svelto [:gsvelto]]

I removed the tracking flags because this only affects our automation scripts, not Firefox per se.

Comment 10

[Gabriele Svelto [:gsvelto]]

Handing this over to Calixte for the actual fix.

Comment 11

[Gabriele Svelto [:gsvelto]]

I discovered something funny: sometimes we can't seem to find a PDB in Microsoft's symbol servers, but if we look for the DLL we also get the PDB. Here's an example, this command:

dump_syms rpcrt4.pdb --code-id 5DA53549140000 --store output --symbol-server "SRV*output*https://msdl.microsoft.com/download/symbols" --verbose error

Fails with:

Impossible to get file rpcrt4.pdb with id 5DA53549140000

But this one succeeds and also finds the corresponding PDB:

dump_syms rpcrt4.dll --code-id 5DA53549140000 --store output --symbol-server "SRV*output*https://msdl.microsoft.com/download/symbols" --verbose error

Maybe we should try both before giving up?

Comment 12

[Gabriele Svelto [:gsvelto]]

Duh, I forgot to post the fix for this one.

Comment 13

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1717909 - Ensure that symbol files for system libraries on Windows contain unwinding information r=calixte

This also reverts 5d32419d6b5b

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bb0fd62efdd4
Ensure that symbol files for system libraries on Windows contain unwinding information r=calixte

Comment 15

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/bb0fd62efdd4

Comment 16

[Gabriele Svelto [:gsvelto]]

FYI this is working as expected and catching a rather large number of instances where CFI data is missing, see this report.