Closed Bug 1794095 Opened 3 years ago Closed 3 years ago

[traceback] SymCacheErrorBadDebugFile: Unhandled bad debug file caused by: parsing error caused by: in section "inline origin record body"

Categories

(Eliot :: General, defect, P2)

Product:
Eliot
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

References

Details

Attachments

(2 files)

pr 2620: bug 1794095: add sym file manipulation scripts
52 bytes, text/x-github-pull-request
Details | Review
pr 2644: bug 1794095: handle SymCacheErrorBadDebugFile errors
52 bytes, text/x-github-pull-request
Details | Review

Sentry: https://sentry.io/organizations/mozilla/issues/3608028371/?query=is%3Aunresolved

Traceback:

SymCacheErrorBadDebugFile: bad debug file
  caused by: parsing error
  caused by: in section "inline origin record body" at 
"INLINE_ORIGIN 579273"
               ^
,
in section "inline origin id" at 
"INLINE_ORIGIN 579273"
               ^
,
expected whitespace at 
"INLINE_ORIGIN 579273"
                     ^

  File "falcon/app.py", line 365, in falcon.app.App.__call__
    responder(req, resp, **params)
  File "markus/main.py", line 515, in _timer_decorator
    return fun(*args, **kwargs)
  File "eliot/symbolicate_resource.py", line 648, in on_post
    results.append(self.symbolicate(stacks, modules, debug_stats))
  File "eliot/symbolicate_resource.py", line 432, in symbolicate
    self.get_symcache(module_info, debug_stats)
  File "eliot/symbolicate_resource.py", line 350, in get_symcache
    symcache = self.parse_sym_file(debug_filename, debug_id, sym_file)
  File "markus/main.py", line 515, in _timer_decorator
    return fun(*args, **kwargs)
  File "eliot/symbolicate_resource.py", line 281, in parse_sym_file
    return parse_sym_file(debug_filename, debug_id, data, self.tmpdir)
  File "eliot/libsymbolic.py", line 129, in parse_sym_file
    symcache = obj.make_symcache()
  File "symbolic/debuginfo.py", line 133, in make_symcache
    self._methodcall(lib.symbolic_symcache_from_object)
  File "symbolic/utils.py", line 57, in _methodcall
    return rustcall(func, self._get_objptr(), *args)
  File "symbolic/utils.py", line 92, in rustcall
    raise exc

I talked with gsvelto about this and we wrote up bug #1791785 that eventually led to a fix in dump_syms. However, we're still seeing the problem, so this bug is about manually fixing the problematic symbols files.

Yesterday, I went through the Sentry events for October and made this list:

  • 441450BE1030C69946EC77871CA83C5D0 libglib-2.0.so.0
  • E32113BBF16D446E4469E4E2FB57323B0 libgobject-2.0.so.0
  • 435BCC7E34F1C98450F4BABE863BDBA80 libsoftokn3.so
  • 9B39A7109EF1C7067EE75F075686EB5D0 libplc4.so
  • 40A596D5CD5E58199912260DB81747880 libfreeblpriv3.so
  • 2CBA64CC6D0E57AB7F6272B686B839D70 libplds4.so

They all fail to parse with symbolic because there's an INLINE_ORIGIN line that has an index, but no name. Something like this:

INLINE_ORIGIN 3

Removing that line allows the files to be parsed and then everything seems fine after that.

I wrote some scripts to download files, let me fix them, verify that they parse, and then upload them which stomps on the problematic file.

https://symbols.mozilla.org/libgobject-2.0.so.0/E32113BBF16D446E4469E4E2FB57323B0/libgobject-2.0.so.0.sym
https://symbols.mozilla.org/libplds4.so/2CBA64CC6D0E57AB7F6272B686B839D70/libplds4.so.sym
https://symbols.mozilla.org/libfreeblpriv3.so/40A596D5CD5E58199912260DB81747880/libfreeblpriv3.so.sym
https://symbols.mozilla.org/libglib-2.0.so.0/441450BE1030C69946EC77871CA83C5D0/libglib-2.0.so.0.sym
https://symbols.mozilla.org/libsoftokn3.so/435BCC7E34F1C98450F4BABE863BDBA80/libsoftokn3.so.sym
https://symbols.mozilla.org/libplc4.so/9B39A7109EF1C7067EE75F075686EB5D0/libplc4.so.sym

New problematic file since yesterday:

  • CF951B6F53184F670E0966A66CD0D9520 libxul.so
Assignee: nobody → willkg
Status: NEW → ASSIGNED
Depends on: 1791785

Gabriele: This recent problematic file is a libxul.so. Does the fact that it's not a system library suggest we've got other scenarios where the INLINE_ORIGIN doesn't have a name?

Also, if I fix this file by removing the INLINE_ORIGIN line, what should I do with INLINE lines that reference it? Should I remove them, too? Should I just give the INLINE_ORIGIN a name of "unknown" or something like that instead?

Flags: needinfo?(gsvelto)

Gabriele urged us to fix dump_syms and not hand-fix sym files which makes sense especially if there were a lot of broken files and hand fixing took a while. I think in this case, I want Eliot to stop kicking up parse issues and the number of files to fix is small and it's pretty straight-forward to do.

The fix Gabriele did in dump_syms is here:

https://github.com/mozilla/dump_syms/commit/c7df854b953b6ddc875afaa415ef1fb2f7bf01f6#diff-acd39ffa6b9248a1cb5527531c92395bb4508638766f31fdc081c1e095a66e09R62-R66

Instead of removing the INLINE_ORIGIN lines, I should instead give them a name of <name omitted>. I'll refix the ones I fixed before and fix the new one accordingly.

Flags: needinfo?(gsvelto)

I re-added the INLINE_ORIGIN lines I removed with the <name omitted> name and uploaded all the fixed files.

I'll keep this open for a week to catch additional problems.

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #2)

Gabriele: This recent problematic file is a libxul.so. Does the fact that it's not a system library suggest we've got other scenarios where the INLINE_ORIGIN doesn't have a name?

I suspect these are from one of Fedora builds for which we started dumping inlined functions like we do for our builds. Fedora splits the debug information of Firefox in two packages: one with the DWARF debug info but no strings, and one with Firefox sources. The DWARF info points to the sources and dump_syms doesn't support following those links, so we're often missing function names when dumping those builds.

I fixed:

  • B68D6DF6915DDCBD20F569A412D6886E0 libssl3.so

I fixed:

  • CDEB344D4860E40AB7E154F05189CC730 libnssutil3.so

I fixed:

  • 015C5165CC511800A25E59423F0D5CB40 libxul.so
  • 7792EBC12F9B97342563B21F4F0C0D4A0 firefox

I fixed:

  • C1E3C105E3B1B521F24D8C37A0F3CF700 libnssckbi.so

I fixed:

  • 3A4AF6D359B804DE5E124A02713562960 libcups.so.2

It's been a while since I fixed one of these, so I'm going to mark this FIXED and if any other issues come up, I'll continue to fix them manually.

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Component: Symbolication → General
Product: Tecken → Eliot

I've fixed a bunch more files. The most recent one was due to the symbol being entirely whitespace. I opened up this issue to cover that:

https://github.com/mozilla/dump_syms/issues/543

That was fixed in dump_syms 2.2.0. Then I fixed:

  • libxul.so 5AC6F470F832952B3246C5EADF3017AA0

I fixed glxtrace.so 63A0092F5FE8497FB1B7CC9E59E031AF0

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: