1920956 - Missing crash annotations on a test after Bug 1634785

Status: RESOLVED INVALID

(Toolkit :: Crash Reporting, defect)

Description

[Nazım Can Altınova [:canova]]

I was trying to land Bug 1634785, but it was backed out due to a web extension test failure, more specifically the toolkit/components/extensions/test/xpcshell/test_ext_background_early_shutdown.js test.

Example logs with some debug logs: https://treeherder.mozilla.org/logviewer?job_id=474911428&repo=try&lineNumber=13130-13173

Upon investigating I noticed that this test was generating some intentional/synthetic crashes. And for some reason, after these patches we are not properly annotating the crash with TestKey=CrashFrame so the test can't safely ignore these crashes. Also all the crashes have "DumperError: MissingAnnotations" key set, that's probably why TestKey is not present.

It's also only failing on Linux opt tests in CI. I'm not able to reproduce this locally yet.
I tried to run the test with this:

./mach test toolkit/components/extensions/test/xpcshell/test_ext_background_early_shutdown.js --tag=remote-webextensions --verbose

Note that the --tag is important here because otherwise the test doesn't properly trigger the crashes.

:gsvelto pointed out that the CI is using a different dynamic linker and it can be because of that, but it's to be investigated.

I added some debug logs (in this treeherder push) to investigate and it looks like this is the root cause:

[task 2024-09-18T23:35:25.334Z] 23:35:25     INFO -  PID 12698 | canova failed to verify the annotation table. magic number: 7752708
[task 2024-09-18T23:35:25.334Z] 23:35:25     INFO -  PID 12698 | canova error retrieving annotations. err: InvalidAnnotationTable

It fails here while trying to verify the annotation table: https://searchfox.org/mozilla-central/rev/af3eccb4999427e42fdcea558297f1cec70dc4db/toolkit/crashreporter/mozannotation_server/src/lib.rs#87-89

So I'm not so sure how this could happen, verify checks the magic number and these appear to be different, 7752708 as oppose the correct value of 1096437581.

I'm unable to continue investigating this further without properly trying to reproduce. Any pointers or help would be appreciated!

Comment 1

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:gsvelto, could you have a look please?

For more information, please visit BugBot documentation.

Comment 2

[Nazım Can Altınova [:canova]]

I spent some more time yesterday to debug this issue and realized that it's actually because of the profiler code and not the crashreporter code (oops!). Apparently it surfaced on the crashreporter code by just pure luck. I filed bug 1932580 for that instead. Sorry for the noise! Let's close this one.

Comment 3

[Gabriele Svelto [:gsvelto]]

Glad you figured it out, I was a bit worried this had stalled.

Comment 4

[Nazım Can Altınova [:canova]]

No worries, I was busy with other things in the meantime so wasn't really stuck on this. And it turned out a profiler issue after all :) Thanks for the help on investigation!