974649 - Add environment variable for crash auto-submission (without the crash reporter open)

Status: RESOLVED INVALID

(Toolkit :: Crash Reporting, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It would be great if we could add another environment variable to allow the auto-submission of crash reports to soccorro. This request is made for various automation frameworks, because none of us can handle the crash reporter. It doesn't close itself, and for really high-frequent crashes affected nodes can have tons of those windows open.

A workaround for now is to set MOZ_CRASHREPORTER_DISABLE=1 and to process the crash via the minidump_stackwalk binary. But it would be much easier for us if that could be automatically done if requested.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The above would be really important for beta and release builds because of the lack of any crashreporter-symbols on the FTP server. For Aurora and Nightly builds it has been requested via bug 973901.

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

Mass-removing myself from cc; search for 12b9dfe4-ece3-40dc-8d23-60e179f64ac1 or any reasonable part thereof, to mass-delete these notifications (and sorry!)

Comment 3

[Alex Franchuk [:afranchuk]]

Gabriele I think this is covered by MOZ_CRASHREPORTER_AUTO_SUBMIT, unless I'm missing something?

Comment 4

[Gabriele Svelto [:gsvelto]]

Yes, it should be. Henrik, is this bug still relevant? As Alex mentioned in the previous comment, setting the MOZ_CRASHREPORTER_AUTO_SUBMIT environment variable will make the crash report to not become visible, but submit the crash to Socorro automatically. This only applies to main process crashes.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I think that over time (I can't believe it's been 11 years since I filed this bug!), we concluded that we shouldn't automatically submit crash reports but rather process them locally. Auto-submitting would result in flooding Socorro with numerous crash reports that we don't need. Therefore, for automation purposes, automatic submission isn't necessary.

However, there might be instances when local crash analysis fails, and we need to request a crash report from a user. Currently, several steps are involved in obtaining the details of a report, or we need to request the minidump files for processing ourselves.

Does MOZ_CRASHREPORTER_AUTO_SUBMIT override the other crash reporter-related environment variables and auto-submit even if the reporter is generally turned off? Probably not.

Comment 6

[Gabriele Svelto [:gsvelto]]

(In reply to Henrik Skupin [:whimboo][⌚️UTC+1] from comment #5)

I think that over time (I can't believe it's been 11 years since I filed this bug!), we concluded that we shouldn't automatically submit crash reports but rather process them locally. Auto-submitting would result in flooding Socorro with numerous crash reports that we don't need. Therefore, for automation purposes, automatic submission isn't necessary.

OK, I think the only issue we still have WRT to automation is bug 867571, that is, make it possible to search for crashes on Socorro (submitted by users) starting from crashes that are present in automation. Right now and until we fix bug 867571 the only way to do it is download the minidump & symbols from the failed task, run rust-minidump locally w/ symbols then manually search the resulting stack frames on Socorro (or run socorr-siggen's signature command on the crash).

Does MOZ_CRASHREPORTER_AUTO_SUBMIT override the other crash reporter-related environment variables and auto-submit even if the reporter is generally turned off? Probably not.

No, it's a no-op if the crash reporter is disabled.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Gabriele Svelto [:gsvelto] (PTO) from comment #6)

OK, I think the only issue we still have WRT to automation is bug 867571, that is, make it possible to search for crashes on Socorro (submitted by users) starting from crashes that are present in automation. Right now and until we fix bug 867571 the only way to do it is download the minidump & symbols from the failed task, run rust-minidump locally w/ symbols then manually search the resulting stack frames on Socorro (or run socorr-siggen's signature command on the crash).

Yes, that's what we currently do but directly when running the tasks in CI. So there is no need to download the minidump and symbols. As result we get the stack printed in the logs. IMO that's what we actually want for CI jobs? But where I could see it useful is for crashes that can be experienced by other users and to assist them to get reported correctly to Soccoro. So lets wait for bug 867571? Do we have to keep this bug open?

Btw. did we already replace the old minidump executable as used by mozcrash with the newer one written in Rust yet when running tests in CI? If not we potentially should file a bug?

Comment 8

[Gabriele Svelto [:gsvelto]]

(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #7)

Yes, that's what we currently do but directly when running the tasks in CI. So there is no need to download the minidump and symbols. As result we get the stack printed in the logs. IMO that's what we actually want for CI jobs? But where I could see it useful is for crashes that can be experienced by other users and to assist them to get reported correctly to Soccoro. So lets wait for bug 867571? Do we have to keep this bug open?

No, I think we can close it.

Btw. did we already replace the old minidump executable as used by mozcrash with the newer one written in Rust yet when running tests in CI? If not we potentially should file a bug?

Yes! We've replaced the old one a while ago, so we're running the exact same tool on CI, socorro and locally.