1835524 - Use BugSplat for SeaMonkey crash reporting

Status: RESOLVED FIXED

(SeaMonkey :: Release Engineering, task)

Description

[Ian Neal]

We are planning to switch to BugSplat for crash reporting.
Provisionally we will use the existing breakpad as that will probably easier than putting in crashpad.

The .mozconfig file will need updating to use BugSplat:
ac_add_options --with-crashreporter-url=https://seamonkey_project.bugsplat.com/post/bp/crash/postBP.php

Still need to workout how to get all the information visible that we want in BugSplat.

Comment 1

[Ian Neal]

Attachment: WIP-1835524-bugsplat-25317.patch

At the moment I have:

• Added and tested building/use of symupload from breakpad tools for Linux.
• Added building of symupload from breakpad tools for mac/windows but these to be checked/tested.
• Put some temporary mapping of parameters to get the crash uploads working.
• Created a script for uploading crash reporter symbols locally but this will probably need converting into mach/make format to use on the builders.
• Tested both a crash report and symbols upload to BugSplat successfully

Still to be done:

• Find out from BugSplat how to get full visibility of all the parameters we upload with the mini-dump.
• Find out why line numbers and file names are not displaying in BugSplat Overview.

Comment 2

[Ian Neal]

Attachment: sym-upload.sh

Temporary linux shell script for symbol uploads

Comment 3

[Ian Neal]

Attachment: 1835524-bugsplat-mr-25318.patch

[Approval Request Comment]
Regression caused by (bug #): n/a
User impact if declined: no crashreports uploaded
Testing completed (on m-c, etc.): 2.53.18 linux
Risk to taking this patch (and alternatives if risky): low
String changes made by this patch: none

BugSplat requires different parameters for upload and the response from the server is in a different format.

We also need to build symupload so that releases can upload the symbols to the server.

The command to upload would be:
symupload <symfilename> "https://seamonkey_project.bugsplat.com/post/bp/symbol/breakpadsymbols.php?appName=SeaMonkey&appVer=<version>"
<version> is built from PackageVersion-BuildID e.g. SeaMonkey-2.53.18b1pre-20230910134258

The .mozconfig needs to contain:
ac_add_options --with-crashreporter-url=https://seamonkey_project.bugsplat.com/post/bp/crash/postBP.php

As BugSplat doesn't allow custom fields, I have combined all the information we might need into the comments field.
At the moment the application name is set to "SeaMonkey" so all crashreports for all versions would appear in one place. Another option would be to split by release channel or by version, so the application name could be SeaMonkey-nightly / SeaMonkey-beta / SeaMonkey-release or SeaMonkey-2.53.18b1pre / SeaMonkey-2.53.18b1 / SeaMonkey-2.53.18

Comment 4

[Ian Neal]

Attachment: 1835524-bugsplat-cr-25318.patch

[Approval Request Comment]
Regression caused by (bug #): n/a
User impact if declined: links in about crashes point to wrong place
Testing completed (on m-c, etc.): 2.53.18 linux
Risk to taking this patch (and alternatives if risky): low
String changes made by this patch: none

Comment 5

[Ian Neal]

The patches will need testing on Windows and macOS

Comment 6

[Ian Neal]

Attachment: 1835524-sym-upload-fix-25318.patch

[Approval Request Comment]
Regression caused by (bug #): 1420363
User impact if declined: symbol upload code cannot be built
Testing completed (on m-c, etc.): 2.53.18preb1 on linux
Risk to taking this patch (and alternatives if risky): medium
String changes made by this patch: none

I've expanded SendRequest to take both JSON and key/value pairs along with related functions.
I've tested on linux but the changes need to be tested on Windows and macos.

Comment 7

[Ian Neal]

Attachment: 1835524-bugsplat-mr-v1_2-25318.patch

Updates since last version:

• Makes use of the updated functions as BugSplat needs parameters and doesn't process the JSON
• BugSplat returns response as JSON so parse it as such
• Fixes submitting crash reports through CrashSubmit.jsm - may have to look at removing duplicated code in a future bug

Comment 8

[Ian Neal]

Attachment: 1835524-sym-upload-fix-v1_1-25318.patch

Fix missing variable name change

Comment 9

[Ian Neal]

Attachment: 1835524-sym-upload-fix-v1_2-25318.patch

Fix missing argument from GenerateRequestBody

Comment 10

[Ian Neal]

Attachment: 1835524-bugsplat-mr-v1_3-25318.patch

Fix key for wstring map to be literal

Comment 11

[Ian Neal]

Attachment: 1835524-bugsplat-mr-v1_5-25318.patch

v1.4 - Fix issue with assigning to wstring
v1.5 - Fix issue with clearing submitted reports

Comment 12

[Ian Neal]

Attachment: 1835524-bugsplat-mr-v1_6-25318.patch

For 1.6 - Include channel in ver parameter to make it easier to find crashes in BugSplat UI

Comment 13

[Frank-Rainer Grahl (:frg)]

Comment on attachment 9352408 [details] [diff] [review]
1835524-bugsplat-cr-25318.patch

LGTM

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/d83ca98a3267
Use BugSplat for SeaMonkey crash reporting - comm part. r=frg

Comment 15

[Frank-Rainer Grahl (:frg)]

Comment on attachment 9357706 [details] [diff] [review]
1835524-bugsplat-mr-v1_6-25318.patch

LGTM r/a+ fro our branch.

toolkit/crashreporter/CrashReports.jsm:

+this.CrashReports.pendingDir = CrashReports_getDir("pending");
+this.CrashReports.reportsDir = CrashReports_getDir();
+this.CrashReports.submittedDir = CrashReports_getDir("submitted");

We could further optimize this by just using the reports dir for the other variables and adding the subdirectory to them but not hot code so just go with it. The two additional Services.dirsvc.get will not hurt anyone I think.

Comment 16

[Frank-Rainer Grahl (:frg)]

Comment on attachment 9357176 [details] [diff] [review]
1835524-sym-upload-fix-v1_2-25318.patch

LGTM. r/a+

Is the title for the patch not misleading? It deals with sending the crash reports and also fixes the upload?

Comment 17

[Frank-Rainer Grahl (:frg)]

https://gitlab.com/seamonkey-project/seamonkey-2.53-comm/-/commit/69528f971ef65bcb57711f1f99eeced4ef3d5bcb
Use BugSplat for SeaMonkey crash reporting - comm part. r=frg a=frg

https://gitlab.com/seamonkey-project/seamonkey-2.53-mozilla/-/commit/38361dcbba3ef4206973c93d5f1e26c75b927044
Use BugSplat for SeaMonkey crash reporting - mozilla part. r=frg a=frg
https://gitlab.com/seamonkey-project/seamonkey-2.53-mozilla/-/commit/c3a5bca7f318caa8eeaff8a28da5d5d26d1b7ac5
Use BugSplat for SeaMonkey crash reporting - fix symupload. r=frg a=frg