1835881 - Add "mac_boot_args" to the "details" page of crash reports, and make it searchable

Status: RESOLVED FIXED

(Socorro :: General, enhancement, P2)

Description

[Steven Michaud [:smichaud] (Retired)]

As of bug 1833680, Breakpad now supports a MozMacosBootargsStream -- it adds one to every crash minidump created on macOS. And a patch has just been landed on rust-minidump's main branch that can read these streams.

So Socorro should include this information in in its crash reports, and make it searchable.

Each instance of this stream has a single value (a string), which can be empty. As with "mac_crash_info", I'd like to include this information in crash reports and match "exists" searches only if it's non-empty. Otherwise the details page would become too noisy and the "exists" search would become meaningless.

I'd have liked to make Breakpad create a MozMacosBootargsStream only if its value would be non-empty, but Breakpad doesn't support this.

Comment 1

[Steven Michaud [:smichaud] (Retired)]

My rust-minidump patch is currently only on the main/master branch. I don't know when it will get included in a release. So I don't know when Socorro can start using it.

Comment 2

[Gabriele Svelto [:gsvelto]]

When we ship the next version of rust-minidump, this should happen within a few weeks. Give it some extra time for the release to be deployed to Socorro and then we can work on this.

Comment 3

[Steven Michaud [:smichaud] (Retired)]

A new version of rust-minidump was released last week that contains support for the MozMacosBootargsStream minidump stream. Now we're waiting for it to be deployed to Socorro. Any ideas about when that will happen? :-)

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Maybe this week.

Comment 5

[Steven Michaud [:smichaud] (Retired)]

Socorro's stackwalker has now been updated to use rust-minidump v0.18.0 and deployed. Can we get started here, Will? Please let me know if you have any questions.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Steven: Are there any crash reports that I can use as an example?

Comment 7

[Steven Michaud [:smichaud] (Retired)]

My patch for bug 1833680 landed on the 115 branch. So any crash report on that branch and up should do for testing. But just now I generated two new crash reports on FF 118.0.1 using about:crashparent and about:crashcontent, on a machine with known kernel boot args (== keepsyms=1 kernel_stack_pages=6):

bp-b96556e8-3940-4933-b697-33dfe0231009
bp-d8e0d008-5772-4cc6-af5a-fd8c00231009

And two more with known-absent kernel boot args (for which, as noted above, a value for "mac_boot_args" exists but is an empty string):

bp-fda05677-0e5f-4ca8-8dc9-4376d0231009
bp-9da29222-7f12-414f-943f-8746a0231009

Is this enough?

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

That's good enough. Thank you!

I'll try to get to this in the next few weeks.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Because we want to know if mac_boot_args exists and is non-empty or not, I'll create a processor rule that sets mac_boot_args iff there's a non-empty value. Then I'll add it to the search index as a string. That should support the following questions:

1. show me crash reports where mac_boot_args exists
2. show me crash reports where mac_boot_args has keep_stack_pages=6
3. show me a facet of mac_boot_args values

Comment 10

[Steven Michaud [:smichaud] (Retired)]

Sounds good to me.

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

I looked at 32,649 crash reports for Firefox for macOS in December 2023 and January 2024. Of those, 2,661 (8%) have a non-empty mac_boot_args value. Of those, 132 are unique across the set of mac_boot_args values.

Based on what I'm seeing, I think the following are true:

1. some mac_boot_args values interesting enough to uniquely identify a system in our data set
2. some mac_boot_args values set the mac address and have unique identifiers in them

Given that, the mac_boot_args field needs to be a protected field. I can create a second field that is a flag which can be public and would work with exists searches.

Comment 12

[Steven Michaud [:smichaud] (Retired)]

Can you just protect certain variables in mac_boot_args? (In the case of kernel_stack_pages=6, the "variable" would be kernel_stack_pages.) Please post a list of variables where you've found sensitive information. I'll research them as best I can, possibly finding more variables that should be considered sensitive.

I'm the one who really wants the mac_boot_args information, and am probably the one best qualified to interpret it (for the purposes of reverse engineering and bug fixing). But I'm not a Mozilla employee. So I won't have access to any of it if you make the entire field protected.

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

It looks like mac_boot_args is a user-generated value and there are lots of variations. It consists of flags and key=value pairs and other things. We can't sanitize it to make it public. I understand that is not what you were hoping for, but it is what it is.

Comment 14

[Gabriele Svelto [:gsvelto]]

A compromise could be to extract significant key/value pairs that we know would be relevant for crash reporting, or sanitize the contents of the annotation on Firefox' side to purge machine-specific data. We already do that for some Windows annotations.

Comment 15

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: [mozilla-services/socorro] bug-1835881: add support for mac_boot_args (#6516)

Comment 16

[Steven Michaud [:smichaud] (Retired)]

(In reply to Gabriele Svelto [:gsvelto] from comment #14)

A compromise could be to extract significant key/value pairs that we know would be relevant for crash reporting, or sanitize the contents of the annotation on Firefox' side to purge machine-specific data. We already do that for some Windows annotations.

I'd vote for sanitizing the mac_boot_args information on the Firefox side to purge machine-specific data.

The two bugs that depend on this one (bug 1801419 and bug 1862279) both involve OCLP, and I know (or can find out) what boot args are set by its installers. But I'm more interested in boot args that aren't set by its installers, which might trigger one or both of those bugs. So I'm not going to be able to come up with a list of key/value pairs that are significant for crash reporting.

Comment 17

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR [mozilla-services/socorro]: bug-1835881: add support for mac_boot_args (#6516) in e37a55e.

Comment 18

[Will Kahn-Greene [:willkg] ET needinfo? me]

This went out just now in bug #1877518. I'll wait until Monday, February 5th to verify that this is working.

Comment 19

[Will Kahn-Greene [:willkg] ET needinfo? me]

I verified the data is being indexed and is now searchable.

I get that this doesn't satisfy some of the underlying needs, but we'll need to implement something different for those and we should do that in different bugs.

Marking as FIXED.