1988674 - Re-process crashes with Marionette and RemoteAgent annotations enabled

Status: RESOLVED WONTFIX

(Socorro :: Processor, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Over on bug 1943891 the Marionette and Remote Agent annotations have been made visible and searchable on Socorro. As mentioned by Bianca we could request re-processing of existing crash reports to know how many times those annotations were part of reported crashes.

To limit this re-processing to only the important crashes please limit it to those entries that match the following criteria:

• Crash Annotations: Marionette=1 and RemoteAgent=1
• MOZ_CRASH Reason (Sanitized): doesn't contain Crash via about:crash
• Product: Firefox, Fenix, Thunderbird
• Version: >=128

Here the relevant queries for Marionette and RemoteAgent

Thanks!

Comment 1

[Bianca Danforth [:bdanforth]]

Hi Henrik. Thanks for explaining what you want and providing some SuperSearch URLs. I don't think the URLs you provided align with what you want, but I'll explain to confirm.

These SuperSearch URLs are currently filtering processed crash reports for a field that wasn't yet added to them, so there are only one or two reports showing up in the results table (reports that were processed after the Elasticsearch index rollover on Monday 9/15). Based on your list above, it sounds like you're looking for those crashes that had Marionette: 1 or RemoteAgent: 1 in the raw crash report, so we would want to replace marionette is true with crash report keys contains Marionette | RemoteAgent.

Also, the date range for the URLs is only for the past week. Did you want to go back further than that? In general, we only keep raw crash reports for 6 months (26 weeks), so we can only go back to March 2025.

I'm going to propose updated SuperSearch URLs with dates ranging for the past 6 months, and to replace the marionette is true with crash report keys contains Marionette. Same goes for the Remote Agent SuperSearch URL.

Do these updated URLs look good to you? Now there are several hundred crash reports that match the combined queries:

• Marionette
• Remote Agent
• Note: The date range (and therefore report counts) will be slightly different, as I plan to search for the past 6 months from the day I reprocess these crashes.

Once you approve the URLs, we'll have what we need to reprocess (which will likely be next week).

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thank you for the update! I didn't know that I needed to use crash report keys for it. Is there a reason why marionette can be directly selected as key? It's a bit irritating.

I've taken a look at those two queries and it looks like they are correct. Wow, I didn't expect so many crash reports and wonder how more reports will be on top after reprocessing the old crash data. Yes, we don't need anything older than 6 months; and if the above query would still produce too much load, I'm happy to get it cut to just 3 months or so.

Thanks!

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hm, actually when I use the pagination of the search results page and navigate to the oldest entries I can see crashes as reported in April this year. Doesn't mean that all reports have already been reprocessed?

Comment 4

[Bianca Danforth [:bdanforth]]

(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #2)

Thank you for the update! I didn't know that I needed to use crash report keys for it. Is there a reason why marionette can be directly selected as key? It's a bit irritating.

It's because I've added it as a new field to search processed crashes as part of Bug 1943891. Once I've reprocessed all the crashes that were processed prior to the index rollover on Monday 9/15, they will show up as expected with the search filter you used initially.

I've taken a look at those two queries and it looks like they are correct. Wow, I didn't expect so many crash reports and wonder how more reports will be on top after reprocessing the old crash data. Yes, we don't need anything older than 6 months; and if the above query would still produce too much load, I'm happy to get it cut to just 3 months or so.

These two Super Search URLs represent the crash reports that are going to be reprocessed, so you should expect roughly that amount.

Honestly, since the fields are public now (thanks to Bug 1943891), and you can search on the raw crash attributes (via crash report keys), I am wondering if you need them to be reprocessed? The only difference in reprocessing them would be that you could search for the older crashes (processed prior to 9/15) using the expected filters (marionette is true) instead of crash report keys contains Marionette (and similar for RemoteAgent). Based on your patch that added the annotations, the searches would produce the same results, since you only set the annotation in the case it's true (or 1) FWICT.

Comment 5

[Sven Marnach :sven]

Reprocessing these crash reports will take only a few minutes of compute time and hardly any effort, now that we have the supersearch queries. Let's just reprocess them all for the last six months.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Bianca Danforth [:bdanforth] from comment #4)

(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #2)

Thank you for the update! I didn't know that I needed to use crash report keys for it. Is there a reason why marionette can be directly selected as key? It's a bit irritating.

It's because I've added it as a new field to search processed crashes as part of Bug 1943891. Once I've reprocessed all the crashes that were processed prior to the index rollover on Monday 9/15, they will show up as expected with the search filter you used initially.

Ah, good to know! So I can store the original queries from my first comment as bookmarks for regular checks.

(In reply to Sven Marnach :sven from comment #5)

Reprocessing these crash reports will take only a few minutes of compute time and hardly any effort, now that we have the supersearch queries. Let's just reprocess them all for the last six months.

Thanks a lot!

Comment 7

[Bianca Danforth [:bdanforth]]

I just reprocessed the crashes with the following commands (in a shell in the app container via just shell):

socorro-cmd fetch_crashids --num="all" --url="https://crash-stats.mozilla.org/search/?major_version=%3E%3D128&moz_crash_reason=%21~Crash%20via%20about%3Acrash&crash_report_keys=~Marionette&product=Firefox&product=Thunderbird&product=Fenix&date=%3E%3D2025-03-19T15%3A10%3A00.000Z&date=%3C2025-09-19T15%3A10%3A00.000Z&_facets=crash_report_keys&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports" > crash-ids1.txt
socorro-cmd fetch_crashids --num="all" --url="https://crash-stats.mozilla.org/search/?major_version=%3E%3D128&moz_crash_reason=%21~Crash%20via%20about%3Acrash&crash_report_keys=~RemoteAgent&product=Firefox&product=Thunderbird&product=Fenix&date=%3E%3D2025-03-19T15%3A11%3A00.000Z&date=%3C2025-09-19T15%3A11%3A00.000Z&_facets=crash_report_keys&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports" > crash-ids2.txt
cat crash-ids1.txt crash-ids2.txt | sort -u | socorro-cmd reprocess

While this did successfully reprocess 354 reports (based on the last command's script output, monitoring the reprocessing queue and checking Sentry and the processor logs), they are still not showing up when searching for crash reports in SuperSearch with the new marionette and remote agent fields.

After some digging, I figured out why. When a crash is reprocessed, it's reprocessed to its original index, with that index's mapping. This is because the name of the index the document is saved to is determined by the date_processed field, which is the original date the report was submitted. This field doesn't change when a crash report is reprocessed. While the latest index (socorro202537 created on 9/15) has a mapping with the two new fields, older indexes do not. This means reprocessing old crashes doesn't update the processed_crash to have the new fields. See also Bug 1686103.

In order to have these older crash reports show up with these new fields in SuperSearch, we would need some kind of migration script to update all past indexes' mappings with the new fields. Then after such a migration, reprocessing old crashes would work as expected. I've reopened Bug 1503591 to do this.

Anyway, in the meantime, for this ticket in particular, I am going to mark it as WONTFIX, and my recommendation to you is to use the crash report keys approach as outlined in this comment and/or use marionette | remote agent is true only for crashes processed after 9/15.

Aside: While reprocessing doesn't "just work" for adding new fields to the processed crash report, it does work for signature changes and when missing symbols files are added. These are slightly more common use cases.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

No worries. Sounds good. Thanks a lot for working on integrating Marionette and Remote Agent!