change deepcopy in processor



2 years ago
2 years ago


(Reporter: willkg, Unassigned)


Firefox Tracking Flags

(Not tracked)


Right now, we make a deepcopy of every raw crash for every crash storage class in the processor. That does a lot of extra work because not all the crash storage classes actually mutate the raw crash.

This bug covers changing the code so that we only do deep copies for crash storage classes that mutate the crash.

Comment 2

2 years ago
Commit pushed to master at
Fixes bug 1318448: reduces deepcopies we do in the processor (#3594)

* Fixes bug 1318448: reduces deepcopies we do in the processor

This changes the PolyCrashStorage code such that it only does a deep
copy of the processed crash if the crash storage says it mutates the
crash data.

Currently, we have 5 crash storage classes configured in production.
This change will reduce the number of deepcopies we're doing from 5 to

* Fix for statsd wrapped crash stores

* Fix the fix

The previous fix didn't do the right thing. This does.


2 years ago
Last Resolved: 2 years ago
Resolution: --- → FIXED
I missed that StatsdCounter doesn't derive from CrashStorageBase.

PR to handle StatsdCounter:
This is on -stage now.

Before the changes:

* processor save_raw_and_processed maxes out at 1.95k crashes every 30m
* CPU hangs out in the high 80s%/low 90s %

After the changes:

* processor save_raw_and_processed maxes out at 2.1k crashes every 20min
* CPU hangs out in the low 50s%

Datadog graph with the appropriate range is here:

Seems like it's good.

We're in a change freeze, so this will go to -prod in a couple of weeks.
This was just pushed to -prod. I added a processor CPU graph to the datadog socorro prod dashboard and watched it for a bit. I don't see any changes in processing (which is expected and fine). Everything seems ok. I think we're done here.
You need to log in before you can comment on or make changes to this bug.