Closed Bug 1309214 Opened 8 years ago Closed 4 years ago

refactor CrashAggregateViewTest to reduce the memory usage

Categories

(Data Platform and Tools :: General, defect, P2)

Product:
Data Platform and Tools
Component:
General
Type:
defect
Points:
3

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: mdoglio, Unassigned)

References

Details

Running the CrashAggregates test suite requires 4GB of RAM dedicated to the JVM. It's not clear why this is happening, but it seems to have something to do with the increasing number of dimensions in the test fixtures.
Blocks: 1255755
Note to myself: also add a testcase for pings with missing dimensions.
Component: Metrics: Pipeline → Datasets: Crash Aggregates
Product: Cloud Services → Data Platform and Tools
My guess is it's the fact that we're generating a unique ping for every possible combination of these dimensions:

https://github.com/mozilla/telemetry-batch-view/blob/4f1563d/src/test/scala/com/mozilla/telemetry/CrashAggregateViewTest.scala#L15
https://github.com/mozilla/telemetry-batch-view/blob/4f1563d/src/test/scala/com/mozilla/telemetry/CrashAggregateViewTest.scala#L154

That adds up to 98304 pings. Depending on the exact size of each ping, we could easily get into hundreds of megabytes or even gigabytes (2k per ping = 200M, 20k per ping = 2G)

Do we really need every combination here for the test to be valid?
Flags: needinfo?(mdoglio)
I don't think we do. I would put this bug on hold for now, as bug 1357793 may make this one a WONTFIX.
Flags: needinfo?(mdoglio)
Moved, per https://bugzilla.mozilla.org/show_bug.cgi?id=1453996
Component: Datasets: Crash Aggregates → Datasets: General
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WONTFIX
Component: Datasets: General → General
You need to log in before you can comment on or make changes to this bug.