refactor CrashAggregateViewTest to reduce the memory usage

NEW
Unassigned

Status

P2
normal
2 years ago
6 months ago

People

(Reporter: mdoglio, Unassigned)

Tracking

(Blocks: 1 bug)

Details

(Reporter)

Description

2 years ago
Running the CrashAggregates test suite requires 4GB of RAM dedicated to the JVM. It's not clear why this is happening, but it seems to have something to do with the increasing number of dimensions in the test fixtures.
(Reporter)

Updated

2 years ago
Blocks: 1255755
(Reporter)

Comment 1

2 years ago
Note to myself: also add a testcase for pings with missing dimensions.
Component: Metrics: Pipeline → Datasets: Crash Aggregates
Product: Cloud Services → Data Platform and Tools
My guess is it's the fact that we're generating a unique ping for every possible combination of these dimensions:

https://github.com/mozilla/telemetry-batch-view/blob/4f1563d/src/test/scala/com/mozilla/telemetry/CrashAggregateViewTest.scala#L15
https://github.com/mozilla/telemetry-batch-view/blob/4f1563d/src/test/scala/com/mozilla/telemetry/CrashAggregateViewTest.scala#L154

That adds up to 98304 pings. Depending on the exact size of each ping, we could easily get into hundreds of megabytes or even gigabytes (2k per ping = 200M, 20k per ping = 2G)

Do we really need every combination here for the test to be valid?
Flags: needinfo?(mdoglio)
(Reporter)

Comment 3

2 years ago
I don't think we do. I would put this bug on hold for now, as bug 1357793 may make this one a WONTFIX.
Flags: needinfo?(mdoglio)
Moved, per https://bugzilla.mozilla.org/show_bug.cgi?id=1453996
Component: Datasets: Crash Aggregates → Datasets: General
You need to log in before you can comment on or make changes to this bug.