1173797 - [Raptor] Reboot test coalesces marks with same name across different apps

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[:Eli Perelman]

When outputting the table of metrics at the end of a test, Raptor is coalescing metrics that share a common name, but that are generated across different apps. For example:

Metric                        Mean       Median     Min        Max        StdDev   p95
----------------------------  ---------  ---------  ---------  ---------  -------  ---------
reboot.loadEnd                25549.500  25443.000  25171.000  26360.000  309.029  26360.000
reboot.ftuSkip                29935.200  29826.000  29630.000  30660.000  296.785  30660.000
reboot.homescreenStart        30339.200  30233.000  30045.000  31103.000  302.535  31103.000
reboot.navigationLoaded       30744.600  30672.000  30378.000  31404.000  320.120  31404.000
reboot.navigationInteractive  30749.800  30677.500  30382.000  31409.000  320.195  31409.000
reboot.visuallyLoaded         31693.600  31629.000  31272.000  32518.000  346.346  32518.000
reboot.contentInteractive     34327.900  34288.000  33881.000  35077.000  307.750  35077.000
reboot.fullyLoaded            35072.400  35029.000  34521.000  35979.000  381.543  35979.000
reboot.osLogoEnd              35257.200  35263.000  34671.000  36060.000  368.366  36060.000

This table has values for fullyLoaded and visuallyLoaded, two metrics that are emitted by both the Homescreen and System app. There should be separate entries for these metrics, whether they are tagged via their context, or separate tables for each context.

Comment 1

[:Eli Perelman]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/30596

Sample of log entries: https://gist.github.com/eliperelman/22d7eee75a3deec2da98

Comment 2

[Robert Wood [:rwood]]

Hmmm... patch works great with a single run of the restart test, however I am seeing negative numbers for the verticalhome fullyLoaded minimum:

restartb2g.fullyLoaded 14697.340 17636.393 -53.043 19099.040 6840.166 18967.595

{"restartb2g.fullyLoaded":[{"time":1434391275720,"suite":"restartb2g","entryType":"mark","context":"verticalhome.gaiamobile.org","epoch":1427192873057,"value":-53.04296875,"branch":"master","device":"flame-kk","memory":"319"},{"time":1434391275720,"suite":"restartb2g","entryType":"mark","context":"verticalhome.gaiamobile.org","epoch":1427192873057,"value":-53.04296875,"branch":"master","device":"flame-kk","memory":"319"},

Full logs here: https://gist.github.com/rwood-moz/dc3bd6e8ebf3e3c9a6fd

Comment 3

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #2)
> Hmmm... patch works great with a single run of the restart test, however I
> am seeing negative numbers for the verticalhome fullyLoaded minimum:
> 
> restartb2g.fullyLoaded 14697.340 17636.393 -53.043 19099.040 6840.166
> 18967.595
> 
> {"restartb2g.fullyLoaded":[{"time":1434391275720,"suite":"restartb2g",
> "entryType":"mark","context":"verticalhome.gaiamobile.org","epoch":
> 1427192873057,"value":-53.04296875,"branch":"master","device":"flame-kk",
> "memory":"319"},{"time":1434391275720,"suite":"restartb2g","entryType":
> "mark","context":"verticalhome.gaiamobile.org","epoch":1427192873057,"value":
> -53.04296875,"branch":"master","device":"flame-kk","memory":"319"},
> 
> Full logs here: https://gist.github.com/rwood-moz/dc3bd6e8ebf3e3c9a6fd

To clarify, the negative value appeared when I ran multiple runs of the restart test (RUNS=20)

Comment 4

[:Eli Perelman]

Yes, I have the same problem, but it's unrelated to this bug. I will try and fix that in a different bug.

Comment 5

[Robert Wood [:rwood]]

Comment on attachment 8622553 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/30596

LGTM and R+ since the negative entry issue is unrelated

Comment 6

[:Eli Perelman]

In master: https://github.com/mozilla-b2g/gaia/commit/84f8a743efba5301fd0b4ab624ce55ec1d51ec2a