[Raptor] Reboot test coalesces marks with same name across different apps

RESOLVED FIXED in 2.2 S14 (12june)

Status

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: Eli, Assigned: Eli)

Tracking

unspecified
2.2 S14 (12june)
ARM
Gonk (Firefox OS)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [systemsfe])

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
When outputting the table of metrics at the end of a test, Raptor is coalescing metrics that share a common name, but that are generated across different apps. For example:

Metric                        Mean       Median     Min        Max        StdDev   p95
----------------------------  ---------  ---------  ---------  ---------  -------  ---------
reboot.loadEnd                25549.500  25443.000  25171.000  26360.000  309.029  26360.000
reboot.ftuSkip                29935.200  29826.000  29630.000  30660.000  296.785  30660.000
reboot.homescreenStart        30339.200  30233.000  30045.000  31103.000  302.535  31103.000
reboot.navigationLoaded       30744.600  30672.000  30378.000  31404.000  320.120  31404.000
reboot.navigationInteractive  30749.800  30677.500  30382.000  31409.000  320.195  31409.000
reboot.visuallyLoaded         31693.600  31629.000  31272.000  32518.000  346.346  32518.000
reboot.contentInteractive     34327.900  34288.000  33881.000  35077.000  307.750  35077.000
reboot.fullyLoaded            35072.400  35029.000  34521.000  35979.000  381.543  35979.000
reboot.osLogoEnd              35257.200  35263.000  34671.000  36060.000  368.366  36060.000

This table has values for fullyLoaded and visuallyLoaded, two metrics that are emitted by both the Homescreen and System app. There should be separate entries for these metrics, whether they are tagged via their context, or separate tables for each context.
(Assignee)

Comment 1

3 years ago
Created attachment 8622553 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/30596

Sample of log entries: https://gist.github.com/eliperelman/22d7eee75a3deec2da98
Attachment #8622553 - Flags: review?(rwood)

Comment 2

3 years ago
Hmmm... patch works great with a single run of the restart test, however I am seeing negative numbers for the verticalhome fullyLoaded minimum:

restartb2g.fullyLoaded 14697.340 17636.393 -53.043 19099.040 6840.166 18967.595

{"restartb2g.fullyLoaded":[{"time":1434391275720,"suite":"restartb2g","entryType":"mark","context":"verticalhome.gaiamobile.org","epoch":1427192873057,"value":-53.04296875,"branch":"master","device":"flame-kk","memory":"319"},{"time":1434391275720,"suite":"restartb2g","entryType":"mark","context":"verticalhome.gaiamobile.org","epoch":1427192873057,"value":-53.04296875,"branch":"master","device":"flame-kk","memory":"319"},

Full logs here: https://gist.github.com/rwood-moz/dc3bd6e8ebf3e3c9a6fd
Flags: needinfo?(eperelman)

Comment 3

3 years ago
(In reply to Robert Wood [:rwood] from comment #2)
> Hmmm... patch works great with a single run of the restart test, however I
> am seeing negative numbers for the verticalhome fullyLoaded minimum:
> 
> restartb2g.fullyLoaded 14697.340 17636.393 -53.043 19099.040 6840.166
> 18967.595
> 
> {"restartb2g.fullyLoaded":[{"time":1434391275720,"suite":"restartb2g",
> "entryType":"mark","context":"verticalhome.gaiamobile.org","epoch":
> 1427192873057,"value":-53.04296875,"branch":"master","device":"flame-kk",
> "memory":"319"},{"time":1434391275720,"suite":"restartb2g","entryType":
> "mark","context":"verticalhome.gaiamobile.org","epoch":1427192873057,"value":
> -53.04296875,"branch":"master","device":"flame-kk","memory":"319"},
> 
> Full logs here: https://gist.github.com/rwood-moz/dc3bd6e8ebf3e3c9a6fd

To clarify, the negative value appeared when I ran multiple runs of the restart test (RUNS=20)
(Assignee)

Comment 4

3 years ago
Yes, I have the same problem, but it's unrelated to this bug. I will try and fix that in a different bug.
Flags: needinfo?(eperelman)

Comment 5

3 years ago
Comment on attachment 8622553 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/30596

LGTM and R+ since the negative entry issue is unrelated
Attachment #8622553 - Flags: review?(rwood) → review+
(Assignee)

Comment 6

3 years ago
In master: https://github.com/mozilla-b2g/gaia/commit/84f8a743efba5301fd0b4ab624ce55ec1d51ec2a
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.