1174864 - [Raptor] Reboot test producing negative values for some metrics

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[:Eli Perelman]

Raptor sometimes displays negative values intermittently for metrics produced during the device reboot test. Most likely this is caused by the inability of the Flame to persist its time across reboots.

We've tried to work around this problem in the past using the DEVICETIME environment variable in Raptor, which captures the $EPOCHREALTIME prior to rebooting the device. If this value is off, or if the device hasn't corrected its time by the time any of the performance markers are hit, they will generate metrics that are too small, hence causing negative values.

Until bug 1110010 is resolved, I'm not sure we can work around this in a meaningful way, so the point of this bug may just be to throw out metrics that produce negative values.

Comment 1

[:Eli Perelman]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/30631

Comment 2

[Robert Wood [:rwood]]

Ran the restart-b2g test several times, no negative values but still some very small invalid ones (see fullyLoaded Min below). Do you want to set a higher threshold above zero for this workaround? Or is this acceptable for now?

Metric                            Mean       Median     Min        Max        StdDev    p95      
--------------------------------  ---------  ---------  ---------  ---------  --------  ---------
restartb2g.navigationLoaded       13885.173  13771.350  13317.802  15978.476  483.727   14518.183
restartb2g.navigationInteractive  13886.439  13773.350  13318.802  15979.476  483.702   14519.183
restartb2g.contentInteractive     17464.939  17324.674  16845.327  19695.476  530.502   18291.183
restartb2g.fullyLoaded            16425.900  17875.655  18.501     18386.788  4950.641  18359.706

Comment 3

[:Eli Perelman]

I think setting the threshold higher will cause some unintended consequences in the future. For example, when using performance.measure(), it would be possible to produce very small results, and these would get tossed. I'd like to say this is acceptable for now; we should never be visualizing negative values for our purposes.

Comment 4

[Robert Wood [:rwood]]

Comment on attachment 8623793 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/30631

Ok, fine with me, as it's just a temporary work-around anyway.

Comment 5

[:Eli Perelman]

In master: https://github.com/mozilla-b2g/gaia/commit/be75625dda4a6d8e5191174267ea88cab7356d16