1045076 - Launch events occasionally incorrectly output timestamp as delta

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect, P1)

Description

[:Eli Perelman]

The test durations for some of the launch events are occasionally outputting values that are equivalent to a timestamp instead of using that as a delta between dates. This is visualized in Datazilla [1] as extreme chart spiking, but for a seemingly unrelated issue as was resolved in bug 1037520.

[1] https://datazilla.mozilla.org/b2g/?branch=master&device=flame&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,settings,sms,video&app=costcontrol&gaia_rev=a5f93c8ba175ae83&gecko_rev=4dc57781185d&plot=avg

Comment 1

[:Eli Perelman]

Jonathan, do you know why the values for costcontrol at this spike point [1] are not visible in the build revision at [2]? All the values in the runs chart appear normal as do the values output from make test-perf, but for some reason there are generated values for average which appear in the large chart, and I do not see where they come from.

[1] http://mzl.la/1AuqA6K
[2] http://jenkins1.qa.scl3.mozilla.com/job/flame.b2g-inbound.perf.gaia/848/artifact/perf.json

Comment 2

[Jonathan Eads ( :jeads )]

The spike values are in the replicates for costcontrol in the testrun.date = 1406511590 in the two json structures returned by the web service call below:

https://datazilla.mozilla.org/b2g/refdata/objectstore/json_blob/revisions?branch=master&gaia_revision=a5f93c8ba175ae83fac6b7740a8414459fbd7a5b&gecko_revision=4dc57781185d&test_id=22&test_type=startup_%3E_moz-app-visually-complete

There's an additional test run available with an identical revision set that does not contain the spikes. The revisions in the two structures are identical.

gaia_revision a5f93c8ba175ae83fac6b7740a8414459fbd7a5b
gecko_revision 4dc57781185d
build_revision 3aa6abd313f965a84aa86c6b213dc154e4875139

The second structure submitted has a testrun.date = 1406511841, the replicates displayed are for the last testrun but the average displayed on the main chart is across both of the testruns because all of the revisions and test/app reference data are identical. So, the "GROUP BY" in SQL is matching the revisions when the mean is calculated https://github.com/mozilla/datazilla/blob/master/datazilla/model/sql/perftest.json#L643.

Not sure why the replicate chart is not showing both sets of replicates, that would be much less misleading.

Comment 3

[:Eli Perelman]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22272

Added sanity check for instances where the timestamp may not have registered correctly or we fail to retrieve values.

Comment 4

[:Eli Perelman]

Comment on attachment 8464131 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22272

With the patches from bug 1037520 now in place, we need this safeguard to go along with it to prevent incorrect timestamp values from being injected into the data that is visualized in Datazilla.

[Bug caused by] (feature/regressing bug #): bug 1037520
[User impact] if declined: none
[Testing completed]: yes
[Risk to taking this patch] (and alternatives if risky): no user-facing risk
[String changes made]: n/a

Comment 5

[Hubert Figuiere [:hub]]

will merge in a bit. Just want to make sure TBPL turns green with the intermitents.

Comment 6

[Hubert Figuiere [:hub]]

Shall we leave this open so that the actual problem gets fixed?

Comment 7

[Hubert Figuiere [:hub]]

Merged
https://github.com/mozilla-b2g/gaia/commit/29a58ec654e598926409c382c9f3b5db3726947a

Comment 8

[:Eli Perelman]

Hey Hub, I've had the worst luck trying to replicate this issue locally. I've used a Flame with 512MB, 273MB, and 319MB, doing runs of 1, 5, 10, and 30 and could not get this issue to replicate. The safeguard I have in this patch reflects that I don't know what the root cause was, but can at least prevent against it. If we want to dive into this issue deeper, I can do it when I get back from PTO.

Comment 9

[:Eli Perelman]

Filed 1049129 to dive into root cause of this issue. Safeguard patch in master gives current resolution.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

v2.0: https://github.com/mozilla-b2g/gaia/commit/cc423dbb92043ca5e449ef04a83cefa6b62a99fe