1080812 - Clock app performance tests are problematic after 5 runs, causing timing spikes

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[:Eli Perelman]

Attachment: Datazilla screenshot

With bug 1079700 I made a change to the performance tests that solidify the metrics captured, but for some reason the Clock app starts reporting extreme durations after 5 runs. This is reflected in the latest overview on Datazilla [1] (see attachment).

[1] https://datazilla.mozilla.org/b2g/

Comment 1

[:Eli Perelman]

From the performance output on Jenkins:

{
      "title": "startup > moz-chrome-interactive",
      "fullTitle": "startup event test > clock > startup > moz-chrome-interactive",
      "duration": 514795,
      "mozPerfDurations": [
        2435,
        909,
        864,
        900,
        849
      ],
      "mozPerfDurationsAverage": 1191.4
    },
    {
      "title": "startup > moz-app-visually-complete",
      "fullTitle": "startup event test > clock > startup > moz-app-visually-complete",
      "duration": 514795,
      "mozPerfDurations": [
        2829,
        1102,
        964,
        1050,
        1351,
        8632,
        8613,
        8483,
        8716,
        8737,
        8535,
        8629,
        8528,
        8546,
        8688,
        8681,
        8573,
        8499,
        8627,
        8761,
        8445,
        8693,
        8651,
        8460,
        8567,
        8719,
        8672,
        8700,
        8711,
        8818
      ]

"moz-chrome-interactive" only ran 5 times, then every other event emitted spikes after that. Investigating.

Comment 2

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Yikes. I'll hold off starting any new runs with that patch in place until you can figure this out.

Comment 3

[:Eli Perelman]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25014

The recent change I made to performance tests uncovered a reliance on having the screen staying on, e.g. no screen timeout. This patch sets the screen to stay on during the performance tests.

Comment 4

[:Eli Perelman]

Attachment: Link to v2.0 Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25033

v2.0 version of the same PR.

Comment 5

[Anthony Ricaud (:rik)]

Comment on attachment 8502961 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25014

I'm not sure which parts you want me to review.

Instead of creating disableScreenTimeout, can't you add 'screen.timeout': 0 directly to the settings like you do for 'lockscreen.enabled':  false?

Comment 6

[:Eli Perelman]

Hey Rik, I was looking for a review of the test for the Dialer app. As far as the screen timeout setting, I had originally tried that, but it didn't seem to work, so I had to go the method route.

Comment 7

[Anthony Ricaud (:rik)]

Comment on attachment 8502961 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25014

Good for me then.

Comment 8

[Arthur Chen [:arthurcc][inactive after 6/30]]

Comment on attachment 8503209 [details] [review]
Link to v2.0 Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25033

r=me for the settings part.

Comment 9

[:Eli Perelman]

Try run for master PR: https://treeherder.mozilla.org/ui/#/jobs?repo=gaia-try&revision=6c0dd4983ae8

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Master: https://github.com/mozilla-b2g/gaia/commit/331934472da3073862ca71fcf79c58fd733d06cd

Comment 11

[:Eli Perelman]

Comment on attachment 8502961 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25014

Requesting uplift of TEST-ONLY changes to v2.1 to fix breakage in performance tests.

[Bug caused by] (feature/regressing bug #): bug 1079700
[User impact] if declined: n/a
[Testing completed]: these are changes to the tests
[Risk to taking this patch] (and alternatives if risky): no user-facing risk
[String changes made]: n/a

Comment 12

[:Eli Perelman]

Comment on attachment 8503209 [details] [review]
Link to v2.0 Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25033

Requesting uplift of TEST-ONLY changes to v2.0 to fix breakage in performance tests.

[Bug caused by] (feature/regressing bug #): bug 1079700
[User impact] if declined: n/a
[Testing completed]: these are changes to the tests
[Risk to taking this patch] (and alternatives if risky): no user-facing risk
[String changes made]: n/a

Comment 13

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Adding on, there's no user impact if declined, but sounds like we need these for valid performance acceptance testing on 2.1 and 2.0 so there's internal impact. With that in mind, QA also requests uplift be granted.

Comment 14

[Tony Chung [:tchung]]

bhavana, can we get this approved for 2.0 and 2.1?   this is blocking us from rerunning perf tests on 2.1 without this patch. 

I believe this is only for testing, and does not affect gecko or gaia.

Comment 15

[bhavana bajaj [:bajaj]]

Comment on attachment 8503209 [details] [review]
Link to v2.0 Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25033

Making an exception on approving a non-blocking bug here as there is no user facing impact/risk. This will help us measure our perf targets better.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

v2.1: https://github.com/mozilla-b2g/gaia/commit/1ea74943cfe525c76a074ca1d7de8e51a70f6b98
v2.0: https://github.com/mozilla-b2g/gaia/commit/9c7dec14e058efef81f2267b724dad0850fc07e4

Comment 17

[Kai-Zhen Li [:seinlin][:kli]]

v2.0m: https://github.com/mozilla-b2g/gaia/commit/9c7dec14e058efef81f2267b724dad0850fc07e4