1079557 - Calendar's cold launch time is over the 1000 ms acceptance threshold for 2.1

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Calendar, defect)

Description

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

See test results at:

https://wiki.mozilla.org/B2G/QA/2014-10-02_Performance_Acceptance#Calendar

Median startup time was 1267 ms for 450 launches. This is also a regression from 2.0, which was median 1142 ms for 480 launches. 

Graphs, raw data and other testing details are on the wiki.

Comment 1

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

[Blocking Requested - why for this release]: performance is over acceptance threshold

Comment 2

[Dylan Oliver [:doliver]]

Duping to existing bug 1059349 which has some detailed diagnostic info.

Comment 3

[Dylan Oliver [:doliver]]

[priority] --> tracking-b2g:+ conversion

Comment 4

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Calendar's median launch for 2.1 is measured at 1182 ms as of 10-31 per https://wiki.mozilla.org/B2G/QA/2014-10-31_Performance_Acceptance#Calendar

Comment 5

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Per https://wiki.mozilla.org/B2G/QA/2014-11-14_Performance_Acceptance#Calendar we're above the 1150 ms revised acceptance target, at 1228 ms.

CC is upcoming, so we're basically at acceptance time now. Flagging Dylan to recommend action, and cc'ing Tony.

Comment 6

[Dylan Oliver [:doliver]]

There have been few if any changes in Calendar in the past month that would be responsible for this additional regression. It would be helpful to get a regression window on this one and that would likely point at a system/platform issue that could be further investigated.

For the Calendar code specifically, I don't believe it is a good use of resources to optimize the 2.1 startup time. We have done a large amount of refactoring in master since the branch was created (most significantly bug 1027707) and any effort spent improving 2.1 will be unlikely to apply to master. 

We do intend to prioritize performance improvements for 2.2 with a target of getting the number back to 1000ms. The refactoring effort has given us options to explore including optimizing module loading and caching startup data.

Comment 7

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

(In reply to Dylan Oliver [:doliver] from comment #6)
> There have been few if any changes in Calendar in the past month that would
> be responsible for this additional regression. It would be helpful to get a
> regression window on this one 

I don't know that it is a regression. When we're talking a 40 ms difference, that could be two sides of a margin of error. In any case, I'm not optimistic about our ability to efficiently window a difference that small; if it's not fairly obvious on Datazilla, it won't be obvious on a windowing exercise either.

Comment 8

[Jacob Shen]

[Tracking Requested - why for this release]:

Comment 9

[Dylan Oliver [:doliver]]

triage: blocking, we'll address this for the 2.2 release.

Comment 10

[Miller Medeiros [:millermedeiros]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/27388

improved `content-interactive` performance from >1.8s to <1.2s.

Comment 11

[Gareth Aye [:gaye] (back from PTO)]

Comment on attachment 8548974 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/27388

Awesome work Miller! Landed on master https://github.com/mozilla-b2g/gaia/commit/f74795eade46f4613741dd9e16fc43a905b40d2b

Comment 12

[Miller Medeiros [:millermedeiros]]

Comment on attachment 8548974 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/27388

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): Bug 1027707
[User impact] if declined: calendar app will take almost 2 seconds to launch instead of ~1.1s
[Testing completed]: manual testing and startup performance testing, old unit/marionette tests still passes.
[Risk to taking this patch] (and alternatives if risky): low risk, most of the changes are just bundling the same dependencies, changing order of execution and replacing the module loader.
[String changes made]: none

Comment 13

[bhavana bajaj [:bajaj]]

Are we seeing any improvements on datazilla on master, or how was this exactly tested to show the improvement in metrics ?

Comment 14

[Miller Medeiros [:millermedeiros]]

I executed `make test-perf` multiple times locally and also did some manual tests (with a bunch of `window.performance.mark` calls); it improved both the "real performance" and "perceived performance" drastically (from ~1.8s to ~1.1s)

datazilla is not displaying the calendar data for some reason (see Bug 1067625)

Comment 15

[Dylan Oliver [:doliver]]

Geo, can you re-run your startup tests for Calendar on master to get new data here?

Comment 16

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Yes, will do. Keeping NI open for the todo.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

v2.2: https://github.com/mozilla-b2g/gaia/commit/eb7881e5c0f5c31d4c892b254c0fae92a4e37a71

Comment 18

[Sherman Chen [:chens] (inactive)]

Gareth, could we use similar approach to improve launch time on v2.1? I found this patch is totally different and can't simply uplift, do you have any suggestion on that? Thanks!

Comment 19

[Miller Medeiros [:millermedeiros]]

Sherman, the code structure on 2.1 was very different from 2.2, some of the optimizations was only possible because I ended up rewriting a big part of the month view for 2.2 and abstracting some DB operations. 2.1 still used a global `Calendar` namespace. It would definitely take a good amount of work to make similar changes to 2.1;

you can try some quick changes like the new `nextTick` implementation: https://github.com/mozilla-b2g/gaia/blob/master/apps/calendar/js/next_tick.js and bundle all the JS/CSS into as few files as possible during the build step. The initialization order also made a big impact on the perceived performance (compare the old `app.init` and `app._init` with new implementation).

the summary of changes on the pull request gives some insight on where I got most perf improvements.

Comment 20

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Attachment: 3.0_perf_calendar.png

Per Dylan's request, ran the test on a current 3.0 (was a nightly from yesterday). Got a bimodal result, with peaks around 1350 and 1450. 

This was without a workload, as well, due to bug 1129069 (can't launch calendar w/ reference workload)

I'm going to retry this on 3.0 to make sure this wasn't a fluke, and also 2.2 for comparison's sake. Keeping my NI alive for that.