Calendar's cold launch time is over the 1000 ms acceptance threshold for 2.1

RESOLVED FIXED in 2.2 S4 (23jan)

Status

defect
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: gmealer, Assigned: mmedeiros, NeedInfo)

Tracking

({perf, regression})

unspecified
2.2 S4 (23jan)
x86
All
Dependency tree / graph

Firefox Tracking Flags

(blocking-b2g:2.2+, b2g-v2.2 fixed, b2g-master fixed)

Details

(Whiteboard: [priority])

Attachments

(2 attachments)

See test results at:

https://wiki.mozilla.org/B2G/QA/2014-10-02_Performance_Acceptance#Calendar

Median startup time was 1267 ms for 450 launches. This is also a regression from 2.0, which was median 1142 ms for 480 launches. 

Graphs, raw data and other testing details are on the wiki.
[Blocking Requested - why for this release]: performance is over acceptance threshold
blocking-b2g: --- → 2.1?
Depends on: 1059349
Duping to existing bug 1059349 which has some detailed diagnostic info.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1059349
blocking-b2g: 2.1? → ---
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Whiteboard: [priority]
[priority] --> tracking-b2g:+ conversion
tracking-b2g: --- → +
Calendar's median launch for 2.1 is measured at 1182 ms as of 10-31 per https://wiki.mozilla.org/B2G/QA/2014-10-31_Performance_Acceptance#Calendar
Per https://wiki.mozilla.org/B2G/QA/2014-11-14_Performance_Acceptance#Calendar we're above the 1150 ms revised acceptance target, at 1228 ms.

CC is upcoming, so we're basically at acceptance time now. Flagging Dylan to recommend action, and cc'ing Tony.
Flags: needinfo?(doliver)
There have been few if any changes in Calendar in the past month that would be responsible for this additional regression. It would be helpful to get a regression window on this one and that would likely point at a system/platform issue that could be further investigated.

For the Calendar code specifically, I don't believe it is a good use of resources to optimize the 2.1 startup time. We have done a large amount of refactoring in master since the branch was created (most significantly bug 1027707) and any effort spent improving 2.1 will be unlikely to apply to master. 

We do intend to prioritize performance improvements for 2.2 with a target of getting the number back to 1000ms. The refactoring effort has given us options to explore including optimizing module loading and caching startup data.
Flags: needinfo?(doliver)
(In reply to Dylan Oliver [:doliver] from comment #6)
> There have been few if any changes in Calendar in the past month that would
> be responsible for this additional regression. It would be helpful to get a
> regression window on this one 

I don't know that it is a regression. When we're talking a 40 ms difference, that could be two sides of a margin of error. In any case, I'm not optimistic about our ability to efficiently window a difference that small; if it's not fairly obvious on Datazilla, it won't be obvious on a windowing exercise either.

Updated

5 years ago
feature-b2g: --- → 2.2+

Comment 8

4 years ago
[Tracking Requested - why for this release]:
feature-b2g: 2.2+ → ---
blocking-b2g: --- → 2.2?
tracking-b2g: ? → ---
triage: blocking, we'll address this for the 2.2 release.
blocking-b2g: 2.2? → 2.2+
Keywords: regression
(Assignee)

Updated

4 years ago
Assignee: nobody → mmedeiros
Status: REOPENED → ASSIGNED
Depends on: 1088381
(Assignee)

Updated

4 years ago
Depends on: 1088402
(Assignee)

Updated

4 years ago
Blocks: 1118438
improved `content-interactive` performance from >1.8s to <1.2s.
Attachment #8548974 - Flags: review?(gaye)
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago4 years ago
Resolution: --- → FIXED
Target Milestone: --- → 2.2 S4 (23jan)
Comment on attachment 8548974 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/27388

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): Bug 1027707
[User impact] if declined: calendar app will take almost 2 seconds to launch instead of ~1.1s
[Testing completed]: manual testing and startup performance testing, old unit/marionette tests still passes.
[Risk to taking this patch] (and alternatives if risky): low risk, most of the changes are just bundling the same dependencies, changing order of execution and replacing the module loader.
[String changes made]: none
Attachment #8548974 - Flags: approval-gaia-v2.2?
Are we seeing any improvements on datazilla on master, or how was this exactly tested to show the improvement in metrics ?
Flags: needinfo?(mmedeiros)
I executed `make test-perf` multiple times locally and also did some manual tests (with a bunch of `window.performance.mark` calls); it improved both the "real performance" and "perceived performance" drastically (from ~1.8s to ~1.1s)

datazilla is not displaying the calendar data for some reason (see Bug 1067625)
Flags: needinfo?(mmedeiros)
See Also: → 1067625
Geo, can you re-run your startup tests for Calendar on master to get new data here?
Flags: needinfo?(gmealer)
Yes, will do. Keeping NI open for the todo.
Attachment #8548974 - Flags: approval-gaia-v2.2? → approval-gaia-v2.2+
Gareth, could we use similar approach to improve launch time on v2.1? I found this patch is totally different and can't simply uplift, do you have any suggestion on that? Thanks!
Flags: needinfo?(gaye)
Sherman, the code structure on 2.1 was very different from 2.2, some of the optimizations was only possible because I ended up rewriting a big part of the month view for 2.2 and abstracting some DB operations. 2.1 still used a global `Calendar` namespace. It would definitely take a good amount of work to make similar changes to 2.1;

you can try some quick changes like the new `nextTick` implementation: https://github.com/mozilla-b2g/gaia/blob/master/apps/calendar/js/next_tick.js and bundle all the JS/CSS into as few files as possible during the build step. The initialization order also made a big impact on the perceived performance (compare the old `app.init` and `app._init` with new implementation).

the summary of changes on the pull request gives some insight on where I got most perf improvements.
Flags: needinfo?(gaye)
Per Dylan's request, ran the test on a current 3.0 (was a nightly from yesterday). Got a bimodal result, with peaks around 1350 and 1450. 

This was without a workload, as well, due to bug 1129069 (can't launch calendar w/ reference workload)

I'm going to retry this on 3.0 to make sure this wasn't a fluke, and also 2.2 for comparison's sake. Keeping my NI alive for that.
You need to log in before you can comment on or make changes to this bug.