1059349 - [Calendar] Startup time regressions of ~100 ms from 2.0 to 2.1

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Gaia::Calendar, defect)

Description

[Tony Chung [:tchung]]

[Blocking Requested - why for this release]:

Per Datazilla, Calendar cold startup numbers have regressed about 100 ms since 2.0 to 2.1.  

Environment: startup_>_moz-app-visually-complete, flame-319MB, workload = light.  human perception goal = 1000ms

* 2.0 (All numbers use median, looking at latest numbers from 8-20 to 8-25)
** https://datazilla.mozilla.org/b2g/?branch=v2.0&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar&app=calendar&gaia_rev=d72f8ad53448aed5&gecko_rev=2a18149b3ae8&plot=avg

    calendar -- High, 1203; Low, 1131; Trend ~1160 

* 2.1  (All numbers use median, looking at numbers from 8-08 to 8-13 (last available))
* https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=30&test=startup_%3E_moz-app-visually-complete&app_list=calendar&app=calendar&gaia_rev=a2219a55145e730e&gecko_rev=97a5d21e57ac&plot=avg

    calendar -- High, 1318; Low, 1219; Trend ~1260 

Please investigate, thanks.

Comment 1

[Candice Serran (:cserran)]

blocking based on performance regression. Team to investigate

Comment 2

[Miller Medeiros [:millermedeiros]]

I really don't think the perf regression was introduced by the calendar app itself, specially since we have regressions on all the productivity apps.

by looking only at the datazilla graph it looks like perf regression was introduced mainly by https://github.com/mozilla-b2g/gaia/commit/62eedafb0657bbec (mozl10n, which affects multiple apps)

there was some changes on the perf graph before that (a few ms), [this commit](https://github.com/mozilla-b2g/gaia/commit/6cc134a8d8b19ee2) also change a little bit how we are measuring the performance and happened at the same day. - so maybe perf did not change, we are only measuring it differently...

and a few other commits that affected the calendar app landed on 2014/07/23 and 2014/07/24:  https://github.com/mozilla-b2g/gaia/commit/3078ac2eb (small CSS change, probably unrelated) and https://github.com/mozilla-b2g/gaia/commit/500ebd79331 (theme color, affects many apps at once)

whatever happened, probably landed between July 23 and 24th.

Comment 3

[Miller Medeiros [:millermedeiros]]

I found a few weird commits between https://github.com/mozilla-b2g/gaia/compare/9dd4bf0...9f4ad73

I'm almost sure that the main regression was caused by commit https://github.com/mozilla-b2g/gaia/commit/9f4ad7301db7b3a034f012f0af4eb97df06c818e (Bug 1042083) but there are also some other regressions and changes in the way performance events are measured.

Here are the average results for 10 runs on my Flame running Gecko 34.0a1 (https://hg.mozilla.org/mozilla-central/rev/6f702709fab6)

# 9f4ad73 (perf regression to startup & dom-loaded)
## Bug 1042083 - Implement App Titlebar
startup: 832.7
moz-chrome-dom-loaded: 1324.4761820000017

# b9240ad (small regression to startup & dom-loaded)
## Bug 1041985 - [Search] Enable places
startup: 713.5
moz-chrome-dom-loaded: 1170.5350103

# 6cc134a (dom-loaded stabilized here)
## Bug 1037520 - Include start time capture in the repeat phase, not test start phase
startup: 685.9
moz-chrome-dom-loaded: 1071.0687652999998

# 9dd4bf0 (dom-loaded is way lower than following commits)
## Bug 1030352 - Delete/undo-delete phone number button in edit dialog does not have an accessibility label
startup: 661.3
moz-chrome-dom-loaded: 823.6056771000007

Comment 4

[Miller Medeiros [:millermedeiros]]

Attachment: perf_results.zip

Performance results for Gaia commits 9f4ad73, b9240ad, 6cc134a & 9dd4bf0 running on:

Device    Flame
Gecko     https://hg.mozilla.org/mozilla-central/rev/6f702709fab6
BuildID   20140722040212
Version   34.0a1

  RUNS=10 MARIONETTE_RUNNER_HOST=marionette-device-host APP=calendar make test-perf

Comment 5

[Miller Medeiros [:millermedeiros]]

and it looks like the performance regression introduced by commit 9f4ad73 was detected and "fixed" by Bug 1043857 a few days later (did not test it yet). I'll keep investigating it tomorrow, but this is a very SLOW (and boring) process..

Comment 6

[Miller Medeiros [:millermedeiros]]

by analyzing the data from datazilla (https://datazilla.mozilla.org/b2g/?branch=master&device=flame-319MB&range=30&test=startup_%3E_moz-app-visually-complete&app_list=calendar&app=calendar&gaia_rev=a2219a55145e730e&gecko_rev=97a5d21e57ac&plot=avg) we can see that the commit https://github.com/mozilla-b2g/gaia/commit/8532cb66d11048c8 (Bug 1043857) reduced visually complete by around 150ms but still 100ms above the old value before 9f4ad73..

we can also see that between https://github.com/mozilla-b2g/gaia/compare/7306065260ceabd3...5c90a723bb824228 performance degraded around 100ms (no info between these points)

so I would say that multiple features are affecting the overall performance. I'll try to dig more info about it.

Comment 7

[Miller Medeiros [:millermedeiros]]

the perf regression between https://github.com/mozilla-b2g/gaia/compare/7306065260ceabd3...5c90a723bb824228 was caused by https://github.com/mozilla-b2g/gaia/commit/166a35556ef4e4dd48a91a4773a1bec674869e55 (Bug 1042422) - we changed the way month view works to actually wait all the calendars to be loaded before rendering the view (which adds ~100ms delay). The thing is that 2.0 also contains this patch (and performance was also affected by it).

The difference between moz-chrome-dom-loaded & moz-app-visually-complete before the patch was ~100ms, after the patch it takes ~200ms.

My investigation is complete, now we need to decide how it is going to be fixed and if we need to address it.

Comment 8

[Tony Chung [:tchung]]

looping in product.  we need to understand what is the requirement what the Calendar app will need to run at to maintain an acceptable baseline performance for 2.1.   We also need requirements defining what the acceptable amount of time it is before we are alerting an unacceptable performance.    

FYI, the 2.0 numbers show ~1160ms.  so is +30ms over acceptable?  +50ms?  +100ms?

Comment 9

[Kevin Grandon :kgrandon]

Fwiw - we have said in the past that we have a flat 1,000ms threshold for launch time across all apps. I'm not sure if that's changed recently or not. Looping in Eli as well because he probably has the most information here.

Comment 10

[:Eli Perelman]

The accepted-upon performance criteria [1] for application cold-launch times was determined to be 1000ms for each application. As far as regression determination, we generally would do investigation for anything that caused an increase of at least 30ms~50ms.

[1] https://wiki.mozilla.org/FirefoxOS/Performance/Release_Acceptance

Comment 11

[Miller Medeiros [:millermedeiros]]

can we confirm if the change to Rocketbar/Places (Bug 1063789) resolved the performance problem or if we still need to do anything else? We don't have 2.1 branch on datazilla yet..

Comment 12

[Tony Chung [:tchung]]

(In reply to Miller Medeiros [:millermedeiros] from comment #11)
> can we confirm if the change to Rocketbar/Places (Bug 1063789) resolved the
> performance problem or if we still need to do anything else? We don't have
> 2.1 branch on datazilla yet..

2.1 branch on datazilla is up at: https://datazilla.mozilla.org/b2g/?branch=v2.1&device=flame-319MB&range=7&test=startup_%3E_moz-app-visually-complete&app_list=calendar,camera,clock,communications/contacts,communications/dialer,costcontrol,email%20FTU,fm,gallery,music,settings,sms,video&app=calendar&plot=avg

but for some reasons i cant see calendar app.  was this removed by chance?

Comment 13

[Miller Medeiros [:millermedeiros]]

unassigned since commits that introduced regression aren't part of the calendar app (comment #6) and I really don't know what to do in this case.

Comment 14

[Duane Dixon [:ddixon]]

(In reply to Miller Medeiros [:millermedeiros] from comment #11)
> can we confirm if the change to Rocketbar/Places (Bug 1063789) resolved the
> performance problem or if we still need to do anything else? We don't have
> 2.1 branch on datazilla yet..

Results: 

Device: Flame 2.1
Build ID: 20140926081140
Gaia: df567808f51a8c7c33b3651542c67a277940d04d
Gecko: 3e3c69b549b0
Version: 34.0a2 
Firmware Version: v180
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

STR: 
1. Tap Rocket Bar and search for desired app.
2. Select app.
3. Allow app to open and observe "Time to Load" output. 

Average Time to Load outputs observed (10 repro averages): 

Rocket Bar search (319 MB) to: 
Calendar app ~850ms 
Contacts app ~1126ms
Settings app ~1316ms

Rocket Bar search (512 MB) to: 
Calendar app ~850ms 
Contacts app ~1037ms
Settings app ~1264ms

Comment 15

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Time to Load in the HUD isn't equivalent to the Datazilla startup tests. It only measures to first paint, not until complete above the fold visual composition. 

Note that 2.1 is in Datazilla on its own branch at this point. Devs can also test any local changes at their desk using make test-perf per https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Gaia_performance_tests

Comment 17

[Candice Serran (:cserran)]

Team confirmed this will not block but high-priority for backlog

Comment 18

[Dylan Oliver [:doliver]]

[priority] --> tracking-b2g:+ conversion

Comment 19

[Bobby Chien]

worksforme for v2.1. Improvement bug 1181018 has been created in v2.5.