Closed Bug 1021037 Opened 10 years ago Closed 10 years ago

Cold load regression in multiple applications of 70ms+ on June5


(Firefox OS Graveyard :: Gaia::Homescreen, defect, P1)

Gonk (Firefox OS)


(Not tracked)

2.0 S4 (20june)


(Reporter: Eli, Assigned: Eli)



(Keywords: perf, regression, Whiteboard: [c=regression p=2 s=2014.06.20 u=][systemsfe])

Datazilla is reporting a regression in several applications for cold load time of 70ms and greater. So far it appears affected apps are: browser, gallery, contacts, video, messages, and camera. Gallery seems to be the biggest jumper from an average of 862ms up to 1103ms. I will root cause from Gallery.
Priority: -- → P1
Gaia last-known-good: a38a6a5
Gecko last-known-good: a5bdabb

Gaia first-known-bad: d2cfef5
Gecko first-known-bad: 78b1f0d

Running b2gperf tests on a Flame as I don't have a Hamachi.


b2gperf: Gaia a38a6a5 (good), Gecko a5bdabb (good)

Results for Gallery, cold_load_time: median:392, mean:401, std: 22, max:455, min:371, all:445,393,384,375,397,443,388,394,386,418,428,435,419,392,371,388,382,435,386,382,391,455,419,383,418,389,383,380,403,394


b2gperf: Gaia d2cfef5 (bad), Gecko a5bdabb (good)

Results for Gallery, cold_load_time: median:369, mean:373, std: 20, max:417, min:338, all:375,417,408,402,360,368,374,344,373,369,358,360,368,409,362,349,369,404,382,354,360,395,396,383,380,382,345,350,338,366


b2gperf: Gaia a38a6a5 (good), Gecko 78b1f0d (bad)

Results for Gallery, cold_load_time: median:391, mean:391, std: 21, max:438, min:343, all:417,343,397,404,384,393,370,389,411,356,366,375,402,389,372,380,396,393,400,438,431,368,393,390,430,405,377,384,388,397


b2gperf: Gaia d2cfef5 (bad), Gecko 78b1f0d (bad)

Results for Gallery, cold_load_time: median:399, mean:402, std: 25, max:471, min:365, all:400,400,399,370,378,437,405,471,431,439,376,403,421,401,371,396,430,368,392,411,400,386,445,381,393,391,365,385,394,431
Assignee: eperelman → mchang
OS: Mac OS X → Gonk (Firefox OS)
Hardware: x86 → ARM
Whiteboard: [c=regression p=2 s= u=] → [c=regression p=3 s= u=]
blocking-b2g: --- → 2.0?
blocking-b2g: 2.0? → 2.0+
Gaia regression. Good gaia, good Gecko:

Results for Gallery, cold_load_time: median:995, mean:985, std: 68, max:1137, min:824, all:1032,824,880,924,990,1022,1092,995,863,1007,905,1065,988,983,1017,947,947,962,991,1009,999,1052,1012,1137,1000

Bad gaia, good gecko:
Results for Gallery, cold_load_time: median:964, mean:1050, std: 365, max:2808, min:877, all:2808,1112,1027,1077,938,1021,1008,1035,902,962,924,904,1032,990,1039,1135,877,892,1023,958,908,964,917,915,884

Looking for a higher mean and more variability.
Severity: normal → blocker
Whiteboard: [c=regression p=3 s= u=] → [c=regression p=3 s= u=2.0]
More variability comes from bug 1009111, but not the actual regression. Bissecting some more.

[2210dbd0a581fc319c6e44bb1d06f8e9bd796058] Bug 1009111 - Enable vertical homescreen by default r=timdream r=vingtetun

Results for Gallery, cold_load_time: median:953, mean:1008, std: 254, max:2209, min:851, all:2209,1075,1086,1087,945,953,851,964,903,852,965,939,1003,895,945,880,1084,952,974,886,913,980,886,1001,990
Results for Gallery, cold_load_time: median:953, mean:1015, std: 268, max:2278, min:839, all:2278,1081,1117,1091,1068,953,920,849,885,839,1018,1041,964,987,912,931,975,949,943,903,861,891,948,991,993
Results for Gallery, cold_load_time: median:983, mean:1023, std: 276, max:2338, min:860, all:2338,1070,1047,1019,978,989,903,860,943,886,1085,977,1025,880,878,897,1070,991,990,885,913,983,1016,1000,976

Previous commit:
Results for Gallery, cold_load_time: median:1018, mean:1020, std: 62, max:1149, min:916, all:1137,1035,981,994,1067,1084,1035,1104,1017,1014,1028,1149,952,916,916,1018,1006,1006,952,950,1031,923,1070,1055,1069
Results for Gallery, cold_load_time: median:1055, mean:1053, std: 64, max:1155, min:932, all:1122,1039,1055,1055,1086,1059,1031,950,1052,1049,1142,1123,1040,1130,955,1155,932,1027,1043,947,956,1124,1063,1122,1085
 Results for Gallery, cold_load_time: median:1059, mean:1051, std: 68, max:1174, min:884, all:1165,1024,1056,1085,1111,1077,1103,902,1014,1029,1086,1088,1086,1052,938,1174,983,1081,1059,1051,1002,884,1042,1084,1101
Kevin, any idea why start up of apps variability shot up so much? I think b2gperf tests when the window.onload fires.
Flags: needinfo?(kgrandon)
Hm, the commit listed in comment 3 turns on the new vertical homescreen by default. It seems that we don't have a test for homescreen launch time to help narrow this down, which is a shame.

For the homescreen we do show a lot of images at one time. I wonder if we could be swapping memory or something and that might cause apps to launch slower?

Something that we may want to do is try setting to the old homescreen:(homescreen.manifestURL=app://, and running these tests to see if the same variability exists.

Also if this does cause longer more variable launch times, then we should change the tests to not account for the homescreen - it should give you more solid timings.
Flags: needinfo?(kgrandon)
Per comment 5, I tried disabling the vertical home screen and enabling the new one. Looks like we found our regression:

 Results for Gallery, cold_load_time: median:984, mean:984, std: 51, max:1131, min:898, all:1061,954,949,987,1042,1024,1021,990,960,1008,1131,984,1008,1052,989,982,916,925,968,916,969,960,997,898,928

Results for Gallery, cold_load_time: median:1001, mean:993, std: 65, max:1111, min:857, all:1111,972,998,1019,1003,917,908,966,919,985,1002,1017,901,893,857,1009,1090,1086,1068,1095,997,1035,1001,1006,985
Results for Gallery, cold_load_time: median:991, mean:982, std: 57, max:1104, min:888, all:1041,1007,1016,904,900,1058,1027,967,991,1017,1004,946,904,888,1104,1049,1042,979,986,1008,912,904,937,942,1019
First bad commit is bug 1009111.
Closed: 10 years ago
Resolution: --- → WORKSFORME
See Also: → 1009111
Reopening with kgrandon's comment.
Resolution: WORKSFORME → ---
Blocks: 1009111
Reassigning to kevin then.
Assignee: mchang → kgrandon
Whiteboard: [c=regression p=3 s= u=2.0] → [c=regression p= s= u=2.0]
From my other bug, something that would be good to measure is if app launch really regresses from the user standpoint. To do this we would need to take a timestamp from the icon touch, to the onload event before and after the vertical homescreen.

Mason/Eli - Would either of you guys be willing to investigate if this really regresses user perceived performance of launching applications, or is this just a testing infrastructure issue?
Flags: needinfo?(mchang)
Flags: needinfo?(eperelman)
Component: Performance → Gaia::Homescreen
See Also: 1009111
Whiteboard: [c=regression p= s= u=2.0] → [c=regression p= s= u=2.0][systemsfe]
I can do some investigation on this.
Assignee: kgrandon → eperelman
Flags: needinfo?(mchang)
Flags: needinfo?(eperelman)
Whiteboard: [c=regression p= s= u=2.0][systemsfe] → [c=regression p=2 s= u=2.0][systemsfe]
Target Milestone: --- → 2.0 S4 (20june)
In doing extensive manual testing on this by getting timings of when the touch occurs to the time of entry into the Gallery app, it appears that the vertical homescreen actually performed better by an average of 50ms on a Flame device. From my measurements it I would say that this regression is invalid. While source-diving, I found that the actual tap location for opening an app was in a new location, and that, coupled with other changes may be why we are seeing what looks like a regression. But from a user-perceived perspective, this actually performs better.

ni? Mason for closing recommendation.
Flags: needinfo?(mchang)
Eli - that's super helpful information to have, thanks so much for putting that together. I do assume that this will regress by a few ms when we land bug 1023011, but this will not be anywhere near 50ms.

I do think we should still investigate why datazilla thinks this is a regression, we could do it here or in another bug. I think the next set of measurements we should take would be the actual time to launching an app. If the test framework is waiting for some homescreen state to start launching apps, perhaps we could've regressed that.

As we have confirmed that this does not cause a user-perceived impact when launching apps (and we actually have a performance improvement), I'm re-noming for 2.0? with the suggestion that we unblock.
blocking-b2g: 2.0+ → 2.0?
After talking with Eli and from comment 12, since this is an inaccurate regression, we can close this bug. The way forward is to use new make test-perf events and instrument the apps to accurately represent when an app is loaded, in bug 996038. Resolving as WFM.
Closed: 10 years ago10 years ago
Depends on: gaia-perf-events
Flags: needinfo?(mchang)
Resolution: --- → WORKSFORME
blocking-b2g: 2.0? → ---
Whiteboard: [c=regression p=2 s= u=2.0][systemsfe] → [c=regression p=2 s=2014.06.20 u=2.0][systemsfe]
No longer blocks: 1009111
Severity: blocker → major
Whiteboard: [c=regression p=2 s=2014.06.20 u=2.0][systemsfe] → [c=regression p=2 s=2014.06.20 u=][systemsfe]
You need to log in before you can comment on or make changes to this bug.