Datazilla is reporting a regression in several applications for cold load time of 70ms and greater. So far it appears affected apps are: browser, gallery, contacts, video, messages, and camera. Gallery seems to be the biggest jumper from an average of 862ms up to 1103ms. I will root cause from Gallery.
Gaia last-known-good: a38a6a5 Gecko last-known-good: a5bdabb Gaia first-known-bad: d2cfef5 Gecko first-known-bad: 78b1f0d Running b2gperf tests on a Flame as I don't have a Hamachi. --- b2gperf: Gaia a38a6a5 (good), Gecko a5bdabb (good) Results for Gallery, cold_load_time: median:392, mean:401, std: 22, max:455, min:371, all:445,393,384,375,397,443,388,394,386,418,428,435,419,392,371,388,382,435,386,382,391,455,419,383,418,389,383,380,403,394 --- b2gperf: Gaia d2cfef5 (bad), Gecko a5bdabb (good) Results for Gallery, cold_load_time: median:369, mean:373, std: 20, max:417, min:338, all:375,417,408,402,360,368,374,344,373,369,358,360,368,409,362,349,369,404,382,354,360,395,396,383,380,382,345,350,338,366 --- b2gperf: Gaia a38a6a5 (good), Gecko 78b1f0d (bad) Results for Gallery, cold_load_time: median:391, mean:391, std: 21, max:438, min:343, all:417,343,397,404,384,393,370,389,411,356,366,375,402,389,372,380,396,393,400,438,431,368,393,390,430,405,377,384,388,397 --- b2gperf: Gaia d2cfef5 (bad), Gecko 78b1f0d (bad) Results for Gallery, cold_load_time: median:399, mean:402, std: 25, max:471, min:365, all:400,400,399,370,378,437,405,471,431,439,376,403,421,401,371,396,430,368,392,411,400,386,445,381,393,391,365,385,394,431
OS: Mac OS X → Gonk (Firefox OS)
Hardware: x86 → ARM
Whiteboard: [c=regression p=2 s= u=] → [c=regression p=3 s= u=]
Gaia regression. Good gaia, good Gecko: Results for Gallery, cold_load_time: median:995, mean:985, std: 68, max:1137, min:824, all:1032,824,880,924,990,1022,1092,995,863,1007,905,1065,988,983,1017,947,947,962,991,1009,999,1052,1012,1137,1000 Bad gaia, good gecko: Results for Gallery, cold_load_time: median:964, mean:1050, std: 365, max:2808, min:877, all:2808,1112,1027,1077,938,1021,1008,1035,902,962,924,904,1032,990,1039,1135,877,892,1023,958,908,964,917,915,884 Looking for a higher mean and more variability.
Severity: normal → blocker
Whiteboard: [c=regression p=3 s= u=] → [c=regression p=3 s= u=2.0]
More variability comes from bug 1009111, but not the actual regression. Bissecting some more. [2210dbd0a581fc319c6e44bb1d06f8e9bd796058] Bug 1009111 - Enable vertical homescreen by default r=timdream r=vingtetun Results for Gallery, cold_load_time: median:953, mean:1008, std: 254, max:2209, min:851, all:2209,1075,1086,1087,945,953,851,964,903,852,965,939,1003,895,945,880,1084,952,974,886,913,980,886,1001,990 Results for Gallery, cold_load_time: median:953, mean:1015, std: 268, max:2278, min:839, all:2278,1081,1117,1091,1068,953,920,849,885,839,1018,1041,964,987,912,931,975,949,943,903,861,891,948,991,993 Results for Gallery, cold_load_time: median:983, mean:1023, std: 276, max:2338, min:860, all:2338,1070,1047,1019,978,989,903,860,943,886,1085,977,1025,880,878,897,1070,991,990,885,913,983,1016,1000,976 Previous commit: 51a39a66f65647430451f7208ecd8cd8a2dbff88 Results for Gallery, cold_load_time: median:1018, mean:1020, std: 62, max:1149, min:916, all:1137,1035,981,994,1067,1084,1035,1104,1017,1014,1028,1149,952,916,916,1018,1006,1006,952,950,1031,923,1070,1055,1069 Results for Gallery, cold_load_time: median:1055, mean:1053, std: 64, max:1155, min:932, all:1122,1039,1055,1055,1086,1059,1031,950,1052,1049,1142,1123,1040,1130,955,1155,932,1027,1043,947,956,1124,1063,1122,1085 Results for Gallery, cold_load_time: median:1059, mean:1051, std: 68, max:1174, min:884, all:1165,1024,1056,1085,1111,1077,1103,902,1014,1029,1086,1088,1086,1052,938,1174,983,1081,1059,1051,1002,884,1042,1084,1101
Kevin, any idea why start up of apps variability shot up so much? I think b2gperf tests when the window.onload fires.
Hm, the commit listed in comment 3 turns on the new vertical homescreen by default. It seems that we don't have a test for homescreen launch time to help narrow this down, which is a shame. For the homescreen we do show a lot of images at one time. I wonder if we could be swapping memory or something and that might cause apps to launch slower? Something that we may want to do is try setting to the old homescreen:(homescreen.manifestURL=app://homescreen.gaiamobile.org/manifest.webapp), and running these tests to see if the same variability exists. Also if this does cause longer more variable launch times, then we should change the tests to not account for the homescreen - it should give you more solid timings.
Per comment 5, I tried disabling the vertical home screen and enabling the new one. Looks like we found our regression: Results for Gallery, cold_load_time: median:984, mean:984, std: 51, max:1131, min:898, all:1061,954,949,987,1042,1024,1021,990,960,1008,1131,984,1008,1052,989,982,916,925,968,916,969,960,997,898,928 Results for Gallery, cold_load_time: median:1001, mean:993, std: 65, max:1111, min:857, all:1111,972,998,1019,1003,917,908,966,919,985,1002,1017,901,893,857,1009,1090,1086,1068,1095,997,1035,1001,1006,985 Results for Gallery, cold_load_time: median:991, mean:982, std: 57, max:1104, min:888, all:1041,1007,1016,904,900,1058,1027,967,991,1017,1004,946,904,888,1104,1049,1042,979,986,1008,912,904,937,942,1019
First bad commit is bug 1009111.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → WORKSFORME
Reopening with kgrandon's comment.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Reassigning to kevin then.
Assignee: mchang → kgrandon
Whiteboard: [c=regression p=3 s= u=2.0] → [c=regression p= s= u=2.0]
From my other bug, something that would be good to measure is if app launch really regresses from the user standpoint. To do this we would need to take a timestamp from the icon touch, to the onload event before and after the vertical homescreen. Mason/Eli - Would either of you guys be willing to investigate if this really regresses user perceived performance of launching applications, or is this just a testing infrastructure issue?
Component: Performance → Gaia::Homescreen
See Also: bug 1009111 →
Whiteboard: [c=regression p= s= u=2.0] → [c=regression p= s= u=2.0][systemsfe]
I can do some investigation on this.
Assignee: kgrandon → eperelman
Status: REOPENED → ASSIGNED
Whiteboard: [c=regression p= s= u=2.0][systemsfe] → [c=regression p=2 s= u=2.0][systemsfe]
In doing extensive manual testing on this by getting timings of when the touch occurs to the time of entry into the Gallery app, it appears that the vertical homescreen actually performed better by an average of 50ms on a Flame device. From my measurements it I would say that this regression is invalid. While source-diving, I found that the actual tap location for opening an app was in a new location, and that, coupled with other changes may be why we are seeing what looks like a regression. But from a user-perceived perspective, this actually performs better. ni? Mason for closing recommendation.
Eli - that's super helpful information to have, thanks so much for putting that together. I do assume that this will regress by a few ms when we land bug 1023011, but this will not be anywhere near 50ms. I do think we should still investigate why datazilla thinks this is a regression, we could do it here or in another bug. I think the next set of measurements we should take would be the actual time to launching an app. If the test framework is waiting for some homescreen state to start launching apps, perhaps we could've regressed that. As we have confirmed that this does not cause a user-perceived impact when launching apps (and we actually have a performance improvement), I'm re-noming for 2.0? with the suggestion that we unblock.
blocking-b2g: 2.0+ → 2.0?
After talking with Eli and from comment 12, since this is an inaccurate regression, we can close this bug. The way forward is to use new make test-perf events and instrument the apps to accurately represent when an app is loaded, in bug 996038. Resolving as WFM.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago → 4 years ago
Depends on: 996038
Resolution: --- → WORKSFORME
Whiteboard: [c=regression p=2 s= u=2.0][systemsfe] → [c=regression p=2 s=2014.06.20 u=2.0][systemsfe]
Severity: blocker → major
Whiteboard: [c=regression p=2 s=2014.06.20 u=2.0][systemsfe] → [c=regression p=2 s=2014.06.20 u=][systemsfe]
You need to log in before you can comment on or make changes to this bug.