1021037 - Cold load regression in multiple applications of 70ms+ on June5

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Gaia::Homescreen, defect, P1)

Description

[:Eli Perelman]

Datazilla is reporting a regression in several applications for cold load time of 70ms and greater. So far it appears affected apps are: browser, gallery, contacts, video, messages, and camera. Gallery seems to be the biggest jumper from an average of 862ms up to 1103ms. I will root cause from Gallery.

Comment 1

[:Eli Perelman]

Gaia last-known-good: a38a6a5
Gecko last-known-good: a5bdabb

Gaia first-known-bad: d2cfef5
Gecko first-known-bad: 78b1f0d

Running b2gperf tests on a Flame as I don't have a Hamachi.

---

b2gperf: Gaia a38a6a5 (good), Gecko a5bdabb (good)

Results for Gallery, cold_load_time: median:392, mean:401, std: 22, max:455, min:371, all:445,393,384,375,397,443,388,394,386,418,428,435,419,392,371,388,382,435,386,382,391,455,419,383,418,389,383,380,403,394

---

b2gperf: Gaia d2cfef5 (bad), Gecko a5bdabb (good)

Results for Gallery, cold_load_time: median:369, mean:373, std: 20, max:417, min:338, all:375,417,408,402,360,368,374,344,373,369,358,360,368,409,362,349,369,404,382,354,360,395,396,383,380,382,345,350,338,366

---

b2gperf: Gaia a38a6a5 (good), Gecko 78b1f0d (bad)

Results for Gallery, cold_load_time: median:391, mean:391, std: 21, max:438, min:343, all:417,343,397,404,384,393,370,389,411,356,366,375,402,389,372,380,396,393,400,438,431,368,393,390,430,405,377,384,388,397

---

b2gperf: Gaia d2cfef5 (bad), Gecko 78b1f0d (bad)

Results for Gallery, cold_load_time: median:399, mean:402, std: 25, max:471, min:365, all:400,400,399,370,378,437,405,471,431,439,376,403,421,401,371,396,430,368,392,411,400,386,445,381,393,391,365,385,394,431

Comment 2

[Mason Chang [Inactive] [:mchang]]

Gaia regression. Good gaia, good Gecko:

Results for Gallery, cold_load_time: median:995, mean:985, std: 68, max:1137, min:824, all:1032,824,880,924,990,1022,1092,995,863,1007,905,1065,988,983,1017,947,947,962,991,1009,999,1052,1012,1137,1000

Bad gaia, good gecko:
Results for Gallery, cold_load_time: median:964, mean:1050, std: 365, max:2808, min:877, all:2808,1112,1027,1077,938,1021,1008,1035,902,962,924,904,1032,990,1039,1135,877,892,1023,958,908,964,917,915,884

Looking for a higher mean and more variability.

Comment 3

[Mason Chang [Inactive] [:mchang]]

More variability comes from bug 1009111, but not the actual regression. Bissecting some more.

[2210dbd0a581fc319c6e44bb1d06f8e9bd796058] Bug 1009111 - Enable vertical homescreen by default r=timdream r=vingtetun

Results for Gallery, cold_load_time: median:953, mean:1008, std: 254, max:2209, min:851, all:2209,1075,1086,1087,945,953,851,964,903,852,965,939,1003,895,945,880,1084,952,974,886,913,980,886,1001,990
Results for Gallery, cold_load_time: median:953, mean:1015, std: 268, max:2278, min:839, all:2278,1081,1117,1091,1068,953,920,849,885,839,1018,1041,964,987,912,931,975,949,943,903,861,891,948,991,993
Results for Gallery, cold_load_time: median:983, mean:1023, std: 276, max:2338, min:860, all:2338,1070,1047,1019,978,989,903,860,943,886,1085,977,1025,880,878,897,1070,991,990,885,913,983,1016,1000,976

Previous commit:
51a39a66f65647430451f7208ecd8cd8a2dbff88
Results for Gallery, cold_load_time: median:1018, mean:1020, std: 62, max:1149, min:916, all:1137,1035,981,994,1067,1084,1035,1104,1017,1014,1028,1149,952,916,916,1018,1006,1006,952,950,1031,923,1070,1055,1069
Results for Gallery, cold_load_time: median:1055, mean:1053, std: 64, max:1155, min:932, all:1122,1039,1055,1055,1086,1059,1031,950,1052,1049,1142,1123,1040,1130,955,1155,932,1027,1043,947,956,1124,1063,1122,1085
 Results for Gallery, cold_load_time: median:1059, mean:1051, std: 68, max:1174, min:884, all:1165,1024,1056,1085,1111,1077,1103,902,1014,1029,1086,1088,1086,1052,938,1174,983,1081,1059,1051,1002,884,1042,1084,1101

Comment 4

[Mason Chang [Inactive] [:mchang]]

Kevin, any idea why start up of apps variability shot up so much? I think b2gperf tests when the window.onload fires.

Comment 5

[Kevin Grandon :kgrandon]

Hm, the commit listed in comment 3 turns on the new vertical homescreen by default. It seems that we don't have a test for homescreen launch time to help narrow this down, which is a shame.

For the homescreen we do show a lot of images at one time. I wonder if we could be swapping memory or something and that might cause apps to launch slower?

Something that we may want to do is try setting to the old homescreen:(homescreen.manifestURL=app://homescreen.gaiamobile.org/manifest.webapp), and running these tests to see if the same variability exists.

Also if this does cause longer more variable launch times, then we should change the tests to not account for the homescreen - it should give you more solid timings.

Comment 6

[Mason Chang [Inactive] [:mchang]]

Per comment 5, I tried disabling the vertical home screen and enabling the new one. Looks like we found our regression:

 Results for Gallery, cold_load_time: median:984, mean:984, std: 51, max:1131, min:898, all:1061,954,949,987,1042,1024,1021,990,960,1008,1131,984,1008,1052,989,982,916,925,968,916,969,960,997,898,928

Results for Gallery, cold_load_time: median:1001, mean:993, std: 65, max:1111, min:857, all:1111,972,998,1019,1003,917,908,966,919,985,1002,1017,901,893,857,1009,1090,1086,1068,1095,997,1035,1001,1006,985
Results for Gallery, cold_load_time: median:991, mean:982, std: 57, max:1104, min:888, all:1041,1007,1016,904,900,1058,1027,967,991,1017,1004,946,904,888,1104,1049,1042,979,986,1008,912,904,937,942,1019

Comment 7

[Mason Chang [Inactive] [:mchang]]

First bad commit is bug 1009111.

Comment 8

[Jason Smith [:jsmith]]

Reopening with kgrandon's comment.

Comment 9

[Mason Chang [Inactive] [:mchang]]

Reassigning to kevin then.

Comment 10

[Kevin Grandon :kgrandon]

From my other bug, something that would be good to measure is if app launch really regresses from the user standpoint. To do this we would need to take a timestamp from the icon touch, to the onload event before and after the vertical homescreen.

Mason/Eli - Would either of you guys be willing to investigate if this really regresses user perceived performance of launching applications, or is this just a testing infrastructure issue?

Comment 11

[:Eli Perelman]

I can do some investigation on this.

Comment 12

[:Eli Perelman]

In doing extensive manual testing on this by getting timings of when the touch occurs to the time of entry into the Gallery app, it appears that the vertical homescreen actually performed better by an average of 50ms on a Flame device. From my measurements it I would say that this regression is invalid. While source-diving, I found that the actual tap location for opening an app was in a new location, and that, coupled with other changes may be why we are seeing what looks like a regression. But from a user-perceived perspective, this actually performs better.

ni? Mason for closing recommendation.

Comment 13

[Kevin Grandon :kgrandon]

Eli - that's super helpful information to have, thanks so much for putting that together. I do assume that this will regress by a few ms when we land bug 1023011, but this will not be anywhere near 50ms.

I do think we should still investigate why datazilla thinks this is a regression, we could do it here or in another bug. I think the next set of measurements we should take would be the actual time to launching an app. If the test framework is waiting for some homescreen state to start launching apps, perhaps we could've regressed that.

As we have confirmed that this does not cause a user-perceived impact when launching apps (and we actually have a performance improvement), I'm re-noming for 2.0? with the suggestion that we unblock.

Comment 14

[Mason Chang [Inactive] [:mchang]]

After talking with Eli and from comment 12, since this is an inaccurate regression, we can close this bug. The way forward is to use new make test-perf events and instrument the apps to accurately represent when an app is loaded, in bug 996038. Resolving as WFM.