Cold load regression in multiple applications of 70ms+ on June5

RESOLVED WORKSFORME

Status

Firefox OS
Gaia::Homescreen
P1
major
RESOLVED WORKSFORME
4 years ago
4 years ago

People

(Reporter: Eli, Assigned: Eli)

Tracking

({perf, regression})

unspecified
2.0 S4 (20june)
ARM
Gonk (Firefox OS)
perf, regression

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [c=regression p=2 s=2014.06.20 u=][systemsfe])

(Assignee)

Description

4 years ago
Datazilla is reporting a regression in several applications for cold load time of 70ms and greater. So far it appears affected apps are: browser, gallery, contacts, video, messages, and camera. Gallery seems to be the biggest jumper from an average of 862ms up to 1103ms. I will root cause from Gallery.
(Assignee)

Updated

4 years ago
Priority: -- → P1
(Assignee)

Comment 1

4 years ago
Gaia last-known-good: a38a6a5
Gecko last-known-good: a5bdabb

Gaia first-known-bad: d2cfef5
Gecko first-known-bad: 78b1f0d

Running b2gperf tests on a Flame as I don't have a Hamachi.

---

b2gperf: Gaia a38a6a5 (good), Gecko a5bdabb (good)

Results for Gallery, cold_load_time: median:392, mean:401, std: 22, max:455, min:371, all:445,393,384,375,397,443,388,394,386,418,428,435,419,392,371,388,382,435,386,382,391,455,419,383,418,389,383,380,403,394

---

b2gperf: Gaia d2cfef5 (bad), Gecko a5bdabb (good)

Results for Gallery, cold_load_time: median:369, mean:373, std: 20, max:417, min:338, all:375,417,408,402,360,368,374,344,373,369,358,360,368,409,362,349,369,404,382,354,360,395,396,383,380,382,345,350,338,366

---

b2gperf: Gaia a38a6a5 (good), Gecko 78b1f0d (bad)

Results for Gallery, cold_load_time: median:391, mean:391, std: 21, max:438, min:343, all:417,343,397,404,384,393,370,389,411,356,366,375,402,389,372,380,396,393,400,438,431,368,393,390,430,405,377,384,388,397

---

b2gperf: Gaia d2cfef5 (bad), Gecko 78b1f0d (bad)

Results for Gallery, cold_load_time: median:399, mean:402, std: 25, max:471, min:365, all:400,400,399,370,378,437,405,471,431,439,376,403,421,401,371,396,430,368,392,411,400,386,445,381,393,391,365,385,394,431
Assignee: eperelman → mchang
OS: Mac OS X → Gonk (Firefox OS)
Hardware: x86 → ARM
Whiteboard: [c=regression p=2 s= u=] → [c=regression p=3 s= u=]

Updated

4 years ago
blocking-b2g: --- → 2.0?

Updated

4 years ago
blocking-b2g: 2.0? → 2.0+
Gaia regression. Good gaia, good Gecko:

Results for Gallery, cold_load_time: median:995, mean:985, std: 68, max:1137, min:824, all:1032,824,880,924,990,1022,1092,995,863,1007,905,1065,988,983,1017,947,947,962,991,1009,999,1052,1012,1137,1000

Bad gaia, good gecko:
Results for Gallery, cold_load_time: median:964, mean:1050, std: 365, max:2808, min:877, all:2808,1112,1027,1077,938,1021,1008,1035,902,962,924,904,1032,990,1039,1135,877,892,1023,958,908,964,917,915,884

Looking for a higher mean and more variability.

Updated

4 years ago
Severity: normal → blocker
Whiteboard: [c=regression p=3 s= u=] → [c=regression p=3 s= u=2.0]
More variability comes from bug 1009111, but not the actual regression. Bissecting some more.

[2210dbd0a581fc319c6e44bb1d06f8e9bd796058] Bug 1009111 - Enable vertical homescreen by default r=timdream r=vingtetun

Results for Gallery, cold_load_time: median:953, mean:1008, std: 254, max:2209, min:851, all:2209,1075,1086,1087,945,953,851,964,903,852,965,939,1003,895,945,880,1084,952,974,886,913,980,886,1001,990
Results for Gallery, cold_load_time: median:953, mean:1015, std: 268, max:2278, min:839, all:2278,1081,1117,1091,1068,953,920,849,885,839,1018,1041,964,987,912,931,975,949,943,903,861,891,948,991,993
Results for Gallery, cold_load_time: median:983, mean:1023, std: 276, max:2338, min:860, all:2338,1070,1047,1019,978,989,903,860,943,886,1085,977,1025,880,878,897,1070,991,990,885,913,983,1016,1000,976

Previous commit:
51a39a66f65647430451f7208ecd8cd8a2dbff88
Results for Gallery, cold_load_time: median:1018, mean:1020, std: 62, max:1149, min:916, all:1137,1035,981,994,1067,1084,1035,1104,1017,1014,1028,1149,952,916,916,1018,1006,1006,952,950,1031,923,1070,1055,1069
Results for Gallery, cold_load_time: median:1055, mean:1053, std: 64, max:1155, min:932, all:1122,1039,1055,1055,1086,1059,1031,950,1052,1049,1142,1123,1040,1130,955,1155,932,1027,1043,947,956,1124,1063,1122,1085
 Results for Gallery, cold_load_time: median:1059, mean:1051, std: 68, max:1174, min:884, all:1165,1024,1056,1085,1111,1077,1103,902,1014,1029,1086,1088,1086,1052,938,1174,983,1081,1059,1051,1002,884,1042,1084,1101
Kevin, any idea why start up of apps variability shot up so much? I think b2gperf tests when the window.onload fires.
Flags: needinfo?(kgrandon)
Hm, the commit listed in comment 3 turns on the new vertical homescreen by default. It seems that we don't have a test for homescreen launch time to help narrow this down, which is a shame.

For the homescreen we do show a lot of images at one time. I wonder if we could be swapping memory or something and that might cause apps to launch slower?

Something that we may want to do is try setting to the old homescreen:(homescreen.manifestURL=app://homescreen.gaiamobile.org/manifest.webapp), and running these tests to see if the same variability exists.

Also if this does cause longer more variable launch times, then we should change the tests to not account for the homescreen - it should give you more solid timings.
Flags: needinfo?(kgrandon)
Per comment 5, I tried disabling the vertical home screen and enabling the new one. Looks like we found our regression:

 Results for Gallery, cold_load_time: median:984, mean:984, std: 51, max:1131, min:898, all:1061,954,949,987,1042,1024,1021,990,960,1008,1131,984,1008,1052,989,982,916,925,968,916,969,960,997,898,928

Results for Gallery, cold_load_time: median:1001, mean:993, std: 65, max:1111, min:857, all:1111,972,998,1019,1003,917,908,966,919,985,1002,1017,901,893,857,1009,1090,1086,1068,1095,997,1035,1001,1006,985
Results for Gallery, cold_load_time: median:991, mean:982, std: 57, max:1104, min:888, all:1041,1007,1016,904,900,1058,1027,967,991,1017,1004,946,904,888,1104,1049,1042,979,986,1008,912,904,937,942,1019
First bad commit is bug 1009111.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → WORKSFORME
See Also: → bug 1009111
Reopening with kgrandon's comment.
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---

Updated

4 years ago
Blocks: 1009111
Reassigning to kevin then.
Assignee: mchang → kgrandon
Whiteboard: [c=regression p=3 s= u=2.0] → [c=regression p= s= u=2.0]
From my other bug, something that would be good to measure is if app launch really regresses from the user standpoint. To do this we would need to take a timestamp from the icon touch, to the onload event before and after the vertical homescreen.

Mason/Eli - Would either of you guys be willing to investigate if this really regresses user perceived performance of launching applications, or is this just a testing infrastructure issue?
Flags: needinfo?(mchang)
Flags: needinfo?(eperelman)

Updated

4 years ago
Component: Performance → Gaia::Homescreen
See Also: bug 1009111
Whiteboard: [c=regression p= s= u=2.0] → [c=regression p= s= u=2.0][systemsfe]
(Assignee)

Comment 11

4 years ago
I can do some investigation on this.
Assignee: kgrandon → eperelman
Status: REOPENED → ASSIGNED
Flags: needinfo?(mchang)
Flags: needinfo?(eperelman)
(Assignee)

Updated

4 years ago
Whiteboard: [c=regression p= s= u=2.0][systemsfe] → [c=regression p=2 s= u=2.0][systemsfe]
Target Milestone: --- → 2.0 S4 (20june)
(Assignee)

Comment 12

4 years ago
In doing extensive manual testing on this by getting timings of when the touch occurs to the time of entry into the Gallery app, it appears that the vertical homescreen actually performed better by an average of 50ms on a Flame device. From my measurements it I would say that this regression is invalid. While source-diving, I found that the actual tap location for opening an app was in a new location, and that, coupled with other changes may be why we are seeing what looks like a regression. But from a user-perceived perspective, this actually performs better.

ni? Mason for closing recommendation.
Flags: needinfo?(mchang)
Eli - that's super helpful information to have, thanks so much for putting that together. I do assume that this will regress by a few ms when we land bug 1023011, but this will not be anywhere near 50ms.

I do think we should still investigate why datazilla thinks this is a regression, we could do it here or in another bug. I think the next set of measurements we should take would be the actual time to launching an app. If the test framework is waiting for some homescreen state to start launching apps, perhaps we could've regressed that.

As we have confirmed that this does not cause a user-perceived impact when launching apps (and we actually have a performance improvement), I'm re-noming for 2.0? with the suggestion that we unblock.
blocking-b2g: 2.0+ → 2.0?
After talking with Eli and from comment 12, since this is an inaccurate regression, we can close this bug. The way forward is to use new make test-perf events and instrument the apps to accurately represent when an app is loaded, in bug 996038. Resolving as WFM.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago4 years ago
Depends on: 996038
Flags: needinfo?(mchang)
Resolution: --- → WORKSFORME

Updated

4 years ago
blocking-b2g: 2.0? → ---
Whiteboard: [c=regression p=2 s= u=2.0][systemsfe] → [c=regression p=2 s=2014.06.20 u=2.0][systemsfe]
No longer blocks: 1009111

Updated

4 years ago
Severity: blocker → major
Whiteboard: [c=regression p=2 s=2014.06.20 u=2.0][systemsfe] → [c=regression p=2 s=2014.06.20 u=][systemsfe]
You need to log in before you can comment on or make changes to this bug.