1049719 - Most applications regressing stable cold_load_time starting Aug-1

Status: RESOLVED INVALID

(Firefox OS Graveyard :: Performance, defect, P1)

Description

[:Eli Perelman]

Starting on August 1st, most applications were displaying erratic cold_load_times with higher amounts of variance as opposed to previous runs.

Comment 1

[:Eli Perelman]

I'll be referencing the SMS app through the regression detection as it seems to have well-defined good and bad points.

Comment 2

[:Eli Perelman]

Gaia last-known-good: b67ddd7
Gecko last-known-good: cf5361b

Gaia first-known-bad: e43382b
Gecko first-known-bad: f6141d6

Comment 3

[:Eli Perelman]

b2gperf --delay=10 --iterations=30 Messages
Against Gaia last-known-good, Gecko last-known-good

Results for Messages, cold_load_time:

median:582,
mean:589,
std: 36,
max:740,
min:551, all:740,676,617,576,588,567,566,599,592,571,587,578,596,602,583,567,559,626,578,560,583,607,582,551,591,582,577,555,571,565

Comment 4

[:Eli Perelman]

b2gperf --delay=10 --iterations=30 Messages
Against Gaia first-known-bad, Gecko last-known-good

Run not completed, b2gperf errors at about 9 tests in. I will bisect this to see if it's impacting the times, even though Datazilla shows no problem.

---

git bisect start e43382b b67ddd7
Bisecting: 72 revisions left to test after this (roughly 6 steps)
[9e5bc4918eec60ca1ef893c5f63d38788a4e9549] Revert "Merge pull request #22134 from lightsofapollo/revite-master"

---

git bisect bad
Bisecting: 36 revisions left to test after this (roughly 5 steps)
[76df34a5c18afd0bb68a37617e1f4adc6ff15d4e] Merge pull request #22291 from timdream/keyboard-state-control

---

git bisect bad
Bisecting: 16 revisions left to test after this (roughly 4 steps)
[add5d975a7e0028d9787e5beb8346ac94ba60c34] Merge pull request #22324 from zacc/bug_1039966

---

git bisect bad
Bisecting: 9 revisions left to test after this (roughly 3 steps)
[7198320c257340009a6d6d0e12e609058881d230] Merge pull request #21507 from azasypkin/bug-1035279-errors-mapping

---

git bisect bad
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[80d9343cdce468eee9642829e68acc9ee0e47487] Merge pull request #22229 from nullaus/bug1039631

---

git bisect bad
Bisecting: 1 revision left to test after this (roughly 1 step)
[97858243558614a4ba9d74601f802e2899588427] Merge pull request #22317 from vingtetun/landing-animation

---

git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[02b17d8c7cbaf1c5c63e371c9f157c4e6b094c24] Bug 1039519 - Implement a smooth transition to show/hide the rocketbar when scrolling up/down a page. r=kgrandon

---

git bisect bad
02b17d8c7cbaf1c5c63e371c9f157c4e6b094c24 is the first bad commit
commit 02b17d8c7cbaf1c5c63e371c9f157c4e6b094c24
Author: Vivien Nicolas <[e-mail]>
Date:   Wed Jul 30 14:51:47 2014 +0200

    Bug 1039519 - Implement a smooth transition to show/hide the rocketbar when scrolling up/down a page. r=kgrandon

:040000 040000 09a18b0d5f7e93a9df575df83845efd66650f971 741deb14a975d258df7668f6b7db18b9458fd34d M	apps
:040000 040000 c0d67ee4ccd83530afe47f9a0acd57b9435ddbb0 d542a7e7ca76841b025811a5812c9445a6c85ffa M	shared

Comment 5

[:Eli Perelman]

Running one more set against the last-known-good to verify the problem goes away:

b2gperf --delay=10 --iterations=30 Messages

Results for Messages, cold_load_time:

median:577,
mean:586,
std: 30,
max:697,
min:546, all:697,626,587,560,641,568,580,569,587,570,580,606,605,572,589,557,574,615,581,624,568,546,575,562,600,567,613,553,571,564

Confirmed. Problem is resolved in the previous commit, Gaia b67ddd7.

Comment 6

[:Eli Perelman]

The issue that is being manifested is the same one reported in bug 1046967: b2gperf crashing part-way through test runs. That issue was reportedly resolved by reverting a Gecko patch. I held Gecko steady for all the Gaia runs above, but if that patch is included in the steady Gecko, it may be that it only manifests when included with Vivien's patch above.

I will try and revert back to an earlier version of Gecko and run the test against Vivien's patch.

Comment 7

[:Eli Perelman]

Instead of trying an earlier Gecko, I will try applying the backing-out patch to the current Gecko for testing:

git format-patch -1 19e5d2d26c417bd79a6c33d7fb1b4bedfb4ec713
0001-Back-out-bug-874353-which-is-suspected-of-causing-bu.patch

git apply 0001-Back-out-bug-874353-which-is-suspected-of-causing-bu.patch

Comment 8

[:Eli Perelman]

So applying the backing-out patch and testing against Vivien's patch runs successfully:

Results for Messages, cold_load_time:

median:587,
mean:596,
std: 34,
max:735,
min:563, all:735,605,595,570,627,575,563,580,571,591,668,570,586,599,601,583,581,596,587,624,569,588,588,630,578,568,568,624,584,589

I will now re-bisect Gaia to find the original regression, but keeping the Gecko patch applied.

Comment 9

[:Eli Perelman]

Results for Messages, cold_load_time: median:592, mean:599, std: 33, max:698, min:569, all:698,666,593,607,569,619,595,668,577,589,573,598,582,572,592,572,591,575,614,592,572,678,571,596,577,599,578,597,571,617

---

Gaia seems fine, moving Gaia back to last-known-good, Gecko to first-known-bad.

Results for Messages, cold_load_time:

median:587,
mean:598,
std: 30,
max:699,
min:564, all:699,577,652,582,591,582,627,601,588,616,577,576,576,604,572,564,569,579,577,654,594,575,598,610,585,619,587,597,647,586

---

That configuration has the exact same median. Trying bad Gaia+Gecko.

Results for Messages, cold_load_time: median:594, mean:601, std: 30, max:715, min:563, all:715,615,573,578,587,585,635,585,598,582,593,599,676,617,625,580,611,563,596,585,585,580,598,582,598,620,614,573,576,612

---

All things being equal, the load time for SMS remained pretty consistent between runs of Gaia and Gecko, both good and "bad". I do not see a real regression here.