851195 - [FTU] "startup_time" performance test fails every time for a week

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::First Time Experience, defect)

Description

[Anthony Ricaud (:rik)]

Last build with a result was 2013-03-07:
https://datazilla.mozilla.org/b2g?branch=master&range=30&test=startup_time&app=communications/ftu&app_list=browser,calendar,camera,clock,communications/contacts,communications/dialer,communications/ftu,contacts,costcontrol,email,fm,fm_radio,ftu,gallery,marketplace,messages,music,pdfjs,phone,settings,sms,template,usage,video&gaia_rev=9d3cf27b276fdce4&gecko_rev=0aa4d546b63e1568

It was using Gecko 0aa4d546b63e1568204d3b0b7df1bbb0509508a4, Gaia revision 9d3cf27b276fdce4. The next build was using the same Gecko and Gaia 3f9b4e1cbcacea71.

Comment 1

[Anthony Ricaud (:rik)]

I believe this commit https://github.com/mozilla-b2g/gaia/commit/449c2c10f388c08cc9b63a84b18c2da312b357b8 is the culprit. I don't have time to investigate now though.

Comment 2

[(inactive after 6/18) Alive Kuo [:alive]]

Hi, I could offer my help but I need to know:

1. Why a change not in FTU would affect the startup time of FTU?
2. I don't see any work is ever done about performance improvement in FTU app itself about lazy loading or something else.... Is FTU already fine tuned?

I am told that app load time in datazilla is the same as ttlview module in system app. The possibility I could figure out is the way to calculate the app load time doesn't apply to current FTU.

Thanks.

Comment 3

[Fernando Campo (:fcampo)]

(In reply to Alive Kuo [:alive] (PTO during 3/27~4/7) from comment #2)
> Hi, I could offer my help but I need to know:
> 
> 1. Why a change not in FTU would affect the startup time of FTU?
I don't see any reason for this, unless the tests are not applying correctly after the change, because there wasn't any change to FTU lately

> 2. I don't see any work is ever done about performance improvement in FTU
> app itself about lazy loading or something else.... Is FTU already fine
> tuned?
Not at all. I'm actually starting with some unit tests this week for the FTU, but apart from that, there's no work on fine tuning done for it, as there was not a high priority issue (user is waiting for the phone to load, so he is expecting some slow stuff going on this)

Comment 4

[Anthony Ricaud (:rik)]

(In reply to Alive Kuo [:alive] (PTO during 3/27~4/7) from comment #2)
> 1. Why a change not in FTU would affect the startup time of FTU?
It doesn't affect the startup time of FTU, it affects our runner. As you can see in [1], we have a script timeout for the FTU app. I believe we can fix this by changing our runner to do |NOFTU=1 make reset-gaia|.

I'm testing this right now.

[1] http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.unagi.gaia.master.mozperftest/152/artifact/perf.json

Comment 5

[Anthony Ricaud (:rik)]

It works :)

Dave: Could you replace |make reset-gaia| by |NOFTU=1 make reset-gaia| in our runner?

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Rik: It's currently "GAIA_OPTIMIZE=1 make reset-gaia" so you want "GAIA_OPTIMIZE=1 NOFTU=1 make reset-gaia"? What affect does this have? I'm just wondering if we should use this in our other jobs...

Comment 7

[Anthony Ricaud (:rik)]

This does not launch the First Time Experience at first launch. That's the one that lets you setup a few things on your phone. So instead of that, you go straight to the lock screen. You should probably enable this for most of the other jobs, except if you have one that test if the one is usable just after purchase.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Updated the mozperftest jobs in Jenkins. I'll apply it to the other jobs later.

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: FTU results are back

This appears to be fixed now. :)