Closed Bug 1093847 Opened 10 years ago Closed 9 years ago

Cost Control's cold launch time is over the 1000 ms acceptance threshold for 2.1

Categories

(Firefox OS Graveyard :: Gaia::Cost Control, defect, P2)

ARM
Gonk (Firefox OS)
defect

Tracking

(tracking-b2g:-)

RESOLVED WONTFIX
tracking-b2g -

People

(Reporter: gmealer, Assigned: mai)

References

Details

[Blocking Requested - why for this release]:

See test results at https://wiki.mozilla.org/B2G/QA/2014-10-31_Performance_Acceptance#Cost_Control

Median launch time was 2482 ms across 480 launches. This was also a significant regression from 2.0's median launch time of 1609 ms across 360 launches.

Bug 1072621 established this test runs the Usage first-time-launch experience, so this timing applies to that.

It seems unlikely we're going to be under 1000 ms for 2.1. However, I'd like an explicit triage decision, and would like to get an updated, more releastic acceptance target for 2.1 if at all possible for future testing.
Triage: This is definitely not something we are going to be able to address in 2.1 - perhaps something for 2.2 and smart data team to look into.
blocking-b2g: 2.1? → 2.2?
Flags: needinfo?(pdolanjski)
Geo, are we able to run this test when it is not in the first-time-launch experience?
Flags: needinfo?(pdolanjski) → needinfo?(gmealer)
(In reply to Peter Dolanjski [:pdol] from comment #2)
> Geo, are we able to run this test when it is not in the first-time-launch
> experience?

Datazilla (and my formal studies) reset the phone prior to testing, with about the only intermediate step being pushing a workload to the phone. 

I did ask in in Bug 1072621 if there was a way to add something to our workload that would pre-initialize Usage and skip FTU. Another option (maybe easier) would be an available user profile setting that skips it. Either way, we're not there yet.

That said, I think I can semi-automate a one-time special run by getting past FTU manually, and disabling the portions that reset the phone. 

I won't be able to do it on a regular basis, though (our schedule counts on the tests running unattended for all apps at once) and there's no way at all to do it for the Datazilla per-build results. We should still explore some way to skip FTU by default for these tests.

I'll do that that one-time run this week and report back on what the performance of that is so we get some sort of idea how it compares to FTU. Keeping the NI alive until then.
OK, I tried staging and running a no-FTU test run, but the test times out. Visually, it's sitting in the Usage app and it never closes.

Is the moz-app-visually-stable event still thrown after the screen stabilizes in the non-FTU flow? If it's only thrown at the end of FTU, that would explain the behavior.
Flags: needinfo?(gmealer) → needinfo?(salva)
Oops, I meant moz-app-visually-complete.
Probably not. Asking :mai who put those events in place last time.
Flags: needinfo?(salva) → needinfo?(marina.rodriguez.iglesias)
Hi,
When the performance events were added, only the ftu-flow events were implemented because the other paths won't be measured. Sorry.
Flags: needinfo?(marina.rodriguez.iglesias)
triage: backlog. we have no visibility of the launch time usage app gonna take in 2.2 yet.
blocking-b2g: 2.2? → backlog
Per https://wiki.mozilla.org/B2G/QA/2014-11-14_Performance_Acceptance#Cost_Control, we're still pretty high at 2606 ms. This does only reflect the FTU flow; without instrumentation I can't precisely measure non-FTU, aside from saying it's visibly faster.

We're at CC for 2.1, so are starting to make acceptance calls. Flagging Peter for recommendations on further action, cc'ing Tony.
Flags: needinfo?(pdolanjski)
Thanks Geo.  I'm really not worried about the FTU time delta given it is only ever experienced once.  It's unfortunate that we don't have accurate measurements beyond the FTU.  Marina, can we instrument it differently in 2.2 to get the post-FTU measurements?
Flags: needinfo?(pdolanjski) → needinfo?(marina.rodriguez.iglesias)
Assignee: nobody → marina.rodriguez.iglesias
Flags: needinfo?(marina.rodriguez.iglesias)
Hi,
The posible scenarios when the costcontrol app is launched are the following:
  1. Check if exists the mandatory API (IccManager, NetworkStats and MobileConnections) if they don’t exist the app shows a message to the user.
  2. Check if exists the simcard
      a. If the simcard is not detected shows a msg to the user and we put a listener over the iccdetected event to relaunch the startup if we detect a simcard.
      b. If the simcard is detected we check its state:
        i. if is different to ready, we show a message to the user and put a listener to recheck the simcard state
        ii. If the simcard is ready, the app continues with the startup process (3)
  3. If the simcard is ready, we read the configuration and settings:
      a. If the iccId don’t have preloaded configuration we launch the FTE process to configure the app (reset period, alarms,…)
      b. The configuration depends the mcc/mnc of the simcard. The app has three different modes:
        i. Data_usage_only. (It’s the most common) --> Traffic graphic of consume
        ii. Prepaid (configurable by the user settings) --> Balance ui 
        iii. Postpaid (configurable by the user settings) --> Telephony ui
        Depending on the application mode the app loads a different ui first
      c. The load of the info of the app is asynchronous then the app usually launch twice the updateUI method, the first usually load the ui, and the second refresh this ui to update the info, to ensure the info is correctly.

Which paths do you want covering with the performance events?

Regards
Flags: needinfo?(pdolanjski)
Flags: needinfo?(gmealer)
Marina, since we are going to remove the check for the SIM card and load directly into the dashboard (after the FTU is completed), does it make sense to focus on instrumenting just that flow (after all of the initial configuration is done)?
Flags: needinfo?(pdolanjski) → needinfo?(marina.rodriguez.iglesias)
I think the primary flows to check are:

A) FTU, happy path (i.e. we launch and we're at the FTU first wizard screen). This is the currently instrumented flow.

B) Post-FTU, happy path (we launch and we're at the standard Usage screen).

I don't think the no-SIM paths need to be tested this way, at least at this time, especially given Peter's comment 13.

In terms of where the events should be thrown, moz-app-visually-complete should be triggered as soon as the screen has stabilized. You might still be loading stuff in the background asynchronously, but any content displayed on-screen should be finalized. If you need guidance on the rest of the events, let us know.

I've NI'd Eli, who's really subject-matter-expert on the instrumentation and can guide further as needed.
Flags: needinfo?(gmealer) → needinfo?(eperelman)
Hi Peter,
I'll prefer wait to the final implementation, make this now is double work. But I don't know what is the purpose of the measures, if you want to analyse the performance of something specific, I don't  mind make this now. If you only want to know the time that the App spends on the start up on 2.2, I'll prefer waiting because the startup is going to change a lot. 
Regards,
Mai
Flags: needinfo?(marina.rodriguez.iglesias)
Geo is absolutely correct here. If I may add, for performance testing in apps there seems to be 2 typical flows: first-time use and normal use, just like Geo said. If you want to automate how these flows are created, I would probably recommend you base it off the `make reference-workload-*` scripts. If you don't install a reference workload, it opens to the first-time-use as expected, but if you install a workload script, have it make the necessary changes to bypass first-time use so you can test the normal-use flow.
Flags: needinfo?(eperelman)
blocking-b2g: backlog → ---
Priority: -- → P2
Main stream is moving to 2.5 on Flame and Aries (Sony Z3C). FxOS 2.1 is low priority now. Mark as wontfix. Please reopen if any special request.
Status: NEW → RESOLVED
Closed: 9 years ago
OS: All → Gonk (Firefox OS)
Hardware: x86 → ARM
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.