Closed
Bug 1093847
Opened 10 years ago
Closed 9 years ago
Cost Control's cold launch time is over the 1000 ms acceptance threshold for 2.1
Categories
(Firefox OS Graveyard :: Gaia::Cost Control, defect, P2)
Tracking
(tracking-b2g:-)
RESOLVED
WONTFIX
tracking-b2g | - |
People
(Reporter: gmealer, Assigned: mai)
References
Details
[Blocking Requested - why for this release]: See test results at https://wiki.mozilla.org/B2G/QA/2014-10-31_Performance_Acceptance#Cost_Control Median launch time was 2482 ms across 480 launches. This was also a significant regression from 2.0's median launch time of 1609 ms across 360 launches. Bug 1072621 established this test runs the Usage first-time-launch experience, so this timing applies to that. It seems unlikely we're going to be under 1000 ms for 2.1. However, I'd like an explicit triage decision, and would like to get an updated, more releastic acceptance target for 2.1 if at all possible for future testing.
Comment 1•10 years ago
|
||
Triage: This is definitely not something we are going to be able to address in 2.1 - perhaps something for 2.2 and smart data team to look into.
blocking-b2g: 2.1? → 2.2?
Flags: needinfo?(pdolanjski)
Comment 2•10 years ago
|
||
Geo, are we able to run this test when it is not in the first-time-launch experience?
Flags: needinfo?(pdolanjski) → needinfo?(gmealer)
Reporter | ||
Comment 3•10 years ago
|
||
(In reply to Peter Dolanjski [:pdol] from comment #2) > Geo, are we able to run this test when it is not in the first-time-launch > experience? Datazilla (and my formal studies) reset the phone prior to testing, with about the only intermediate step being pushing a workload to the phone. I did ask in in Bug 1072621 if there was a way to add something to our workload that would pre-initialize Usage and skip FTU. Another option (maybe easier) would be an available user profile setting that skips it. Either way, we're not there yet. That said, I think I can semi-automate a one-time special run by getting past FTU manually, and disabling the portions that reset the phone. I won't be able to do it on a regular basis, though (our schedule counts on the tests running unattended for all apps at once) and there's no way at all to do it for the Datazilla per-build results. We should still explore some way to skip FTU by default for these tests. I'll do that that one-time run this week and report back on what the performance of that is so we get some sort of idea how it compares to FTU. Keeping the NI alive until then.
Reporter | ||
Comment 5•10 years ago
|
||
OK, I tried staging and running a no-FTU test run, but the test times out. Visually, it's sitting in the Usage app and it never closes. Is the moz-app-visually-stable event still thrown after the screen stabilizes in the non-FTU flow? If it's only thrown at the end of FTU, that would explain the behavior.
Flags: needinfo?(gmealer) → needinfo?(salva)
Reporter | ||
Comment 6•10 years ago
|
||
Oops, I meant moz-app-visually-complete.
Comment 7•10 years ago
|
||
Probably not. Asking :mai who put those events in place last time.
Flags: needinfo?(salva) → needinfo?(marina.rodriguez.iglesias)
Assignee | ||
Comment 8•10 years ago
|
||
Hi, When the performance events were added, only the ftu-flow events were implemented because the other paths won't be measured. Sorry.
Flags: needinfo?(marina.rodriguez.iglesias)
Comment 9•10 years ago
|
||
triage: backlog. we have no visibility of the launch time usage app gonna take in 2.2 yet.
blocking-b2g: 2.2? → backlog
Reporter | ||
Comment 10•10 years ago
|
||
Per https://wiki.mozilla.org/B2G/QA/2014-11-14_Performance_Acceptance#Cost_Control, we're still pretty high at 2606 ms. This does only reflect the FTU flow; without instrumentation I can't precisely measure non-FTU, aside from saying it's visibly faster. We're at CC for 2.1, so are starting to make acceptance calls. Flagging Peter for recommendations on further action, cc'ing Tony.
Flags: needinfo?(pdolanjski)
Comment 11•10 years ago
|
||
Thanks Geo. I'm really not worried about the FTU time delta given it is only ever experienced once. It's unfortunate that we don't have accurate measurements beyond the FTU. Marina, can we instrument it differently in 2.2 to get the post-FTU measurements?
Flags: needinfo?(pdolanjski) → needinfo?(marina.rodriguez.iglesias)
Assignee | ||
Updated•10 years ago
|
Assignee: nobody → marina.rodriguez.iglesias
Flags: needinfo?(marina.rodriguez.iglesias)
Assignee | ||
Comment 12•10 years ago
|
||
Hi, The posible scenarios when the costcontrol app is launched are the following: 1. Check if exists the mandatory API (IccManager, NetworkStats and MobileConnections) if they don’t exist the app shows a message to the user. 2. Check if exists the simcard a. If the simcard is not detected shows a msg to the user and we put a listener over the iccdetected event to relaunch the startup if we detect a simcard. b. If the simcard is detected we check its state: i. if is different to ready, we show a message to the user and put a listener to recheck the simcard state ii. If the simcard is ready, the app continues with the startup process (3) 3. If the simcard is ready, we read the configuration and settings: a. If the iccId don’t have preloaded configuration we launch the FTE process to configure the app (reset period, alarms,…) b. The configuration depends the mcc/mnc of the simcard. The app has three different modes: i. Data_usage_only. (It’s the most common) --> Traffic graphic of consume ii. Prepaid (configurable by the user settings) --> Balance ui iii. Postpaid (configurable by the user settings) --> Telephony ui Depending on the application mode the app loads a different ui first c. The load of the info of the app is asynchronous then the app usually launch twice the updateUI method, the first usually load the ui, and the second refresh this ui to update the info, to ensure the info is correctly. Which paths do you want covering with the performance events? Regards
Flags: needinfo?(pdolanjski)
Flags: needinfo?(gmealer)
Comment 13•10 years ago
|
||
Marina, since we are going to remove the check for the SIM card and load directly into the dashboard (after the FTU is completed), does it make sense to focus on instrumenting just that flow (after all of the initial configuration is done)?
Flags: needinfo?(pdolanjski) → needinfo?(marina.rodriguez.iglesias)
Reporter | ||
Comment 14•10 years ago
|
||
I think the primary flows to check are: A) FTU, happy path (i.e. we launch and we're at the FTU first wizard screen). This is the currently instrumented flow. B) Post-FTU, happy path (we launch and we're at the standard Usage screen). I don't think the no-SIM paths need to be tested this way, at least at this time, especially given Peter's comment 13. In terms of where the events should be thrown, moz-app-visually-complete should be triggered as soon as the screen has stabilized. You might still be loading stuff in the background asynchronously, but any content displayed on-screen should be finalized. If you need guidance on the rest of the events, let us know. I've NI'd Eli, who's really subject-matter-expert on the instrumentation and can guide further as needed.
Flags: needinfo?(gmealer) → needinfo?(eperelman)
Assignee | ||
Comment 15•10 years ago
|
||
Hi Peter, I'll prefer wait to the final implementation, make this now is double work. But I don't know what is the purpose of the measures, if you want to analyse the performance of something specific, I don't mind make this now. If you only want to know the time that the App spends on the start up on 2.2, I'll prefer waiting because the startup is going to change a lot. Regards, Mai
Flags: needinfo?(marina.rodriguez.iglesias)
Comment 16•10 years ago
|
||
Geo is absolutely correct here. If I may add, for performance testing in apps there seems to be 2 typical flows: first-time use and normal use, just like Geo said. If you want to automate how these flows are created, I would probably recommend you base it off the `make reference-workload-*` scripts. If you don't install a reference workload, it opens to the first-time-use as expected, but if you install a workload script, have it make the necessary changes to bypass first-time use so you can test the normal-use flow.
Flags: needinfo?(eperelman)
Updated•9 years ago
|
blocking-b2g: backlog → ---
tracking-b2g:
--- → backlog
Updated•9 years ago
|
Priority: -- → P2
Comment 17•9 years ago
|
||
Main stream is moving to 2.5 on Flame and Aries (Sony Z3C). FxOS 2.1 is low priority now. Mark as wontfix. Please reopen if any special request.
Status: NEW → RESOLVED
Closed: 9 years ago
OS: All → Gonk (Firefox OS)
Hardware: x86 → ARM
Resolution: --- → WONTFIX
You need to log in
before you can comment on or make changes to this bug.
Description
•