Closed Bug 996038 (gaia-perf-events) Opened 10 years ago Closed 10 years ago

[meta] Formulate new events for indicating the loading progress of apps

Categories

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect, P1)

ARM
Gonk (Firefox OS)
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Eli, Assigned: Eli)

References

()

Details

(Keywords: perf, Whiteboard: [c=automation p= s= u=])

User Story

The following events needed to be implemented in all core apps for more realistic performance metrics and future functionality:

'moz-chrome-dom-loaded': Emitted when the application designates that core chrome or navigation interface exists in the DOM and has been marked as ready for display, e.g. not `display: none`.

'moz-chrome-interactive': Emitted when the application designates that the core chrome or navigation interface has its events bound and is ready for user interaction.

'moz-app-visually-complete': Emitted when the application designates that it is visually loaded, e.g. the "above-the-fold" content exists in the DOM and has been marked as ready for display: e.g. not `display: none`.

'moz-content-interactive': Emitted when the application designates that it has bound events for the minimum set of functionality to allow the user to interact with the "above-the-fold" content made available in 'moz-app-visually-complete'.

'moz-app-loaded': Emitted when the application has been completely loaded, e.g. any relevant "below-the-fold" functionality has been injected into the DOM, been marked visible, and is ready for user interaction, and any required startup background processing has completed.


These events are to be triggered off the window element at the appropriate time, e.g. `window.dispatchEvent(new CustomEvent('moz-content-interactive'))`

Also, the ability to use the new events is fenced in by a whitelist in the startup test. That means the app name will need to be placed in the `mozLaunch` whitelist for master [1], and the whitelistedApps and whitelistedUnifiedApps arrays [2] [3] for v2.0.

[1] https://github.com/mozilla-b2g/gaia/blob/master/tests/performance/config.json#L23
[2] https://github.com/mozilla-b2g/gaia/blob/v2.0/tests/performance/startup_events_test.js#L11
[3] https://github.com/mozilla-b2g/gaia/blob/v2.0/tests/performance/startup_events_test.js#L26
Identify exactly how cold launch is being measured by b2gperf. Confirm which of these it's measuring, and file bugs where apps don't conform.

- App Chrome Visible
- App Content Visible
- App Interaction Ready
- App Content Ready
Status: NEW → ASSIGNED
Priority: -- → P1
Whiteboard: [c= p= s= u=] → [c=devtools p= s= u=]
Keywords: perf
None. b2gperf measures firstPaint. firstPaint event may (but doesn't have to - see incremental layout) fire when HTML+CSS+onload event is completed, but in terms of the Stages it does not directly fit into any of them.

The closest one is "App Chrome Visible" but more and more of the default apps rely on so much asynchronous JS code to generate chrome (which usually happens post firstPaint and causes reflow/flickering) that firstPaint becomes irrelevant.
I think this might be a situation where there's not an external/system way to know this, and we need to have triggers raised from the apps themselves when they think they've hit these points.
Yup. We currently do not have proper way to measure those points. 

There is a conversation going on on dev.gaia: https://groups.google.com/d/msg/mozilla.dev.gaia/OBiGuysU_uM/0jGG4L9voMIJ
According to [1], b2gperf determines cold launch via `apploadtime`, which is triggered in Gaia from [2]. The method surrounding the triggering of this event is a wrapper for `mozbrowserloadend`. I do not yet have a definitive answer, but from what I've been told `mozbrowserloadend` maps to `window.onload`. So yes, technically `window.onload` doesn't categorically fall into one of these states.

The next goal is to determine for the core apps what timings exist for the outlined states, and where those fall in relation to `apploadtime` in b2gperf. If the timings are not within our established guidelines outlined in [3], then we will file respective bugs.

[1] https://github.com/mozilla/b2gperf/blob/master/b2gperf/launchapp.js#L12
[2] https://github.com/mozilla-b2g/gaia/blob/master/apps/system/js/app_window.js#L739
[3] https://wiki.mozilla.org/B2G/Performance/Responsiveness#Goals
To clarify my previous comment, `window.onload` doesn't necessarily map to any of the given states, but each app should have one of these states that are completed by the time `window.onload` occurs. THAT is the true purpose of this bug, to find and fix apps that don't complete a certain rendering phase by the time b2gperf measures its startup time.
(In reply to Eli Perelman from comment #4)
> According to [1], b2gperf determines cold launch via `apploadtime`, which is
> triggered in Gaia from [2]. The method surrounding the triggering of this
> event is a wrapper for `mozbrowserloadend`. I do not yet have a definitive
> answer, but from what I've been told `mozbrowserloadend` maps to
> `window.onload`. So yes, technically `window.onload` doesn't categorically
> fall into one of these states.

Interesting. It means that it changed since b2gperf has been introduced: https://groups.google.com/d/msg/mozilla.dev.gaia/shtLUwzqm-Y/8gyGlWjBDWkJ

It also means that b2gperf doesn't measure "App Chrome Visible" but rather something closer to "App Interaction Ready".
Yeah, looks like later in that thread was this comment:

"We actually recently stopped using firstPaint for measuring app startup as it was bogus and didn't match real life user experience."

Your statement is probably mostly correct, but it would really depend on how much each application defers to load once `window.onload` is hit. Hence why using something external is faulty, as it gives engineers something to work around to make numbers look good, knowingly or not. It would be much better to emit an event at each of the stages to get a truer sense of when each individual application has loaded. For some apps it may make sense to be shown as loaded at `window.onload`, in which case a simulated event could be triggered, and for others that may not be the case, in which the event could be simulated whenever it makes sense.
My current proposition is to use make test-perf to trigger the following events using `PerformanceTestingHelper.dispatch`:

'chrome-visible'
'content-visible'
'interaction-ready'
'content-ready'

I think it's obvious which interaction each of these events map to. :) I will write up some bugs to get these events integrated into all of the apps once it's decided that these are the standardized event names that we want to use.
I think we could use eideticker to help visualizing/validating all of these, except for perhaps 'interaction-ready' (which is more difficult). It would be fairly straightforward to listen for these events using marionette, then insert them into the detailed timeline much like we do here for the "stable frame" (which we compute using an algorithm):

http://eideticker.mozilla.org/b2g/detail.html?id=95127b00cf3c11e39c00f0def1767b24#/framediffsums
William, I think that would be extremely useful.

If it's not apparent by Comment 7, my position is that b2gperf using `apploadtime` and hence `window.onload` is INSUFFICIENT for measuring cold launch. It cannot legitimately state across all applications when an application is ready to be interacted with by the user. The fact that switching interactions between sync and async and vice-versa and have them affect test outcomes with no perceivable difference in the UI is reason enough to implement a different mechanism for truly tracking application launch time.
Eli: it seems that your thinking goes in the direction of using those events exclusively for perf testing and not part of the app bootstrapping. Is that correct?
I think that was my initial viewpoint, but do you see value in creating a unified set of events that could encompass both?
(In reply to Eli Perelman, :Eli from comment #12)
> I think that was my initial viewpoint, but do you see value in creating a
> unified set of events that could encompass both?

Not sure yet.

I like the events you proposed and I see having them just for performance measurements a massive step forward.

But I also think that they may potentially have deeper integration with the platform.

First example that I'd like to tackle is the consequence of non-deterministic firstPaint. Analyzing the bootstrap process of all Gaia apps (bug 1000593) makes me confident that the current approach of platform guessing when to do firstPaint is not sustainable.

I doubt we will able to provide a polished user experience of app bootstrapping unless we allow an app to instruct the platform when its chrome is ready.

And 'chrome-visible' event (or 'chrome-ready'?) would be exactly such a thing.

In the future I can imagine more optimizations like:
 - before 'interaction-ready' touch events should not bubble to app
 - after 'content-visible' reflows should be deprioritized since they're less likely
 - between app startup and 'content-visible' we could deprioritize any background activities to speed up app launching

I can also imagine UX being able to design much more refined application bootstrapping experience if they were able to tie it to the events you described.

What do you think?
(In reply to Zbigniew Braniecki [:gandalf] from comment #13)
> First example that I'd like to tackle is the consequence of
> non-deterministic firstPaint. Analyzing the bootstrap process of all Gaia
> apps (bug 1000593) makes me confident that the current approach of platform
> guessing when to do firstPaint is not sustainable.
> 
> I doubt we will able to provide a polished user experience of app
> bootstrapping unless we allow an app to instruct the platform when its
> chrome is ready.

I guess we could also mimic that by making Gaia not make the app iframe visible until the event is fired. Not sure how different that is from delaying firstPaint, but sounds easier :)
If we were to go the route that events are useful outside of perf testing, which to me seem to be the case, then I would suggest not using `PerformanceTestingHelper.dispatch` to directly trigger these events, but instead trigger DOM events off of the window or document. We can then include a little script for perf testing that will listen for these DOM events and invoke the relevant dispatch methods.

Do we have any convention for the naming of DOM events, e.g. hyphenation, casing? Any other outstanding thoughts so we can move forward on this?
Gandalf, just wondering if you had any more comments regarding this before we move forward.
Flags: needinfo?(gandalf)
No, that sounds good.

The only suggestion I have is to name them to represent the state of the System, not App.

So instead of 'chrome-visible' -> 'chrome-ready' (because then it serves both purposes, measuring and app2system information).

I started prototyping a patch that allows an app to block displaying app chrome until 'chrome-ready'.

Feel free to move forward and let me know if I can help.
Flags: needinfo?(gandalf)
Maybe:

'chrome-ready'
'visible-content-ready'
'interaction-ready'
'content-ready' (or 'data-ready')
Gandalf, I like your naming suggestions but a problem with a few is that they can convey the wrong intent. For example, 'visible-content-ready' implies that visible content is ready to be interacted with, but that's really what 'interaction-ready' is about.

If we look at this from the perspective of need to measure for performance reasons, what I'm trying to achieve is determine the amount of time that has passed between when the chrome is visible and when the user can start using it. Same with the "above-the-fold" content, time difference between when it's visible and when the user can interact. The last events would measure the time difference between when the full application has loaded, e.g. the below-the-fold-content, and when it is ready for interaction.

To me this says we need 6 events:

'chrome-visible' and 'chrome-ready'
'above-the-fold-visible' and 'above-the-fold-ready' || 'core-visible' and 'core-ready'
'full-visible' and 'full-ready' || 'remainder-visible' and 'remainder-ready' || 'entire-visible' and 'entire-ready'

Just throwing a few names out there, but it seems these 3 main points always come in pairs of when they are visible and when they are ready to be used.
Based on this, I'm not convinced we should split out "chrome ready" vs. "content ready." Do we even want apps where you can start using chrome before the content is there?

More than anything, I'm concerned that this'll make the timeline considerably less linear and comparable app over app. I'd expect a lot of apps to come out chrome-vis, atf-vis, full-vis (which is just both together, right?), chrome-ready, atf-ready, full-ready. Others might come out chrome-vis, chrome-ready, atf-vis, full-vis, atf-ready, full-ready, etc.

Also, some of it (remainder-visible/entire-visible) doesn't click with me yet because the idea of loading content in the background, past above-the-fold, is that it won't be visible unless the user specifically scrolls to it.

I really liked the idea of the simple four-part mapping above, which equates (in these terms) to chrome-visible, content-visible, full-ready, remainder-ready, where remainder-ready applies -only- to background content preloaded without additional user input. 

I feel like if the other states turn out to be more useful than I'm imagining, they could be added later.
(In reply to Geo Mealer [:geo] from comment #20)
> Based on this, I'm not convinced we should split out "chrome ready" vs.
> "content ready." Do we even want apps where you can start using chrome
> before the content is there?

I believe that the answer is yes. A great example is an SMS app. "New message" button can and should be clickable ('chrome-interactive') before the visible portion of SMSes are loaded from DB ('above-the-fold-ready').

> More than anything, I'm concerned that this'll make the timeline
> considerably less linear and comparable app over app. I'd expect a lot of
> apps to come out chrome-vis, atf-vis, full-vis (which is just both together,
> right?), chrome-ready, atf-ready, full-ready. Others might come out
> chrome-vis, chrome-ready, atf-vis, full-vis, atf-ready, full-ready, etc.

And I believe that it's a more realistic scenario. Some apps will have chrome visible quick, but interactive later (like, 50-70ms later should be unnocticable). If we try to block displaying chrome until it's interactive then user will perceive it as slower app loading without any benefit for him. (users don't click on items withing <100ms after the chrome is visible).

> Also, some of it (remainder-visible/entire-visible) doesn't click with me
> yet because the idea of loading content in the background, past
> above-the-fold, is that it won't be visible unless the user specifically
> scrolls to it.

Yes. But only when it's loaded (think, browser's awesomebar or bookmarks) we can measure stable memory consumption of the app. Measuring it before is just random.
 
> I really liked the idea of the simple four-part mapping above, which equates
> (in these terms) to chrome-visible, content-visible, full-ready,
> remainder-ready, where remainder-ready applies -only- to background content
> preloaded without additional user input. 

For performance I believe it all makes sense. It's just that there's a possible additional value of an app instrumenting the system (and/or Gecko) about when certain parts of it are ready. That allows different optimizations. For example, we could become deterministic about when we'll display the app (after chrome-ready, to be certain that we avoid FOUCs).
 
> I feel like if the other states turn out to be more useful than I'm
> imagining, they could be added later.

I agree. But I believe that we should have a consistent vision of the events, even if we implement only a subset of them right now.

Eli: I believe that the real distinction between 'ready' and 'visible' is only important before firstPaint. The other three can be combined to any way you like them to be (there's no different between 'above-the-fold-ready' and 'above-the-fold-visible')

Another approach to consolidate the event list (throwing in names as well :)), based on what Eli proposed:

App -> 'chrome-ready': App sends it when the chrome of the app is ready to be displayed
System -> 'chrome-visible': System sends it when it draws the app in foreground

App -> 'chrome-interactive': App sends it when the chrome elements are bind to actions

App -> 'above-the-fold-ready': App sends it when it loaded all the data that impacts screen drawing (screen becomes "stable")

App -> 'app-ready': App sends it when it loaded all data necessary for the app to operate ('background data')


At 'chrome-ready', we can instrument the system to display in foreground (if all CSS, l10n etc. is ready)
At 'chrome-visible', we measure when the chrome of the app becomes visible
At 'chrome-interactive', we measure when the app is becoming "clickable" (and possibly enable touch events on the app)
At 'above-the-fold-ready', we measure when the users perceives the screen as loaded (and possibly take screenshots, or resume background services that we suspended when the app was launched)
At 'app-ready', we measure memory consumption and maybe cache the app for future faster cold-start.

How does it look to you guys?
For 'chrome-interaction', are you proposing that it is measuring all events for the app or just the ones related to the chrome? If that's the case, for 'above-the-fold-ready' is there a distinction between when the content is visible and when it's ready to be interacted with?
(In reply to Eli Perelman, :Eli from comment #22)
> For 'chrome-interaction', are you proposing that it is measuring all events
> for the app or just the ones related to the chrome?

Hmm, the way I thought about it is that at some point, an app will bind events to it's UI nodes. Like:

function prepareChrome() {
  var button = document.createElement('button');
  
  documen.body.appendChild(button);
  xxx.emitEvent('chrome-ready');
  enableUI();
}

function enableUI() {
  document.getElementById('newmsg').addEventListener('click', onNewMsgClicked);

  (...)

  xxx.emitEvent('chrome-interactive');
  loadMessages();
}

function loadMessages() {
  var visibleMsgs = 10;
  var loadedMsgs = 0;

  messages.onLoaded(function(msg) {
    loadedMsgs++;
    if (loadedMsgs == visibleMsgs) {
      xxx.emitEvent('above-the-fold-ready');
    }
    if (loadedMsgs == messages.totalNumber) {
      xxx.emitEvent('app-ready');
    }
  });

  messages.load();
}

> If that's the case, for
> 'above-the-fold-ready' is there a distinction between when the content is
> visible and when it's ready to be interacted with?


There may be no difference. There may be an app which will be interactive as soon as it's visible. But there also may be an app which will be able to show chrome before enabling UI, and that sounds beneficial to the perceived performance.

Does it make sense? I followed https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines
Apologies, I think maybe I didn't make myself clear. I think there needs to be the distinction between when the above-the-fold content is visible vs. when it's ready to be interacted with. The `app-ready' event I agree can be used for the rest of the remaining content for below-the-fold.
Ah, apologies as well. Misunderstood you!

So you'd like to split above-the-fold-ready from above-the-fold-visible? Can you describe the scenario in which those two are run in separate task queues?
See Also: → 936676
One more idea from the guardian's presentation at Front Trends 2014 [1] to replace "above-the-fold-ready". They use a name "app-visually-complete".

[1] https://speakerdeck.com/patrickhamann/breaking-news-at-1000ms-front-trends-2014
Component: General → Gaia::PerformanceTest
OS: Mac OS X → Gonk (Firefox OS)
Hardware: x86 → ARM
Whiteboard: [c=devtools p= s= u=] → [c=automation p= s= u=]
See Also: → 924912
Blocks: 1011577
Blocks: 1011570
See Also: → 837669
After re-reviewing every talking point I could find for this bug, I have settled on the following nomenclature:

'chrome-visible': Emitted when the application designates that core chrome or navigation interface has been made visible in the DOM.

'chrome-interactive': Emitted when the application designates that the core chrome or navigation interface has events bound and is ready for user interaction.

'app-visible': Emitted when the application designates that it is visually loaded, e.g. the "above-the-fold" content has been made visible in the DOM.

'app-interactive': Emitted when the application designates that it has bound the minimum set of functionality to allow the user to interact with the "above-the-fold" content.

'app-complete': Emitted when the application has been completely loaded, e.g. any relevant "below-the-fold" functionality has been injected into the DOM, been marked visible, and is ready for user interaction.

I am moving forward with this, so please speak now while it's early if you have any qualms. :)
Changing 'app-complete' -> 'app-loaded'. The vocabulary seems to be a better fit with existing terminology e.g. DOMContentLoaded.
That looks awesome!

Just to make sure - 'chrome-visible' is emitted by an app, so in the future we could block displaying the app by the system on this event. Is that correct?
Yes, all of these events will be emitted in the Gaia application, from JavaScript.
cool!

One note for the future is that we may want to add one more event because 'chrome-visible' is different than 'firstPaint'. I'd like to aim at making sure that firstPaint is blocked by 'chrome-visible', but that's something to deal with later.
Hey,

just want to make sure you know about this generic test already: [1]

The logic is to launch an app and then record all events until the 'startup-path-done' event.

[1] https://github.com/mozilla-b2g/gaia/blob/master/tests/performance/startup_events_test.js
Might also want to add an event to something like 'app-stable'. The way we measure memory during start up time is transient. For example, on the music app, we load the app, grab a memory profile with b2g-info, then report that data. 

The memory profile of the music app is actually, the music app starts scanning the music library and memory usage increases. After we load the artwork, the memory usage is cleaned up by the GC and memory usage becomes stable. We may want some event like that for this test case as well.
No longer blocks: 1006129
Whiteboard: [c=automation p= s= u=] → [c=automation p=4 s= u=]
Whiteboard: [c=automation p=4 s= u=] → [c=automation p=2 s= u=]
Summary: Identify Cold Launch app states measured by b2gperf → [meta] Formulate new events for indicating the loading progress of apps
Depends on: 1015384
Depends on: 1015385
Depends on: 1015386
Depends on: 1015388
Depends on: 837669
Depends on: 1015390
Depends on: 1015391
Depends on: 1015405
Depends on: 1015412
No longer depends on: 819080, 837669
After much bikeshedding and banging heads together, here is the agreed-upon nomenclature for the startup loading events:

'moz-chrome-dom-loaded': Emitted when the application designates that core chrome or navigation interface exists in the DOM and has been marked as ready for display, e.g. not `display: none`.

'moz-chrome-interactive': Emitted when the application designates that the core chrome or navigation interface has its events bound and is ready for user interaction.

'moz-app-visually-complete': Emitted when the application designates that it is visually loaded, e.g. the "above-the-fold" content exists in the DOM and has been marked as ready for display: e.g. not `display: none`.

'moz-content-interactive': Emitted when the application designates that it has bound events for the minimum set of functionality to allow the user to interact with the "above-the-fold" content made available in 'moz-app-visually-complete'.

'moz-app-complete': Emitted when the application has been completely loaded, e.g. any relevant "below-the-fold" functionality has been injected into the DOM, been marked visible, and is ready for user interaction, and any required startup background processing has completed.

(In reply to Mason Chang [:mchang] from comment #33)
> Might also want to add an event to something like 'app-stable'. The way we
> measure memory during start up time is transient. For example, on the music
> app, we load the app, grab a memory profile with b2g-info, then report that
> data.

Mason, this would also mean that your request to know when the app is in a stable state upon launch should be fulfilled by the 'moz-app-complete' event, barring of course any further interactions that would cause more processing to take place.

The important event to remember here is 'moz-content-interactive'. This will be the default event used by Datazilla to report app launch times and will use `make test-perf` to replace the cold launch numbers being generated by b2gperf.
Another important change is the loading of the "performance_testing_helper.js" script. Currently most applications load this script deferred. Deferring the loading of this script is OK as long as it doesn't affect the measurements of the outlined events.

In my testing of loading this script on my Flame, the highest I could do was 5ms to load and finish execution of the script file. Using <script defer> will still cause the script to be loaded in order, so when in doubt if the performance_testing_helper script isn't loading in the correct order, just make it non-deferred, the execution time is very minimal.
Awesome work Eli!

> The important event to remember here is 'moz-content-interactive'. This will be the default event used by Datazilla to report app launch times and will use `make test-perf` to replace the cold launch numbers being generated by b2gperf.

Will we have datazilla reports for all events?
(In reply to Zibi Braniecki [:gandalf] from comment #36)

> Will we have datazilla reports for all events?

Datazilla already reports all events.

(In reply to Eli Perelman, :Eli from comment #35)

> In my testing of loading this script on my Flame, the highest I could do was
> 5ms to load and finish execution of the script file. Using <script defer>
> will still cause the script to be loaded in order, so when in doubt if the
> performance_testing_helper script isn't loading in the correct order, just
> make it non-deferred, the execution time is very minimal.

I'dl ike to know what happens on Buri :) Closer to our targets !
(In reply to Julien Wajsberg [:julienw] from comment #37)

> I'dl ike to know what happens on Buri :) Closer to our targets !

I would too, but unfortunately I don't have one. :)
The important thing to remember about using <script defer> is that it doesn't delay the fetching and downloading of the script, only the evaluation of it. The only thing that has changed in regards to the performance script is that it is now being evaluated before the other scripts which should be a very minimal impact. It was still be "downloaded" before the other scripts.
IIRC, not using "defer" was delaying everything after it until the script was executed. Maybe putting the non-deferred script at the end of the HTML page can be a safe idea.

But I haven't followed the latest changes about script loading, maybe nowadays Gecko does not stop everything when encountering a non-deferred <script>.
(In reply to Eli Perelman, :Eli from comment #39)
> The important thing to remember about using <script defer> is that it
> doesn't delay the fetching and downloading of the script, only the
> evaluation of it. The only thing that has changed in regards to the
> performance script is that it is now being evaluated before the other
> scripts which should be a very minimal impact. It was still be "downloaded"
> before the other scripts.

But in production scripts will be merged into one for deferred scripts and one for non-deferred. So "download" part in case we keep it "deferred" is non-existing.

defer preserves the order of scripts, so if you put it on top, you'll ensure that it's content will be evaluated before any other deferred script.
As I also stated in the PR, the main script file loaded at [1] is NOT deferred, hence it will run before the performance script has a chance to bind its events.

[1] https://github.com/eliperelman/gaia/blob/bug-1015405-settings-implement-new-perf-events/apps/settings/index.html#L549
ah, right. forgot that.
No longer blocks: 1011577
Whiteboard: [c=automation p=2 s= u=] → [c=automation p= s= u=]
Blocks: 1018334
User Story: (updated)
Depends on: 1021814
MDN updated with our new events implementation details:

https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines
User Story: (updated)
I didn't update the comments here to reflect that 'moz-app-complete' was renamed to 'moz-app-loaded', my mistake.
User Story: (updated)
Depends on: 1024014
I'll be on PTO from June 13 2014 to June 22 2014, so if there are implementation questions, please feel free to ping :hub, :jdarcangelo, or :gandalf. :hub will also be providing reviews in my absence.
Sorry, that should be :justindarc for Justin.
Blocks: 1026646
Blocks: 1026662
As an update, we on the perf team are switching to using 'moz-app-visually-complete' as our primary benchmark for determining startup regression.
(In reply to Eli Perelman, :Eli from comment #48)
> As an update, we on the perf team are switching to using
> 'moz-app-visually-complete' as our primary benchmark for determining startup
> regression.

Excellent! Can we start moving to moz-chrome-interactive for datazilla's default view instead of cold-start?

My recent experience with cold-start test in b2gperf indicates that it lost any reliability - Even quite significant delays in getting to chrome-interactive were not noticable on Flame-master's cold-start test results.
Once we have a few more apps completed and visible in Datazilla, we are going to switch the default view to be 'moz-app-visually-complete' as the default view. This is tracked by bug 1011577.
Depends on: 1036436
No longer blocks: 1026646
Depends on: 1026646
Do we also have a bug to start using moz-app-loaded for test-perf's memory testing?
I don't think so, but maybe Hub has a more definitive answer.
Flags: needinfo?(hub)
No bug that I know.
Flags: needinfo?(hub)
Depends on: 1037140
Depends on: 1037520
Alias: gaia-perf-events
Depends on: 1042563
See Also: → 1044297
Depends on: 1043283
User Story: (updated)
User Story: (updated)
Assignee: eperelman → nobody
See Also: → 1063602
I consider the current implementation of this complete. Any outstanding blocking bugs/apps will be migrated to User Timing along with the current applications when the time comes.
Assignee: nobody → eperelman
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.