bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

Shift make test-perf start time capture moment to closer to user perceived app start

RESOLVED FIXED in Firefox OS v2.0

Status

Firefox OS
Gaia::PerformanceTest
P1
normal
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: Eli, Assigned: Eli)

Tracking

({perf})

unspecified
2.1 S1 (1aug)
ARM
Gonk (Firefox OS)
Dependency tree / graph

Firefox Tracking Flags

(b2g-v2.0 fixed, b2g-v2.1 fixed)

Details

(Whiteboard: [c=automation p=4 s= u=])

Attachments

(2 attachments)

(Assignee)

Description

4 years ago
Our starting moment for make test-perf essentially occurs at document-element-inserted [1], which is after app touch, iframe instantiation, and splash screen. We need to move this starting moment to occur right after app touch. The timestamp captured at [2] appears to be the best existing candidate as tracing through the code shows that this would occur close to 0ms after touch.

[1] https://github.com/mozilla-b2g/gaia/blob/master/tests/performance/performance_helper_atom.js#L115
[2] https://mxr.mozilla.org/mozilla-central/source/dom/apps/src/Webapps.js#477
(Assignee)

Updated

4 years ago
Assignee: nobody → eperelman
(Assignee)

Updated

4 years ago
Whiteboard: [c=automation p= s= u=] → [c=automation p= s=2014.07.18 u=]
(Assignee)

Updated

4 years ago
Whiteboard: [c=automation p= s=2014.07.18 u=] → [c=automation p= s= u=]
(Assignee)

Updated

4 years ago
Whiteboard: [c=automation p= s= u=] → [c=automation p=3 s=2014.07.18 u=]
(Assignee)

Updated

4 years ago
Whiteboard: [c=automation p=3 s=2014.07.18 u=] → [c=automation p=3 s= u=]
We currently programmatically launch the app.

"App Touch" as you put it is when the homescreen get a tap. That mean instrumenting the homescreen. I'd rather instrument that separately.
(Assignee)

Comment 2

4 years ago
The goals [1] we have defined for the launching of applications is the aggregate of time between the app launch, whether user-initiated or simulated, and the invocation of "moz-app-visually-complete". The aggregate goal is to be within 1000ms, and is what we are basing our release acceptance on [2].

So the only way we can back up these goals with our measurements is to get the start time as close to user-perceived app launch as possible.

[1] https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines#Goals
[2] https://wiki.mozilla.org/FirefoxOS/Performance/Release_Acceptance#2.0
(Assignee)

Updated

4 years ago
Status: NEW → ASSIGNED
Do we launch the apps directly, or do we inject a tap to launch? Also, where does the elapsed-time measurement happen?

Typically, for a metric, you'd anchor this sort of timing on the impulse on the script side (i.e. the tap or call to a launch method). But that assumes the test is measuring the elapsed time between impulse and and the event independently.

The big reason to do it on the system side is if the events contain their own timing result, but if we're measuring the timeline from the test script then it might not be necessary to change on the system side.
We launch the app directly as said in Comment 1.
Thanks for clarifying. Programmatically could also mean user event injection, which uses a different code path.

If the test code intercepts the events directly, might make sense for the timeline to be calculated on the test side, starting at the programmatic call. If nothing else, could be a crutch while we tune the rest.
(Assignee)

Updated

4 years ago
Whiteboard: [c=automation p=3 s= u=] → [c=automation p=4 s= u=]
(Assignee)

Comment 6

4 years ago
Created attachment 8460768 [details]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22058/files

This patch captures the launch creation timestamp and passes it along for use in the tests. The startup tests have been modified to allow overloading of the results to provide a launch time delta.
Attachment #8460768 - Flags: review?(hub)
Attachment #8460768 - Flags: review?(alive)

Updated

4 years ago
Attachment #8460768 - Attachment mime type: text/plain → text/x-github-pull-request

Updated

4 years ago
Attachment #8460768 - Attachment mime type: text/x-github-pull-request → text/plain

Updated

4 years ago
Attachment #8460768 - Flags: review?(hub) → review+
Comment on attachment 8460768 [details]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22058/files

This is expected to slow down the performance result since you move the start point earlier. Please note this.
Attachment #8460768 - Flags: review?(alive) → review+
(Assignee)

Comment 8

4 years ago
Alive is correct, this will show an increase in the start time for all applications in the *new* launch events being captured by make test-perf, not the ones currently being used in b2gperf.
(Assignee)

Comment 9

4 years ago
Comment on attachment 8460768 [details]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22058/files

Requesting uplift to 2.0 as it is important for meeting our release performance acceptance criteria and providing more accurate numbers for the criteria.

[Feature/regressing bug #]: bug 996038
[User impact if declined]: none
[Describe test coverage new/current, TBPL]: These changes only affect numbers gathered for a single performance test
[Risks and why]: Low, as there are no user-perceived changes
[String/UUID change made/needed]: n/a
Attachment #8460768 - Flags: approval-gaia-v2.0?
(Assignee)

Updated

4 years ago
Keywords: checkin-needed
Merged
https://github.com/mozilla-b2g/gaia/commit/be8a862c815422acbc5f8839cf1e5080742db6cc
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
There is a regression, Need a followup.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 12

4 years ago
Created attachment 8461464 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22114
Attachment #8461464 - Flags: review?(hub)
(Assignee)

Comment 13

4 years ago
Comment on attachment 8461464 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22114

This patch needs the same gaia-approval-v2.0 for the same reason as the other patch.
Attachment #8461464 - Flags: approval-gaia-v2.0?
Comment on attachment 8461464 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22114

that's it.

r=me
Attachment #8461464 - Flags: review?(hub) → review+
Merged
https://github.com/mozilla-b2g/gaia/commit/6cc134a8d8b19ee2a30fea06ccb0013b36169c5a
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
(Assignee)

Updated

4 years ago
Depends on: 1045076

Updated

4 years ago
Attachment #8460768 - Flags: approval-gaia-v2.0? → approval-gaia-v2.0+

Updated

4 years ago
Attachment #8461464 - Flags: approval-gaia-v2.0? → approval-gaia-v2.0+
(Assignee)

Updated

4 years ago
See Also: → bug 1049129
You need to log in before you can comment on or make changes to this bug.