Closed Bug 1070615 Opened 8 years ago Closed 8 years ago

Make changes to Gaia performance tests to support running home screen and system tests

Categories

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect, P1)

ARM
Gonk (Firefox OS)
defect

Tracking

(tracking-b2g:backlog)

RESOLVED FIXED
2.1 S7 (24Oct)
tracking-b2g backlog

People

(Reporter: Eli, Assigned: Eli)

References

Details

(Whiteboard: [systemsfe])

Attachments

(1 file)

The mechanisms behind Gaia performance tests, `make test-perf`, currently support running tests against individual applications that have clearly defined launch lifecycles, but don't have a well-defined approach for testing against other types of apps like the home screen and system. We need to make changes to `make test-perf` to support running these tests.
Also activities. See bug 1007421.
See Also: → 1007421
... and keyboard apps, but I don't know if the bug already exists.
Maybe this should block bug 940164, would be nice to have some bare minimum perf indicators before doing the next wave of refactorings
blocking-b2g: --- → backlog
Priority: -- → P1
Target Milestone: 2.1 S5 (26sep) → 2.1 S6 (10oct)
Depends on: 1079700
This patch enables tests to be run against the system app. This introductory patch is meant only for doing manual tests using `make test-perf` and as such, will not report any results to Datazilla. There are a few markers introduced into the System and Vertical Home apps, and is captured by storing a value on an object rather than events as with other performance events, as using events was very buggy with the reboot process.

Usage is:

APP=system TEST_BOOT=2 make test-perf

The JSON reported back currently contains 3 measurements:

mozSystemLoadEnd - When the System app finishes its execution of window.onload
mozOsLogoEnd - When the System app removes the boot logo
mozHomescreenStart - When the Homescreen has first JS initialization

I have reserved TEST_BOOT=1 if we decide that this test should start reporting the Datazilla in the future. A Datazilla-compatible implementation is a larger time effort, and this patch is merely to enable one-off measurements of the System app launch from boot. Since the Flame's timing implementation can be buggy at boot, these timestamps are measured from performance.requestStart. For a closer measurement of how long the boot process took, you can look at the test start, end, and duration, e.g.:

"start": "2014-10-09T18:26:55.874Z",
"end": "2014-10-09T18:27:23.413Z",
"duration": 27539

but this is not a precise measurement of boot, just a close estimate.
Attachment #8502676 - Flags: review?(kgrandon)
Attachment #8502676 - Flags: review?(jlal)
Attachment #8502676 - Flags: feedback?(mhenretty)
Another note, the Datazilla-compatible implementation would include taking advantage of the RUNS variable, so this test will only run once.
Comment on attachment 8502676 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25007

Not super-happy about the shim, but I think this is fine for now. We should probably plan on making that shim thing real at some point :) Thanks for the patch!
Attachment #8502676 - Flags: review?(kgrandon) → review+
Yeah, I'm not happy about it either, and I've got some things in the works with Web API to change it. :)
Comment on attachment 8502676 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25007

I'm not the expert here, but this seems to give us what we want, so fb+ for that!
Attachment #8502676 - Flags: feedback?(mhenretty) → feedback+
Comment on attachment 8502676 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/25007

James gave a verbal OK on this in IRC.
Attachment #8502676 - Flags: review?(jlal)
Keywords: checkin-needed
This causes the unit tests to fail, probably because window.mozPerformance needs to be shimmed or something. Please fix and make sure we have a green run before landing.
Flags: needinfo?(eperelman)
Keywords: checkin-needed
Hmm, weird, window.mozPerformance IS shimmed in shared/js/performance_testing_helper.js. I'll look into why the unit test is failing.
Flags: needinfo?(eperelman)
Looks like the unit tests for the failing files use their own requires for the performance_testing_helper. I have since updated these files and recommitted. Waiting for a green try run before continuing.
Target Milestone: 2.1 S6 (10oct) → 2.1 S7 (Oct24)
https://github.com/mozilla-b2g/gaia/commit/3dd94aa4020f32a7e7217c673e6006c289abf99b
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Depends on: 1069863
This test has issues running as-is, see bug 1069863 comment 33. I have a local script running to work around these issues and gather data, but this test should not be used in any official capacity until bug 1069863 is resolved.
Depends on: 1061797
blocking-b2g: backlog → ---
You need to log in before you can comment on or make changes to this bug.