1078157 - Automatic test for app launch time and loading time with silk

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Performance, defect)

Description

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Project silk will change the input, refresh driver and compositor timing. We need a automatic test script to check the launch/loading time regression problem easier.

Comment 1

[Mason Chang [Inactive] [:mchang]]

Can we use the current measurements of make test-perf to test start up times? We should be checking start up times on mostly every commit. 

Eli - Where are we with measuring start up times? Is it still make test-perf and on datazilla or are we moving to something else? Thanks!

Comment 2

[:Eli Perelman]

Yes, we are still using `make test-perf` to measure the launch latency of all apps. These measurements are captured in Datazilla [1].

[1] https://datazilla.mozilla.org/b2g/

Comment 3

[Jerry Shih[:jerry] (UTC+8) (inactive)]

I can see some data using "make test-perf".
Thanks.

---
{
  "stats": {
    "start": "2014-10-20T02:59:18.658Z",
    "end": "2014-10-20T02:59:51.998Z",
    "duration": 33340,
    "application": "calendar"
  },
}
,
{
  "stats": {
    "start": "2014-10-20T02:59:52.505Z",
    "end": "2014-10-20T03:00:25.967Z",
    "duration": 33462,
    "application": "camera"
  },
}
,
{
  "stats": {
    "start": "2014-10-20T03:00:26.256Z",
    "end": "2014-10-20T03:00:59.758Z",
    "duration": 33502,
    "application": "clock"
  },
}
,
{
  "stats": {
    "start": "2014-10-20T03:01:00.518Z",
    "end": "2014-10-20T03:01:34.088Z",
    "duration": 33570,
    "application": "communications/contacts"
  },
}

......

Comment 4

[Mason Chang [Inactive] [:mchang]]

Please just make sure we continue to measure the same start up event as the rest of b2g. The guidelines are here - https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_app_responsiveness_guidelines.

Eli, what's the event we should be listening to again? Content visible? Thanks!

Comment 5

[:Eli Perelman]

It depends. If you want to measure the launch time of an application, that would be using the "moz-app-visually-complete" event. If you want to know what event is used for when to make measurements, it's "moz-app-loaded", but "moz-app-loaded" is not to be used itself for determining any launch time.

Comment 6

[Bobby Chien]

Raptor [1] replaced test-perf for app launch testing.

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Automated_testing/Raptor