1079938 - The first run of |APP=... make test-perf| is _always_ an outlier, skewing the average result up

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[Etienne Segonzac (:etienne)]

Some commands/results pasted below for reference:

REPORTER=ConsoleMozPerf RUNS=5 MARIONETTE_RUNNER_HOST=marionette-device-host APP=calendar make test-perf
  title: startup > moz-app-loaded
  fullTitle: startup event test > calendar > startup > moz-app-loaded
  duration: 64044
  mozPerfDurations: [3129.777238,1641.949062,1633.46974,1588.753541,1570.5650520000002]
  mozPerfDurationsAverage: 1912.9029266000002

REPORTER=ConsoleMozPerf RUNS=5 MARIONETTE_RUNNER_HOST=marionette-device-host APP=communications/dialer make test-perf
  title: startup > moz-app-loaded
  fullTitle: startup event test > communications/dialer > startup > moz-app-loaded
  duration: 61192
  mozPerfDurations: [3000.92979,1449.5203640000002,1215.854219,1166.8730209999999,1254.756354]
  mozPerfDurationsAverage: 1617.5867496

Wonder if the homescreen is not busy rendering icons while we do the first launch.

Comment 1

[Hubert Figuiere [:hub]]

or cold start vs warm.

Comment 2

[:Eli Perelman]

Since test-perf is being deprecated, this issue wont be fixed. On the upside, in the replacement for test-perf the issue has been resolved, as for cold launch we prime the application before starting the performance test. See bug 1126567.