1015427 - make test-perf not running in same order on datazilla

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect, P1)

Description

[Mason Chang [Inactive] [:mchang]]

+++ This bug was initially created as a clone of Bug #1015426 +++

From bug 1013387, bug 1013381, we had false alarm memory regressions because make test-perf increases the size of system.* values after every run. Because the music and email app were run in different order between the two revisions, the system.* values went up when in reality it was because make test-perf didn't execute the tests in the same order, causing a false alarm. Either reset the system between tests or run make test-perf in the same order every time.

Comment 1

[Mason Chang [Inactive] [:mchang]]

hub, can you take a look at this please?

Comment 2

[Hubert Figuiere [:hub]]

Sadly the current design of the make test-perf doesn't make for a deterministic order in which app tests are run as it scan through the filesystem.

Maybe this is where we should implement a definite order and use that as a baseline.

Comment 3

[Hubert Figuiere [:hub]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/19779

Yuren, while this fix is for make test-perf, it will impact in some way the integration test. We just sort the app list to keep the order deterministic on the testing slaves.

Comment 4

[Hubert Figuiere [:hub]]

Merged
https://github.com/mozilla-b2g/gaia/commit/d9874b2e74065c43f421bed0c97a02bfd662eb32