1011567 - [meta] Run Performance Tests on a per checkin-basis

Status: RESOLVED DUPLICATE of bug 1042550

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect, P1)

Description

[Mason Chang [Inactive] [:mchang]]

We currently spend a lot of time doing regression bisection for performance issues. We should run the performance tests on a per checkin basis so we can get to the same point as functional tests: back out if a checkin creates a performance regression.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

In the short term finding the fewest number of tests we need per build would make this more of a reality.  If it is cold_load_time and a couple apps, lets stick with that until we get this going and can sort out a better story for more devices.

This means that our other tests might take longer to run (every 4 hours instead of every 2) depending on the number of devices we have available in any given day.

Would a better method for backfilling data automatically work better?  What I mean by this is we could prioritize running test X on 8 builds to find the problematic cause.  This would make current builds in the test queue be skipped, but probably just missing one build.

The bottleneck is # of available devices- knowing how fast we change devices and the history of android, we will always have trouble keeping a full set of available devices.

Comment 2

[Mason Chang [Inactive] [:mchang]]

Hmm, after some experimenting, I think it's pretty possible to run some b2gperf start up tests on every check in for GAIA only. I've been playing with it the past few days, and I can run 5 apps on every Gaia check in with 1 flame. I think with 3-4 flames, we could cover every gaia check in for the core apps. The only problem is updating Gecko, which is an ugly problem at the moment. I think this would be a good start.

Comment 3

[Jonathan Griffin (:jgriffin)]

Updating gecko isn't hard; we'd just use builds made by buildbot, which contain the correct versions of gecko and gaia.  Flashing is an extra step that increases total turnaround time, but I agree it should be possible to run at least a subset of tests per-commit on Flames.