1426674 - Speed up DAMP by not pre-running firefox on the new profile

Status: RESOLVED FIXED

(Testing :: Talos, enhancement)

Description

[Alexandre Poirot [:ochameau]]

Here:
https://searchfox.org/mozilla-central/source/testing/talos/talos/ffsetup.py#200-201
We run Firefox once, before running the tests.
This step is very slow. Firefox stays open for quite a bit and it drastically slow down iteration when working on DAMP or any DAMP test script.

I'm trying to suggest everyone in DevTools to engage with DAMP and contribute to test scripts, but `./mach talos-test --activeTests damp` is very slow to startup, everytime. bug 1409046 is another way to speed it up.

I would like to suggest simply not pre-running Firefox as it doesn't seem to really affect DAMP results:
https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=mozilla-central&newProject=try&newRevision=638f8f1b68034c222702b903ff245493aaf84234&originalSignature=edaec66500db21d37602c99daa61ac983f21a6ac&newSignature=edaec66500db21d37602c99daa61ac983f21a6ac&showOnlyConfident=1&framework=1&selectedTimeRange=172800
  "close" tests are known to be flaky and aren't very important.
  So I would say it is worth the trade for a few seconds faster to run.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this is a great idea- we would have to find a way to do this either locally, or conditionally for damp only.  adding this to the list to look at in January- although we have a lot of high priority stuff in January, so this might get bumped a bit.  I really think fixing the venv stuff would be an overall win, although I think that is a few days of work.

Comment 2

[Alexandre Poirot [:ochameau]]

As for bug 1409046, I'm willing to contribute if that's not too complex.

Do you think this patch is ok or should I do something more generic, involving DAMP manifest?

Comment 3

[Alexandre Poirot [:ochameau]]

Attachment: Bug 1426674 - Disable profile pre-run for DAMP.

DAMP doesn't seem to require running Firefox once before running the tests.
DAMP results are about the same with/without warming up the profile.

Review commit: https://reviewboard.mozilla.org/r/209136/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/209136/

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8938429 [details]
Bug 1426674 - Disable profile pre-run for DAMP.

https://reviewboard.mozilla.org/r/209136/#review214894

oh, this is real simple- and addresses all my concerns!  thanks for putting this together

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b5b951c9a615
Disable profile pre-run for DAMP. r=jmaher

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/b5b951c9a615