1436262 - With runByManifest, Android mochitest-chrome still runs longer

Status: RESOLVED FIXED

(Testing :: Mochitest, enhancement)

Description

[Geoff Brown [:gbrown]]

Even after bug 1435624, setting runByManifest still adds 10 to 20 minutes to each task run time for some mochitest-chrome jobs.

Opportunities for optimization:
 - do not start/stop the web server each time
 - faster detection of test completion
 - faster application startup?

...or we could just run in more chunks.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this is the same issue in desktop tests- we took a 30+ second hit for each directory, I expect that hit is amplified on android.

Comment 2

[Geoff Brown [:gbrown]]

Attachment: detect test completion more efficiently

Firefox/test startup takes 30 to 60 seconds, but I don't think there's much that can be done to optimize that.

Web server start/stop takes 5 to 10 seconds and could perhaps be eliminated, but I kind-of like the additional sense of isolation with a new web server instance per manifest - I don't want to optimize it...at least not today.

There is a frequent 10 (sometimes 20) second delay after tests have completed and before the harness notices that Firefox has closed:

[task 2018-02-07T00:14:08.487Z] 00:14:08     INFO -  226 INFO Passed:  3151
[task 2018-02-07T00:14:08.487Z] 00:14:08     INFO -  227 INFO Failed:  0
[task 2018-02-07T00:14:08.488Z] 00:14:08     INFO -  228 INFO Todo:    3
[task 2018-02-07T00:14:08.488Z] 00:14:08     INFO -  229 INFO Mode:    non-e10s
[task 2018-02-07T00:14:08.488Z] 00:14:08     INFO -  230 INFO Slowest: 70755ms - chrome://mochitests/content/chrome/dom/animation/test/chrome/test_animation_performance_warning.html
[task 2018-02-07T00:14:08.488Z] 00:14:08     INFO -  231 INFO SimpleTest FINISHED
[task 2018-02-07T00:14:29.782Z] 00:14:29     INFO -  INFO | automation.py | Application ran for: 0:13:20.796362
[task 2018-02-07T00:14:29.782Z] 00:14:29     INFO -  INFO | zombiecheck | Reading PID log: /tmp/tmpxvipdwpidlog

This delay can be nearly eliminated with this simple change.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8949151 [details] [diff] [review]
detect test completion more efficiently

Review of attachment 8949151 [details] [diff] [review]:
-----------------------------------------------------------------

nice easy review

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/705fdefaf5b1
Improve efficiency of detection of end of test Android test runs; r=jmaher

Comment 5

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/705fdefaf5b1