999036 - Let mozregression call a test framework to determine regression range automatically #41

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[William Lachance (:wlach)]

[import from github]

whimboo: 

Right now it doesn't seem to be possible to run mozregression with a test framework like Mozmill to let it determine the regression range automatically. In most cases we have a simplified test we could make use of for that purpose.

So having a command line option like --test and --framework would be great.

Comment 1

[Julien Pagès (:parkouss)]

I think that could be a great feature for mozregression.

We could implement this by allowing users to specify their own mozregression.test_runner.TestRunner subclass at runtime. We could add a command line parameter (say '--test-class') which will specify the TestRunner class to use. The default value for this option would be 'mozregression.test_runner.ManualTestRunner' to keep the current behaviour (asking user to type 'good', 'bad', ...).

A subclass would then be written like this:


from mozregression.test_runner import TestRunner

class AlwaysGoodTestRunner(TestRunner):
    def evaluate(self, build_info):
        # as an example, each build will be good.
        return 'g'


Then we could use the following command line:

mozregression -g 2014-11-10 -b 2014-11-15 --test-class 'mymodule.AlwaysGoodTestRunner'


This is a plugin system for test runners.

----------------

We could also implement a Mozmill version inside mozregression, and users could use it with:


mozregression -g 2014-11-10 -b 2014-11-15 --test-class 'mozregression.test_runner.MozmillTestRunner'


We will probably need some options specifics to TestRunner classes, and this could be done with another parameter:


mozregression -g 2014-11-10 -b 2014-11-15 --test-class 'mozregression.test_runner.MozmillTestRunner' --test-options -t somefile.js

Where everything after --test-options will be passed for the test class initialization (using argparse.REMAINDER)


Or maybe instead of a MozmillTestRunner subclass we could implement a more generic ProcessTestRunner:


mozregression -g 2014-11-10 -b 2014-11-15 --test-class 'mozregression.test_runner.ProcessTestRunner' --test-options --command 'mozmill -app firefox -b %(binary)s -t path/to/test.js'

Where we can replace some values (like '%(binary)s' here) at runtime, and/or pass some environment variables to the testing subprocess.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Why can't it simply be a command line string you specify to launch whatever test harness you have, and check its exit code? With 0 returned all is good, and anything else bad.

Comment 3

[Julien Pagès (:parkouss)]

(In reply to Henrik Skupin (:whimboo) from comment #2)
> Why can't it simply be a command line string you specify to launch whatever
> test harness you have, and check its exit code? With 0 returned all is good,
> and anything else bad.

Why not. I agree that it may be an overkill to define a plugin system, and what you propose may be sufficient and is simpler.

Still there are some values that needs to be translated, the first one that I can see is the binary path (in every case except fennec I suppose). The advantage of a python hook is that it allows to manipulate every accessible variable directly, at the cost of writing python code, knowing some of the mozregression internals and probably the need to play a bit with PYTHONPATH.

I like your proposal, because it will be a lot simpler and language independant. What do you think of something like:

mozregression -g 2014-11-10 -b 2014-11-15 --test-command '/path/to/executable args'

where '/path/to/executable args' will be called in a subprocess with some environment variables, like:

MOZREGRESSION_APP # the tested app (firefox, fennec, ...)
MOZREGRESSION_BINARY # binary path to the tested app, when applicable
MOZREGRESSION_CHSET # the changeset of the tested app
...

Then we could use mozmill like this I suppose:

mozregression -g 2014-11-10 -b 2014-11-15 --test-command 'mozmill -app firefox -b $MOZREGRESSION_BINARY -t path/to/test.js'

or even better write a simple executable script for the command.

What do you think ?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For me that looks fine! Maybe make it a `--command` option, because it's any kind of command you could execute. See the `hg bisect` version of it:

-c --command CMD use command to check changeset state

For placeholders I like your idea from comment 1 by using %. That way we might be able to map known options of mozregression to the command.

Comment 5

[Julien Pagès (:parkouss)]

Ok, I think that we found something great.

I agree with the -c, --command. Let's reuse a command name that people might already know. :)

For the placeholders, I think I prefer the env vars idea (`hg bisect` also use this technique). There is one problem that I can see with '%', sometimes users may have to escape them, so it could be tricky.

But this will be rare (and maybe never a problem), so I think that we could implement the two possibilities this way:

ENV VAR               | PLACEHOLDER
----------------------------------------
MOZREGRESSION_APP     | %(app)s
MOZREGRESSION_BINARY  | %(binary)s
...                   | ...

Comment 6

[Julien Pagès (:parkouss)]

I'm working on it. I will wait for Bug 1118967 - not strictly required, but it may help to automate runs for fennec.

Comment 7

[Julien Pagès (:parkouss)]

Attachment: Test command to automatize mozregression runs

William, please tell if that's OK for you, particularly the two methods placeholders / env vars;

I'm not sure what to think on this - simplicity of use of placeholders, safety and standard of env vars. I choosed to implement the two methods. :)

Comment 8

[William Lachance (:wlach)]

Comment on attachment 8546733 [details] [review]
Test command to automatize mozregression runs

This looks great, thanks! The only thing missing is documentation, but we can land this as-is and add that later.

Comment 9

[Julien Pagès (:parkouss)]

I rebased it (due to changes of bug Bug 1120068) so it can be merged now, and will test it on windows today. I will tell you more about that when this is done.

For the documentation, I intended to add it in gh-pages branch, as a follow up of bug 1110222. Maybe a new section would be great, say "Automatic Run" or something better, with a useful example and details. We can discuss about it more once the code is merged in master. :)

Comment 10

[Julien Pagès (:parkouss)]

Well, I will not be able to test this under windows until last week. However, I did some tests on linux (steps are here: http://parkouss.github.io/mozregression/documentation/automatic-bisection.html#test-case-with-mozmill) with success.

Comment 11

[Julien Pagès (:parkouss)]

I am thinking that instead of %(VAR)s for placeholder we could use {VAR}. That's easier to read and write, and esily done with string.format. I can update the pull request since this is not merged.

What do you think guys ?

Comment 12

[Julien Pagès (:parkouss)]

Henrik, what do you think the idea of using {VAR} placeholders in place of %(VAR)s ?

mozregression --good 2014-12-03 --bad 2015-01-10 --command 'mozmill --binary {binary} -t test1.js'

mozregression --good 2014-12-03 --bad 2015-01-10 --command 'mozmill --binary %(binary)s -t test1.js'

William seems to be busy with something else, but I'm pretty sure that if you like the idea, he will too. :)

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sounds fine to me, it's a lesser code-based approach and might be closer what people expect to use.

Comment 14

[Julien Pagès (:parkouss)]

Great! Thanks for the fast answer.

I rebased the pull request, and merged it in master. Documentation will come later, probably with bug 1110222.