780353 - talos should have a test runner

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Jeff Hammel]

It would be nice to have a one "click" test runner to run the talos
tests.  Since I'm the only one that runs the tests, my current methodology

cd tests
for i in *.py; do python $i; done

is probably mostly fine, but it would be easy to write a script that
did the equivalent and could be extended.  Ideally manifestparser and
moztest would be used, but itd be better not to incur these depdencies
just for testing.  A `python setup.py test` approach could also be used.

Comment 1

[Jeff Hammel]

There are several ways of doing this none of which is particularly hard.  One such way is using something like https://github.com/mozilla/mozbase/blob/master/test.py for guidance

Comment 2

[Johannes Vogel (:nebelhom)]

If nobody is going for it, I would like to give it a try. Any suggested bedtime reading that I should be aware of?

"Ideally manifestparser and moztest would be used, but itd be better not to incur these depdencies just for testing."

So that means, not using manifestparser and moztest?

Comment 3

[Jeff Hammel]

(In reply to nebelhom from comment #2)
> If nobody is going for it, I would like to give it a try. Any suggested
> bedtime reading that I should be aware of?

mozbase's test.py serves a similar need:

https://github.com/mozilla/mozbase/blob/master/test.py

nose is also very popular these days: http://nose.readthedocs.org/en/latest/ . damned if i know how to wired it up to setup.py, but maybe mdas or jgriffin in #ateam could help.  Ideally it would run with `python setup.py test`

> "Ideally manifestparser and moztest would be used, but itd be better not to
> incur these depdencies just for testing."
> 
> So that means, not using manifestparser and moztest?

It depends.  If they're test-only and it can work from `python setup.py test` then fine.  Otherwise, not so much

Comment 4

[Johannes Vogel (:nebelhom)]

Attachment: First Patch to this bug

Could you just try this for me please. It is a very simple change and I am scared that it is "too" simple.

Essentially added test_suite='tests' to setup(...)

The missing piece to the puzzle was adding an empty __init__.py to the tests folder. This works fine with me although the test_configuration.py always fails, but that is unrelated.

Hopefully, this is what we were looking for....

Comment 5

[Jeff Hammel]

Comment on attachment 663204 [details] [diff] [review]
First Patch to this bug

Sorry, but we shouldn't do this.  This makes `tests` a top level module -- in other words, it will override 'tests' in the top-level python namespace.

Comment 6

[Johannes Vogel (:nebelhom)]

Attachment: seperate test.py from setup.py

I know it is not what you asked for, but it works and could make things easier for you for now.

I will still speak to other folks, but so far I couldn't find a way around the import issue. I used the site module in this case to import the files in the tests directory, but when used to together with setup.py test, it throws me an AttributeError

AttributeError: 'module' object has no attribute 'test_oauth'

test_oauth is found in
./talos/lib/python2.7/site-packages/oauth2-1.5.211-py2.7.egg/tests

I "could" try and add all necessary directory paths to PYTHONPATH manually, but that is sounds very silly to me.

I'll be back if I find anymore out...

Comment 7

[Jeff Hammel]

Comment on attachment 663391 [details] [diff] [review]
seperate test.py from setup.py

This won't really work.  We don't want to have to add new tests into the test runner each time we add a new test (or remove a test, etc). Instead, we should gather tests using something like https://github.com/mozilla/mozbase/blob/master/test.py#L22 .  An additional function will be needed to gather the files from the tests directory:

test_files = [os.path.join(tests_dir, i) for i in os.listdir(tests_dir) if i.endswith('.py')]

+tests_dir = os.path.join(os.getcwd(), 'tests')

You'll want the directory of the file, not the directory of the user:

tests_dir = os.path.join(os.path.dirname(os.path.abspath(__here__)), 'tests')

Comment 8

[Johannes Vogel (:nebelhom)]

Attachment: setup.py adjustment with runtests.py

After speaking to carljm, I was able to connect the test_suite to setup.py using runtests.py. I am sure that it still needs work, but I hope it is a step in the right direction.

I just wanted to mention for completion's sake (maybe it is useful some time later) regarding imports, carljm mentioned the following:

"
i think the "proper" solution to avoid polluting the top-level namespace is to put the tests inside the existing talos namespace
talos.tests
which involves moving them in the source tree, dunno if that's a problem
"

Anyhow, hope this is better.

Comment 9

[Jeff Hammel]

Comment on attachment 664121 [details] [diff] [review]
setup.py adjustment with runtests.py

works like a charm.  thanks!

Comment 10

[Jeff Hammel]

pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=97a8a925b721

Comment 11

[Jeff Hammel]

looks pretty green: http://hg.mozilla.org/build/talos/rev/30e7bcdcb54f

Comment 12

[Mozilla RelEng Bot]

Try run for 97a8a925b721 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=97a8a925b721
Results (out of 72 total builds):
    exception: 5
    success: 66
    failure: 1
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-97a8a925b721