Closed Bug 780353 Opened 8 years ago Closed 7 years ago

talos should have a test runner

Categories

(Testing :: Talos, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: k0scist, Unassigned)

References

Details

(Whiteboard: [good first bug][mentor=jhammel][lang=py])

Attachments

(1 file, 2 obsolete files)

It would be nice to have a one "click" test runner to run the talos
tests.  Since I'm the only one that runs the tests, my current methodology

cd tests
for i in *.py; do python $i; done

is probably mostly fine, but it would be easy to write a script that
did the equivalent and could be extended.  Ideally manifestparser and
moztest would be used, but itd be better not to incur these depdencies
just for testing.  A `python setup.py test` approach could also be used.
Blocks: 703014
There are several ways of doing this none of which is particularly hard.  One such way is using something like https://github.com/mozilla/mozbase/blob/master/test.py for guidance
Whiteboard: [good first bug][mentor=jhammel][lang=py]
If nobody is going for it, I would like to give it a try. Any suggested bedtime reading that I should be aware of?

"Ideally manifestparser and moztest would be used, but itd be better not to incur these depdencies just for testing."

So that means, not using manifestparser and moztest?
(In reply to nebelhom from comment #2)
> If nobody is going for it, I would like to give it a try. Any suggested
> bedtime reading that I should be aware of?

mozbase's test.py serves a similar need:

https://github.com/mozilla/mozbase/blob/master/test.py

nose is also very popular these days: http://nose.readthedocs.org/en/latest/ . damned if i know how to wired it up to setup.py, but maybe mdas or jgriffin in #ateam could help.  Ideally it would run with `python setup.py test`

> "Ideally manifestparser and moztest would be used, but itd be better not to
> incur these depdencies just for testing."
> 
> So that means, not using manifestparser and moztest?

It depends.  If they're test-only and it can work from `python setup.py test` then fine.  Otherwise, not so much
Attached patch First Patch to this bug (obsolete) — Splinter Review
Could you just try this for me please. It is a very simple change and I am scared that it is "too" simple.

Essentially added test_suite='tests' to setup(...)

The missing piece to the puzzle was adding an empty __init__.py to the tests folder. This works fine with me although the test_configuration.py always fails, but that is unrelated.

Hopefully, this is what we were looking for....
Attachment #663204 - Flags: review?(jhammel)
Comment on attachment 663204 [details] [diff] [review]
First Patch to this bug

Sorry, but we shouldn't do this.  This makes `tests` a top level module -- in other words, it will override 'tests' in the top-level python namespace.
Attachment #663204 - Flags: review?(jhammel) → review-
Attached patch seperate test.py from setup.py (obsolete) — Splinter Review
I know it is not what you asked for, but it works and could make things easier for you for now.

I will still speak to other folks, but so far I couldn't find a way around the import issue. I used the site module in this case to import the files in the tests directory, but when used to together with setup.py test, it throws me an AttributeError

AttributeError: 'module' object has no attribute 'test_oauth'

test_oauth is found in
./talos/lib/python2.7/site-packages/oauth2-1.5.211-py2.7.egg/tests

I "could" try and add all necessary directory paths to PYTHONPATH manually, but that is sounds very silly to me.

I'll be back if I find anymore out...
Attachment #663204 - Attachment is obsolete: true
Attachment #663391 - Flags: review?(jhammel)
Comment on attachment 663391 [details] [diff] [review]
seperate test.py from setup.py

This won't really work.  We don't want to have to add new tests into the test runner each time we add a new test (or remove a test, etc). Instead, we should gather tests using something like https://github.com/mozilla/mozbase/blob/master/test.py#L22 .  An additional function will be needed to gather the files from the tests directory:

test_files = [os.path.join(tests_dir, i) for i in os.listdir(tests_dir) if i.endswith('.py')]

+tests_dir = os.path.join(os.getcwd(), 'tests')

You'll want the directory of the file, not the directory of the user:

tests_dir = os.path.join(os.path.dirname(os.path.abspath(__here__)), 'tests')
Attachment #663391 - Flags: review?(jhammel) → review-
After speaking to carljm, I was able to connect the test_suite to setup.py using runtests.py. I am sure that it still needs work, but I hope it is a step in the right direction.

I just wanted to mention for completion's sake (maybe it is useful some time later) regarding imports, carljm mentioned the following:

"
i think the "proper" solution to avoid polluting the top-level namespace is to put the tests inside the existing talos namespace
talos.tests
which involves moving them in the source tree, dunno if that's a problem
"

Anyhow, hope this is better.
Attachment #663391 - Attachment is obsolete: true
Attachment #664121 - Flags: review?(jhammel)
Comment on attachment 664121 [details] [diff] [review]
setup.py adjustment with runtests.py

works like a charm.  thanks!
Attachment #664121 - Flags: review?(jhammel) → review+
Whiteboard: [good first bug][mentor=jhammel][lang=py] → [good first bug][mentor=jhammel][lang=py][talos-checkin-needed]
looks pretty green: http://hg.mozilla.org/build/talos/rev/30e7bcdcb54f
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Try run for 97a8a925b721 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=97a8a925b721
Results (out of 72 total builds):
    exception: 5
    success: 66
    failure: 1
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/jhammel@mozilla.com-97a8a925b721
Whiteboard: [good first bug][mentor=jhammel][lang=py][talos-checkin-needed] → [good first bug][mentor=jhammel][lang=py]
Depends on: 801633
You need to log in before you can comment on or make changes to this bug.