507643 - Create a unit test system for the Mozmill APIs themselves.

Status: RESOLVED FIXED

(Testing Graveyard :: Mozmill, defect, P1)

Description

[cmtalbert]

I think we can use the Mozmill JUM api for this, and we don't need to create anything at all.  But we need to write the tests for the elements lib API and the controller API.

Steps for this:
1. Ensure the JUM interface will work for us (pretty sure it will)
2. Write tests
3. Check tests in.  As these are unit tests for the mozmill APIs themselves, I think we should check them in to the Mozmill source code repo.  That way when a developer for Mozmill writes a patch it will be trivial for his patch to include the necessary test as well.

And once we have these tests we need to enforce adding a test with every change to these APIs.  I'm rather ashamed we haven't done this yet.

Feel free to discuss/refine the proposal below...

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Everything works that way. See bug 498538 as an example. The test you can find here: http://code.google.com/p/mozmill/source/browse/trunk/test/test_focus.js

Tests should be checked in here:
http://code.google.com/p/mozmill/source/browse/#svn/trunk/test

Eventually we should create an extra folder for unit tests itself. So we don't collide with existing example tests. But finally we should cover all this work on the bugs itself by using the in-testsuite flag. That way we can directly see where tests are missing.

Comment 2

[Mikeal Rogers [:mikeal] (mrogers@)]

Some of those tests are very out of date at the moment. We should remove the preferences test and a few others as they were really just sample tests we wrote when building the API and don't explicitly test mozmill itself.

Comment 3

[cmtalbert]

Replying to comment 1 and 2, yes Henrik that is where we should put them.  And there is definitely old cruft in that directory, so this bug is needed to get us into a state where we can easily land new tests, which means svn deleting the cruft that is currently in there.

Once we have the directory structure setup, the idea of how the tests ought to be written and a few tests checked in, we can close this bug, and track places where tests are needed using the in-test-suite flag

Comment 4

[cmtalbert]

The framework for this tester has landed in https://github.com/mozautomation/mozmill/commit/b0e62e7f56238378794e1e12bf72f2d52e9263c4

We just need to wire the tests in to close this bug.

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Wire tests into Mutt framework

I also made some changes to mutt.py, so that it a) uses the correct ManifestParser subclass (which allows us to automatically ignore disabled tests) and b) doesn't search for all-tests.ini in the cwd, but rather its location relative to mutt.py.

Comment 6

[Andrew Halberstadt [:ahal]]

The full diff which includes moving all files from mozmill/test to mutt/tests can be found here: https://github.com/ahal/mozmill/commit/3317f7e2b180cc805d6b82dc05c01beb2094a170

Comment 7

[Andrew Halberstadt [:ahal]]

Comment on attachment 522777 [details] [diff] [review]
Wire tests into Mutt framework

Re-assigning to Jeff as Clint is busy and we probably want this now.

Comment 8

[Andrew Halberstadt [:ahal]]

(not to say that Jeff isn't busy)

Comment 9

[Jeff Hammel]

Comment on attachment 522777 [details] [diff] [review]
Wire tests into Mutt framework

+                                       default=os.path.join(os.path.dirname(__file__), "tests/all-tests.ini"),

This should almost definitely be  `os.path.join(os.path.dirname(__file__), "tests", "all-tests.ini")`

+        results = test_all_python(mp.active_tests(type='python'), options)

This will work for now.  manifestparser should probably be improved such that this actually does what it should, but that's another story that I don't want to go into here (ping me in #ateam if you care)

r=me if you fix the os.path.join issue

Comment 10

[Andrew Halberstadt [:ahal]]

master: https://github.com/mozautomation/mozmill/commit/bc4310b0e2f76b35a73bb275dfa03ce13f3553c9