844363 - Overhaul mozpool's tests

Status: RESOLVED FIXED

(Testing Graveyard :: Mozpool, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

* Split them up into multiple test scripts
* Use more mixins for common tasks
* Get better coverage
* Make it clear(er) what's well-tested and what is not

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug844363.patch

This has full coverage for the DB stuff and handlers, plus all of the coverage that the old test suite did.

----

Ran 189 tests in 2.588s
...
Name                              Stmts   Miss  Cover
-----------------------------------------------------
mozpool/__init__                      1      0   100%
mozpool/async                        27      0   100%
mozpool/bmm/__init__                  0      0   100%
mozpool/bmm/api                      43     16    63%
mozpool/bmm/handlers                 69      0   100%
mozpool/bmm/ping                      6      0   100%
mozpool/bmm/pxe                      18      0   100%
mozpool/bmm/relay                   122     12    90%
mozpool/bmm/scripts                  82      6    93%
mozpool/bmm/sut                      54     45    17%
mozpool/config                       38     17    55%
mozpool/db/__init__                  19      0   100%
mozpool/db/base                      60      0   100%
mozpool/db/device_requests           22      0   100%
mozpool/db/devices                  106      0   100%
mozpool/db/environments               6      0   100%
mozpool/db/exceptions                 1      0   100%
mozpool/db/images                    30      0   100%
mozpool/db/imaging_servers           14      0   100%
mozpool/db/inventorysync             45      0   100%
mozpool/db/model                     12      0   100%
mozpool/db/pool                      16      0   100%
mozpool/db/pxe_configs               33      0   100%
mozpool/db/requests                  64      0   100%
mozpool/db/scripts                   18      0   100%
mozpool/lifeguard/__init__            0      0   100%
mozpool/lifeguard/devicemachine     401    203    49%
mozpool/lifeguard/handlers           39      0   100%
mozpool/lifeguard/inventorysync      63      0   100%
mozpool/mozpool/__init__              0      0   100%
mozpool/mozpool/handlers             82      0   100%
mozpool/mozpool/requestmachine      172    104    40%
mozpool/statedriver                  86     60    30%
mozpool/statemachine                110     13    88%
mozpool/util                         25      4    84%
mozpool/web/__init__                  0      0   100%
mozpool/web/handlers                 67      1    99%
mozpool/web/server                   46     18    61%
-----------------------------------------------------
TOTAL                              1997    499    75%

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 717399 [details] [diff] [review]
bug844363.patch

In the course of further hacking, I've made some adjustments here.  I'll squash those in and re-post this for review (unless, of course, you've already started).

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug844363.patch

OK, the second big one.  This changes a bit of the DB stuff - mostly docstrings about where NotFound is returned, and some hacks to support testing.

This:
 * breaks up tests into lots of smaller files, rougly corresponding to modules in the regular code
 * has 100% coverage for DB and handlers -- let's keep it that way!
 * makes runtests a simple stub that finds the tests and runs them
   * 'nosetests mozpool' works fine too, and helps when running only one file
 * adds ./runcoverage.sh that gets a nice coverage report
 * fixes a bug in relay.py where socket.timeout was not caught
 * implements some integration testing of the state machines, but doesn't shoot for 100% coverage there
 * implements a bunch of purpose-specific TestCase mixins to handle common requirements

I hope you like running it as much as I liked writing it!

Comment 4

[Mark Côté [:mcote]]

Comment on attachment 719675 [details] [diff] [review]
bug844363.patch

Review of attachment 719675 [details] [diff] [review]:
-----------------------------------------------------------------

This looks great, huge improvement over the old tests.  I'll admit to having skimmed a lot of it, but I like what I saw, and it runs fine on my machine. :)

::: mozpool/db/images.py
@@ +21,2 @@
>          """
>          Get information about all visibile images, represented as dictionaries

Just realized that "visible" is misspelled here.

::: mozpool/mozpool/handlers.py
@@ +2,4 @@
>  # License, v. 2.0. If a copy of the MPL was not distributed with this
>  # file, You can obtain one at http://mozilla.org/MPL/2.0/.
>  
> +from __future__ import absolute_import

Why is this required here?

::: mozpool/test/integration/test_devicemachine.py
@@ +18,5 @@
> +        ('start_powercycle', 'mozpool.bmm.api.API.start_powercycle'),
> +    ]
> +
> +    def setUp(self):
> +        super(Tests,self).setUp()

Missing a space after the comma.

@@ +24,5 @@
> +        self.add_device('dev1', hardware_type_id=hw_id,
> +                relayinfo='relayhost:bank1:relay2')
> +
> +    def tearDown(self):
> +        super(Tests,self).tearDown()

And here.  And in most files actually.

::: mozpool/test/util.py
@@ +91,5 @@
> +    data to the database, as well.
> +    """
> +
> +    def setUpMixin(self):
> +        # note that we can't use an im-memory DB, as that is not usable

s/im/in/

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Checked in with the requested fixes.  Thanks!