668345 - create initial testing framework for AUS3

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

This bug is intended to track the creation of, and addition of initial tests to, a testing framework for AUS3.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: augment existing tests to be easier to run & easier to create

This patch started off as a simple wrapper to run-tests.py that used tests.manifest to make it easy to run through all of the tests. I ended up coming up with some ideas to improve things further, especially while working on fallback support. Here's the gist of things:
- Switch from using huge nightly backups, to using tarballs from /opt/aus2/snippets/staging. I think this makes it a tad clearer what's being tested, and what the state of the snippets is. It was easier to add new tests with this, too, because I didn't have to look inside of nightly backups to find out which one I needed.
- As part of the above, I renamed the existing test directory to be version based rather than date based.
- The other big thing I did was extend generate-json.py to build up JSON files based on a candidates directory. Note that it doesn't add most of the platform-independent data, mainly just MAR information. It also doesn't handle the fact that we have multiple Mac build targets.
- Remove checked in versions of update.db, since runtests.sh uses "-c".

Hopefully I didn't take too many liberties with your initial work, do let me know if I've gone too far.

I've also created some tests for bug 668348, but I'll post those in that bug.

I still want to:
- Create some very small tests (maybe as small as a single snippet) as unit-style tests. With enough of these, it would be superfast to do sanity checks on any changes (as opposed to the ~30s it takes to run the full set with the tarballs). Some ideas are: version globbing, fallback support, rule priorities, throttling, ppc blocking - and I'm sure there's lots more. In my own playing around I found that most failures I could create would end up failing 100s or 1000s of tests, so I think these could be really useful in addition to the large swaths.
- Rewrite runtests.sh in python - for maintainability, really.
- Create a way to selective run tests, rather than always running the whole lot.
- Cleanup generate-json.py a LOT, the platform translation stuff is really awful right now.

Sorry for the poor descriptions on the binary files, I've forgotten how to make Mercurial play nice with them.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: improve platfom translation logic, add more tests, improve wrapper

This patch mostly adds on the first one. I added a couple of small tests for version globbing & rule priorities, revamped the platform translation to be much more understandable, and updated the test running script to make selective test running possible.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 544254 [details] [diff] [review]
improve platfom translation logic, add more tests, improve wrapper

Good start!

>diff --git a/aus-data-snapshots/3.6.12-3.6.13/20101027-Firefox-3.6.12-beta.tar.gz b/aus-data-snapshots/3.6.12-3.6.13/20101027-Firefox-3.6.12-beta.tar.gz

We could limit the number of locales to make running these tests a lot faster. If we pick the top 5, and add ja/ja-JP-mac and any other platform specific cases. But I'm hoping we set up python based unit-tests too, as we don't want to hang on to these snippets forever. Feel free to rip into test-rules.py if it makes avoiding shell easier.

>diff --git a/generate-json.py b/generate-json.py
>+def isMac(platform, version):
>+    if 'osx' in platform or 'Universal' in platform or 'u-ppc-i386' in platform:
...
>+def isMac64(platform, version):
>+    if 'mac64' in platform or 'osx64' in platform or 'Darwin_x86_64' in platform or 'u-i386-x86_64' in platform:

Can we put the ''u-i386-x86_64' test over in isMac() and treat the mac64 build as a temporary insanity ? That's how I'm thinking of it these days.

>+def getPlatforms(platform, version):
>+    if isMac(platform, version):
>+        # Most ppc+i386 universal builds use the old style mapping
>+        if version < '3.5.16' or (version >= '3.6a1' and version < '3.6.13') or (version >= '3.7a1' and version < '4.0b5'):

I suspect we'll need a proper version comparator here, something that understands alpha/beta/rc and possibly dotted versions. eg from python 2.6.6:
>>> '3.6.1' >= '3.6a1'
False
>>> '3.6b5' < '3.6.13'
False
so both of those fail one side of the 'and'.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> Comment on attachment 544254 [details] [diff] [review] [review]
> improve platfom translation logic, add more tests, improve wrapper
> 
> Good start!
> 
> >diff --git a/aus-data-snapshots/3.6.12-3.6.13/20101027-Firefox-3.6.12-beta.tar.gz b/aus-data-snapshots/3.6.12-3.6.13/20101027-Firefox-3.6.12-beta.tar.gz
> 
> We could limit the number of locales to make running these tests a lot
> faster. If we pick the top 5, and add ja/ja-JP-mac and any other platform
> specific cases.

That's a good point. If we do this, we can probably add all the files directly, too, instead of using tarballs. (Much easier to work with within Mercurial, IMHO).

> But I'm hoping we set up python based unit-tests too, as we
> don't want to hang on to these snippets forever.

Yeah, I wasn't quite sure how to go about this at first. Now that I've got a better grasp on things, I think I may have a better idea how to unit test this stuff. I'll poke at it a bit more.

> Feel free to rip into
> test-rules.py if it makes avoiding shell easier.

OK :)

> >diff --git a/generate-json.py b/generate-json.py
> >+def isMac(platform, version):
> >+    if 'osx' in platform or 'Universal' in platform or 'u-ppc-i386' in platform:
> ...
> >+def isMac64(platform, version):
> >+    if 'mac64' in platform or 'osx64' in platform or 'Darwin_x86_64' in platform or 'u-i386-x86_64' in platform:
> 
> Can we put the ''u-i386-x86_64' test over in isMac() and treat the mac64
> build as a temporary insanity ? That's how I'm thinking of it these days.

What exactly do you mean by treating it as temporary insanity for our purposes here? Ignore it?

> >+def getPlatforms(platform, version):
> >+    if isMac(platform, version):
> >+        # Most ppc+i386 universal builds use the old style mapping
> >+        if version < '3.5.16' or (version >= '3.6a1' and version < '3.6.13') or (version >= '3.7a1' and version < '4.0b5'):
> 
> I suspect we'll need a proper version comparator here, something that
> understands alpha/beta/rc and possibly dotted versions. eg from python 2.6.6:
> >>> '3.6.1' >= '3.6a1'
> False
> >>> '3.6b5' < '3.6.13'
> False
> so both of those fail one side of the 'and'.

Good catch. Do you have any thoughts on depending on the build/tools libraries? The version comparator is probably something that should live in there, and unifying the platform translations with the ones in release/platforms.py would be good, too. I don't think we'd want to depend on it in such a way that we have to update AUS with a new version every week, but maybe we can manually take code drops from it from time to time?

Comment 5

[John Hopkins (:jhopkins)]

We should get some Thunderbird test data in there as well.  Ping me on IRC?

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #5)
> We should get some Thunderbird test data in there as well.  Ping me on IRC?

I don't really think there's Thunderbird specific stuff in AUS, but I'll be happy to use it in some of the tests.

Comment 7

[John Hopkins (:jhopkins)]

I see a number of version comparisons determining the platform and other things. For example:

+def isMac64(platform, version):
+    if 'mac64' in platform or 'osx64' in platform or 'Darwin_x86_64' in platform or 'u-i386-x86_64' in platform:
+        return True
+    if platform == 'mac' and version >= '4.0b7':
+        return True
+    return False

4.0b7 is Firefox specific.

Comment 8

[[e-mail] (:bhearsum)]

(In reply to comment #7)
> I see a number of version comparisons determining the platform and other
> things. For example:
> 
> +def isMac64(platform, version):
> +    if 'mac64' in platform or 'osx64' in platform or 'Darwin_x86_64' in
> platform or 'u-i386-x86_64' in platform:
> +        return True
> +    if platform == 'mac' and version >= '4.0b7':
> +        return True
> +    return False
> 
> 4.0b7 is Firefox specific.

Oh, that. That's just test related code. The version stuff might go away, too, depending on nthomas' comment. If not, I'm happy to support other apps here, too.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #4)
> > Can we put the ''u-i386-x86_64' test over in isMac() and treat the mac64
> > build as a temporary insanity ? That's how I'm thinking of it these days.
> 
> What exactly do you mean by treating it as temporary insanity for our
> purposes here? Ignore it?

I've tended to think of the change of mac builds in terms of buildbot platforms, so macosx -> macosx64, but bug 558180 has a better approach: we've had a macosx-universal build which has been
* Darwin_Universal-gcc3
* Darwin_{ppc,x86}-gcc3-u-ppc-i386
* Darwin_{x86,x86_64}-gcc3-u-i386-x86_64
and 64bit 'slim' build that was just Darwin_x86_64-gcc3 and only existed for a few releases. I'm hoping that simplifies the logic, but it's possible it just moves the complexity around.

> Good catch. Do you have any thoughts on depending on the build/tools
> libraries? 

Hmm. Utilites like generate-json.py might be fine, but I'm not so keen on a runtime requirement for the server.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to comment #9)
> (In reply to comment #4)
> > > Can we put the ''u-i386-x86_64' test over in isMac() and treat the mac64
> > > build as a temporary insanity ? That's how I'm thinking of it these days.
> > 
> > What exactly do you mean by treating it as temporary insanity for our
> > purposes here? Ignore it?
> 
> I've tended to think of the change of mac builds in terms of buildbot
> platforms, so macosx -> macosx64, but bug 558180 has a better approach:
> we've had a macosx-universal build which has been
> * Darwin_Universal-gcc3
> * Darwin_{ppc,x86}-gcc3-u-ppc-i386
> * Darwin_{x86,x86_64}-gcc3-u-i386-x86_64
> and 64bit 'slim' build that was just Darwin_x86_64-gcc3 and only existed for
> a few releases. I'm hoping that simplifies the logic, but it's possible it
> just moves the complexity around.

I think I see what you mean, and it looks pretty easy to do here, and I _think_ it ends up cleaner. Need to test it some more before I know for sure.

> > Good catch. Do you have any thoughts on depending on the build/tools
> > libraries? 
> 
> Hmm. Utilites like generate-json.py might be fine, but I'm not so keen on a
> runtime requirement for the server.

Makes sense to me.

Comment 11

[[e-mail] (:bhearsum)]

Attachment: change mac platform translations, get rid of runtests.sh

generate-json.py:
- Changes the Mac platform translations per your suggestions and indeed, I think it's clearer this way.
- I added a simple version comparator. It works for most of our version numbers.

test-rules.py:
- Made some adjustments to make testing individual things possible
- Dropped update.db in favour of an in-memory SQlite database
- I ripped out the static test, because it didn't seem very useful now that we've got quite a few other tests. Do correct me if I'm off base there, though.

This also fixes a tiny bug in AUS-server.py.

Comment 12

[[e-mail] (:bhearsum)]

Attachment: a few unit tests

Here's one possible approach to writing some unit tests. It might be a bit early to go further with this because we may end up making changes that break this pretty seriously (eg, slotting in a database wrapper like SQLAlchemy, or just refactoring all the parsing functions). Once we're in a place where we're more confident that the class/method signatures are more static, I think these sort of tests will be useful.

Unlike the similar tests I showed you last month, these focus on the actual workhorse methods rather than the little helpers.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 545680 [details] [diff] [review]
change mac platform translations, get rid of runtests.sh

>diff --git a/AUS-server.py b/AUS-server.py
>--- a/AUS-server.py
>+++ b/AUS-server.py
>@@ -70,7 +70,7 @@ if __name__ == "__main__":
>     AUS = AUS3(dbname=options.db)
> 
>     Handler = AUS3HTTPServer
>-    httpd = SocketServer.TCPServer(("", int(options.port)), Handler)
>+    httpd = SocketServer.TCPServer(("", options.port), Handler)

What is this patch relative to ? The int(options.port) isn't present in 
 http://hg.mozilla.org/users/nthomas_mozilla.com/aus3-proto/file/be442ee5ecda/AUS-server.py

Comment 14

[[e-mail] (:bhearsum)]

(In reply to comment #13)
> Comment on attachment 545680 [details] [diff] [review] [review]
> change mac platform translations, get rid of runtests.sh
> 
> >diff --git a/AUS-server.py b/AUS-server.py
> >--- a/AUS-server.py
> >+++ b/AUS-server.py
> >@@ -70,7 +70,7 @@ if __name__ == "__main__":
> >     AUS = AUS3(dbname=options.db)
> > 
> >     Handler = AUS3HTTPServer
> >-    httpd = SocketServer.TCPServer(("", int(options.port)), Handler)
> >+    httpd = SocketServer.TCPServer(("", options.port), Handler)
> 
> What is this patch relative to ? The int(options.port) isn't present in 
>  http://hg.mozilla.org/users/nthomas_mozilla.com/aus3-proto/file/
> be442ee5ecda/AUS-server.py

Whoops, I committed the _fix_ for this on my default branch instead of my working branch, so this diff ended up backwards.

Comment 15

[[e-mail] (:bhearsum)]

Attachment: fix AUS-server.py diff, support limiting by locale in generate-json.py, minimize test cases

I figured I should minimise these test cases too, based on your feedback in bug 668348. I went so far as to only test a single platform, too, since none of these tests are for platform specific things.

Additionally, I added supported to limit generate-json.py by locale, since we probably won't be importing full sets of data for tests at this point.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 545940 [details] [diff] [review]
fix AUS-server.py diff, support limiting by locale in generate-json.py, minimize test cases

>diff --git a/AUS-server.py b/AUS-server.py
>+    httpd = SocketServer.TCPServer(("", int(options.port)), Handler)

type="int" might be nicer in the parser.add_option for the port.
 
>diff --git a/AUS.py b/AUS.py
>+            # XXX: Not sure we should be specifying patch types here, but it's
>+            # required for tests that have null snippets in them at the time
>+            # of writing.
>+            return {"partial": "", "complete": ""}

Seems fine for now. There's a bunch of dodgy handling of function returns that will need cleaning up sometime.

>diff --git a/aus-data-snapshots/3.6.12-3.6.13/snippets/Firefox/3.6.13/Darwin_Universal-gcc3/20101122044323/de/beta/complete.txt b/aus-data-snapshots/3.6.12-3.6.13/snippets/Firefox/3.6.13/Darwin_Universal-gcc3/20101122044323/de/beta/complete.txt

I was surprised all these 3.6.13 build1/2 snippets weren't failing because of missing json files for those buildsets. Presumably testQuery['name'] is getting None from AUS.identifyRequest (and that's not fatal like I thought it was), but the from: "*" match in the 3.6.13 json provides the complete and that matches our cludge. Not sure why the fake partials match though.

We should make a call on whether identifyRequest failing should be fatal or not, which I think boils down to how strict we want to be on data integrety and if we want to limit updates to our own builds or not. Doesn't have to be right here right now, but any thoughts ? 

>diff --git a/aus-data-snapshots/version-globbing/rules.sql b/aus-data-snapshots/version-globbing/rules.sql
>+INSERT INTO update_paths (priority, mapping, throttle, update_type, product, version, channel) VALUES (50, 'Firefox-3.6.12-build1', 100, 'minor', 'Firefox', '3.6.1*', 'release');

I bet '3.6.1*' doesn't match '3.6.1', but shell would. That's probably a feature.

>diff --git a/generate-json.py b/generate-json.py
>+def isMac(platform, version):
...
>+    if platform == 'macosx64' and versionGte(version, '4.0b8'):

I think you want 4.0b7 here, that was the first build of the new universal (see stage:~cltbld/old-candidates-dirs/firefox). Same in isMac64().

>+# StrictVersion _almost_ works for all of our version numbers.

So close!

>+def getPlatforms(platform, version):
...
>+        if (versionGte(version, '3.5.16') and versionLt(version, '3.6a1')) or \

I'd have a preference to writing this as
          if (version.startswith("3.5") and versionGte(version, '3.5.16'))
so I don't have to work back from knowing 3.6a1 is the first release of the next branch. If that's OK with you there are a few other places like that.

>+def getInfoFile(ftpPlatform, bbPlatform):
>+    return "%s_info.txt" % bbPlatform

ftpPlatform isn't used.

test-rules.py should say which testdir it's working on when reporting the results.

r+ with the nits fixed.

Comment 17

[[e-mail] (:bhearsum)]

(In reply to comment #16)
> Comment on attachment 545940 [details] [diff] [review] [review]
> fix AUS-server.py diff, support limiting by locale in generate-json.py,
> minimize test cases
> 
> >diff --git a/AUS-server.py b/AUS-server.py
> >+    httpd = SocketServer.TCPServer(("", int(options.port)), Handler)
> 
> type="int" might be nicer in the parser.add_option for the port.

Ah, good point.

> >diff --git a/aus-data-snapshots/3.6.12-3.6.13/snippets/Firefox/3.6.13/Darwin_Universal-gcc3/20101122044323/de/beta/complete.txt b/aus-data-snapshots/3.6.12-3.6.13/snippets/Firefox/3.6.13/Darwin_Universal-gcc3/20101122044323/de/beta/complete.txt
> 
> I was surprised all these 3.6.13 build1/2 snippets weren't failing because
> of missing json files for those buildsets. Presumably testQuery['name'] is
> getting None from AUS.identifyRequest (and that's not fatal like I thought
> it was), but the from: "*" match in the 3.6.13 json provides the complete
> and that matches our cludge. Not sure why the fake partials match though.

I'm pretty sure the partials match because of fakePartials=1:
(from AUS.expandRelease)
            if 'fakePartials' in relData and relData['fakePartials'] and len(updateData['patches']) == 1 and \
              updateData['patches'][0]['type'] == 'complete':

> We should make a call on whether identifyRequest failing should be fatal or
> not, which I think boils down to how strict we want to be on data integrety
> and if we want to limit updates to our own builds or not. Doesn't have to be
> right here right now, but any thoughts ?

The only potential downside I can see to serving updates to things we can identify is the case that someone somehow manages to build something that's *not* Firefox, but identifies to AUS like it, and we give it a complete update that morphs it into Firefox. That's extremely farfetched though, and might be impossible. So, I think I lean towards being more liberal unless we can come up with a compelling reason to be strict.

> 
> >diff --git a/aus-data-snapshots/version-globbing/rules.sql b/aus-data-snapshots/version-globbing/rules.sql
> >+INSERT INTO update_paths (priority, mapping, throttle, update_type, product, version, channel) VALUES (50, 'Firefox-3.6.12-build1', 100, 'minor', 'Firefox', '3.6.1*', 'release');
> 
> I bet '3.6.1*' doesn't match '3.6.1', but shell would. That's probably a
> feature.

Hm, yeah. The tricky thing here is that we *do* want to be able to express "3.6.10 through 3.6.19", but we probably don't want to add too many metacharacters. The sucky part about the current behaviour is that it's different than what '*' means everywhere else, which could be confusing. Maybe we should support '*' and '+'.

> >diff --git a/generate-json.py b/generate-json.py
> >+def isMac(platform, version):
> ...
> >+    if platform == 'macosx64' and versionGte(version, '4.0b8'):
> 
> I think you want 4.0b7 here, that was the first build of the new universal
> (see stage:~cltbld/old-candidates-dirs/firefox). Same in isMac64().

Whoops, yeah.

> >+def getPlatforms(platform, version):
> ...
> >+        if (versionGte(version, '3.5.16') and versionLt(version, '3.6a1')) or \
> 
> I'd have a preference to writing this as
>           if (version.startswith("3.5") and versionGte(version, '3.5.16'))
> so I don't have to work back from knowing 3.6a1 is the first release of the
> next branch. If that's OK with you there are a few other places like that.

Good point, I'll fix that up.

> >+def getInfoFile(ftpPlatform, bbPlatform):
> >+    return "%s_info.txt" % bbPlatform
> 
> ftpPlatform isn't used.
> 
> test-rules.py should say which testdir it's working on when reporting the
> results.

Will fix these.

Comment 18

[[e-mail] (:bhearsum)]

Attachment: address additional review comments

Comment 19

[[e-mail] (:bhearsum)]

Comment on attachment 546173 [details] [diff] [review]
address additional review comments

I addressed all of your comments here, Nick. I also found a bug in AUS-server.py that caused headerArchitecture not to be set except on Mac, which I "fixed" (for some definition of "fixed"). Also removed a couple unused imports, and added an "assert json" to shut pyflakes up about it being unused.

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 545681 [details] [diff] [review]
a few unit tests

Review of attachment 545681 [details] [diff] [review]:
-----------------------------------------------------------------

Nick, you mentioned on IRC this week that an explanation of the unit tests would be helpful. Hopefully this helps:

::: test_AUS.py
@@ +19,5 @@
> +            return None
> +
> +class TestGetMatchingRules(unittest.TestCase):
> +    def setUp(self):
> +        self.aus = MockAUS3()

The idea here is that we want to test getMatchingRules' logic _only_. Having them fail because the sqlite driver had an issue, or the filesystem filled up, etc. isn't useful, and can be confusing. So, instead of relying on the sqlite driver, sample data that may or may not be isolated to these tests, etc., we use an AUS3 instance with a Mock'ed db.

@@ +30,5 @@
> +            'product': 'Firefox', 'channel': 'release', 'buildTarget': 'Linux_x86-gcc3',
> +            'buildID': '20110707010101', 'locale': 'en-US', 'osVersion': None,
> +            'distribution': None, 'distVersion': None, 'headerArchitecture': None,
> +            'name': 'Firefox-5.0.1-build1', 'version': '5.0.1'
> +        })

This test makes sure that in a situation where the only update paths are not relevant to the incoming query, that we don't actually get any results.

@@ +47,5 @@
> +    def testMatchesWithGlobbing(self):
> +        self._runTestWithMatches([{'mapping': 'Firefox-3.6.19-build1', 'version': '3.6*'}])
> +
> +    def testMatchesNoGlobbing(self):
> +        self._runTestWithMatches([{'mapping': 'Firefox-3.6.19-build1', 'version': '3.6.17'}])

These two test that matching by explicit version works, and also matching by globbing works.

@@ +51,5 @@
> +        self._runTestWithMatches([{'mapping': 'Firefox-3.6.19-build1', 'version': '3.6.17'}])
> +
> +    def testMultipleMatches(self):
> +        self._runTestWithMatches([{'mapping': 'Firefox-3.6.19-build1', 'version': '3.6*', 'priority': 70},
> +                                  {'mapping': None, 'version': '3.6*', 'priority': 100}])

This one ensures that when there are multiple matching rules, all are returned.

@@ +57,5 @@
> +
> +class TestEvaluateRules(unittest.TestCase):
> +    def setUp(self):
> +        self.aus = MockAUS3()
> +        self.aus.getMatchingRules = Mock()

The evaluateRules tests are very similar to above, except that in this scenario, we don't want to be subject to any issues that happen in getMatchingRules, so we mock it instead of the database.

@@ +71,5 @@
> +        self.assertEquals(res, expected)
> +
> +    def testOneMatch(self):
> +        matches = [{'mapping': 'Firefox-3.6.19-build1', 'version': '3.6.17', 'priority': 70}]
> +        self._runTestWithMatches(matches, matches[0])

This tests that in a case where there's a single row matching, that evaluateRules returns it.

@@ +76,5 @@
> +
> +    def testMultipleMatches(self):
> +        matches = [{'mapping': 'Firefox-3.6.19-build1', 'version': '3.6*', 'priority': 70},
> +                   {'mapping': None, 'version': '3.6*', 'priority': 100}]
> +        self._runTestWithMatches(matches, matches[1])

This tests that when multiple rows match, that evaluateRules returns the higher priority one. (When we figure out the full algorithm for multiple matches, we'll probably want some additional tests here.)

These two would probably be more less opaque if the query passed to evaluateRules was 3.6.17 instead of 5.0.1, or even just 'fake' in most spots.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 545681 [details] [diff] [review]
a few unit tests

Thanks for your explanations Ben.

>diff -r 0fbae1e40cce test_AUS.py
>+    def testNoResults(self):
>+        self.aus.db.execute.return_value = FakeCursor([
>+            {'mapping': 'Firefox-3.6.19-build1', 'version': '3.6*'}

This confused me at first because it wasn't obvious why a 3.6.x result would come from an update query for 5.0.1. After looking at the code I see that makes sense, as we hit the db then throw away matches that don't apply for the version & channel before returning.

>+    def _runTestWithMatches(self, matches, expected):
>+    def testOneMatch(self):
>+    def testMultipleMatches(self):

I agree it would be less confusing to not update 5.0.1 to 3.6.x. Either realistic data or completely synthetic would be better - both require wrapping your head around the setup when you first read the tests (staleness vs strangeness) so I don't have any preference right now. Tempting to use real buildID's if you stick with what's there, I found myself scanning the query parameters to try to spot anything that might be important for the test.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Reporting improvements

Makes test-rules.py have output like this:
Testing aus-data-snapshots/3.6.12-3.6.13
PASS   (324 PASS, 0 FAIL, 0.95 sec, 341.1 tests/second)

Testing aus-data-snapshots/fallback-falls-back
PASS   (2 PASS, 0 FAIL)
...

and suppresses the warning from AUS.expandRelease that we were hitting in rule-priorities. Applies on top of the patch here.

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 546493 [details] [diff] [review]
Reporting improvements

I landed this patch, plus your reporting improvements on the default branch of my user repository. http://hg.mozilla.org/users/bhearsum_mozilla.com/aus3-proto/rev/d68f1a353177

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Merged to 'upstream' with the 3 small changes that were also on ben's default:
 http://hg.mozilla.org/users/nthomas_mozilla.com/aus3-proto/pushloghtml?fromchange=be442ee5ecda&tochange=3d9a6ebe3182

Comment 25

[[e-mail] (:bhearsum)]

I'm going to call this bug FIXED. We've got unittests for the db layer already, and I'm starting to think that the rule tests serve us much better than the mock-based ones from the first patch here.

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 546493 [details] [diff] [review]
Reporting improvements

This was checked in a long time ago.

Comment 27

[[e-mail] (:bhearsum)]

Comment on attachment 546173 [details] [diff] [review]
address additional review comments

This was checked in a long time ago.