650380 - Build a compatibility tier and add a test for Fx5, even if it's a no-op

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Developer Pages, defect, P1)

Description

[Wil Clouser [:clouserw]]

From bug 649863 comment 6:  We need a compatibility tier where we can specify groups of tests as checking compatibility between versions.  This bug is for creating that tier and its supporting unit tests.

Comment 1

[Matt Basta [:basta]]

I built this as half of the compatibility tier/flag support. It should fulfill the needs of this bug:

https://github.com/mattbasta/amo-validator/commit/d189572cf45139374f098c5cc6fc7339e31058bf

In short, it allows a group of tests to specify app GUIDs and versions that it can be run for. For instance, a test could be restricted to only Firefox 5.0.0 (or any other specific version). It also supports using ranges of versions, so an add-on supporting Firefox 3.0 through 4.0 would be validated by a test that's set to run for 3.6.

Adding line-item tests is fairly straightforward and should be done when we get there. Wil, if you could check my reasoning, I'll create a pull and get this reviewed.

Comment 2

[Wil Clouser [:clouserw]]

It seems fine but I can't actually test it today as my mondays are all meetings. :-/  Kumar is going to check it out though.

Comment 3

[Kumar McMillan [:kumar]]

The patch looks good to me

Comment 4

[Matt Basta [:basta]]

Awesome, thanks

https://github.com/mozilla/amo-validator/pull/22

Comment 5

[Kumar McMillan [:kumar]]

I had an after thought.  I think we need some indication that a test is part of a specific group in the JSON itself.  That is, to fulfill this screen in bug 649863, I will need to sort the results into the specific compatibility groups.  

Maybe add something like for_appversions to the message dict like this?

{"validation": {
    "messages": [{
        // ...
        "message": "This function call is not supported",
        "tier": 1,
        "for_appversions": {
            '{ec8030f7-c20a-464f-9b0e-13a3a9e97384}': ['5.0.0']
        },
        "type": "error"
    }]
}}

Comment 6

[Kumar McMillan [:kumar]]

reopening so I don't forget about this...

Comment 7

[Matt Basta [:basta]]

Kumar: I added the "for_appversions" item to messages. It should be null by default unless version information was provided.

for_appversions is now a parameter on the message functions on error bundles:

err.error(
    err_id=("test_error", ),
    error="A test error",
    description="Error description",
    for_appversions={"guid": ["version1", "version2"]})

The commit is here:

https://github.com/mattbasta/amo-validator/commit/ea88eb9e5c63f65d92d88e2adf64f1638485401f

Comment 8

[Kumar McMillan [:kumar]]

thanks Matt