696511 - Failures in mobile compatibility validation aren't failures

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Compatibility Tools, defect, P2)

Description

[Jorge Villalobos [:jorgev] (he/him)]

Looking at the failure log for the mobile compatibility validation results, none of the 3 listed failures have any errors in the validation results.

Here they are:
https://addons.mozilla.org/en-US/developers/addon/autopager/validation-result/84713
https://addons.mozilla.org/en-US/developers/addon/autopager-lite/validation-result/84716
https://addons.mozilla.org/en-US/developers/addon/1tudien-47255/validation-result/84717

We need to figure out what's going on here, and see if there are also related errors in the success log.

Comment 1

[Kumar McMillan [:kumar]]

You can see what's happening by checking compatibility for autopager against Mobile 9.* here https://addons-dev.allizom.org/en-US/developers/addon/check-compatibility

It fails the 'navigator.language may not behave as expected' check.  However, that only applies to 5.* so I don't know why it's running.  I am investigating.

Do you know if autopager is expected to pass/fail a Mobile 9.* check?

Comment 2

[Jorge Villalobos [:jorgev] (he/him)]

(In reply to Kumar McMillan [:kumar] from comment #1)
> It fails the 'navigator.language may not behave as expected' check. 
> However, that only applies to 5.* so I don't know why it's running.  I am
> investigating.

I don't recall if there was a bug filed about this, but we had a similar problem on desktop Firefox and it was fixed. The compatibility validation should only fail for the tests of the versions being tested. Since the bump was defined from 8 to 9, only the validations for 9 should apply.

Comment 3

[Kumar McMillan [:kumar]]

Looking at the code, the bug is in only a certain group of tests:
- navigator.language
- unsupported interfaces
- app.update.timer
- data URLs
- nsINavHistoryObserver
... a couple more

These tests are always run, regardless of what app/version you are validating for.  This is a bug.  I just validated autopager against Firefox 9.0 and I see the same problem as with Mobile. https://addons-dev.allizom.org/en-US/developers/addon/check-compatibility  The test cases are filtered out by the UI since they are tagged for different versions which is why they don't show up.

Comment 4

[Kumar McMillan [:kumar]]

Fixed in the validator: https://github.com/mozilla/amo-validator/commit/0905d3233915eac4c9bcc22b4e52c89bdc3df32d
Added a functional regression test: https://github.com/mozilla/test-amo-validator/commit/adf2931be0afbfbafd839158c01fa8a89e40a66e
Pushed to Zamboni: https://github.com/mozilla/zamboni/commit/0e0dd58