Failures in mobile compatibility validation aren't failures

RESOLVED FIXED in 6.2.9

Status

defect
P2
normal
RESOLVED FIXED
8 years ago
4 years ago

People

(Reporter: jorgev, Assigned: kumar)

Tracking

unspecified
6.2.9

Details

()

Looking at the failure log for the mobile compatibility validation results, none of the 3 listed failures have any errors in the validation results.

Here they are:
https://addons.mozilla.org/en-US/developers/addon/autopager/validation-result/84713
https://addons.mozilla.org/en-US/developers/addon/autopager-lite/validation-result/84716
https://addons.mozilla.org/en-US/developers/addon/1tudien-47255/validation-result/84717

We need to figure out what's going on here, and see if there are also related errors in the success log.
You can see what's happening by checking compatibility for autopager against Mobile 9.* here https://addons-dev.allizom.org/en-US/developers/addon/check-compatibility

It fails the 'navigator.language may not behave as expected' check.  However, that only applies to 5.* so I don't know why it's running.  I am investigating.

Do you know if autopager is expected to pass/fail a Mobile 9.* check?
(In reply to Kumar McMillan [:kumar] from comment #1)
> It fails the 'navigator.language may not behave as expected' check. 
> However, that only applies to 5.* so I don't know why it's running.  I am
> investigating.

I don't recall if there was a bug filed about this, but we had a similar problem on desktop Firefox and it was fixed. The compatibility validation should only fail for the tests of the versions being tested. Since the bump was defined from 8 to 9, only the validations for 9 should apply.
Looking at the code, the bug is in only a certain group of tests:
- navigator.language
- unsupported interfaces
- app.update.timer
- data URLs
- nsINavHistoryObserver
... a couple more

These tests are always run, regardless of what app/version you are validating for.  This is a bug.  I just validated autopager against Firefox 9.0 and I see the same problem as with Mobile. https://addons-dev.allizom.org/en-US/developers/addon/check-compatibility  The test cases are filtered out by the UI since they are tagged for different versions which is why they don't show up.
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.