Closed Bug 1714673 Opened 8 months ago Closed 8 months ago

toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_*.js test failures on Android

Categories

(Toolkit :: Blocklist Implementation, defect, P1)

Unspecified
Android
defect

Tracking

()

RESOLVED FIXED
91 Branch
Tracking Status
firefox-esr78 --- wontfix
firefox89 --- wontfix
firefox90 --- wontfix
firefox91 --- fixed

People

(Reporter: robwu, Assigned: robwu)

References

(Regression)

Details

(Keywords: regression)

Attachments

(3 files)

When a RemoteSettings-based blocklist was introduced, all Android tests were skipped in bug 1257565.

In bug 1706391, I re-enabled most tests to maintain a degree of test coverage, bu the patch stack got backed out for failures unrelated to my changes.
By the looks of it, it seems that the gfx blocklist isn't working on Android (or the tests are broken). I'm relanding the patches with the tests disabled again as before, but it's probably worth investigating why the gfx blocklist tests are failing on Android.

From https://treeherder.mozilla.org/jobs?repo=autoland&revision=ccd2f9b9c513ad8dca7cfbd3c6dc7fc4d200f7a0&selectedTaskRun=dMYxlgAuSFeR2a3WEaWrkA.0 :

TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_DriverNew.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_DriverNew.js | checkBlacklist - [checkBlacklist : 66] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_OK.js | checkBlacklist - [checkBlacklist : 53] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_GTE_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_GTE_OK.js | checkBlacklist - [checkBlacklist : 53] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_No_Comparison.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_No_Comparison.js | checkBlacklist - [checkBlacklist : 49] 1 == 4
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_OK.js | checkBlacklist - [checkBlacklist : 54] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Version.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Version.js | checkBlocklist - [checkBlocklist : 60] 1 == 3
Return code: 1 

Oddly, I do recall running all tests (even with --verify) locally, in the x64 Android emulator (opt build, artifacts) without failures.

On try, the xpcshell tests on opt builds haven't failed.

So it might be a debug-only failure...

It's a debug-only failure because all test_gfxBlacklist* tests skip when nsIGfxInfoDebug is unavailable. Consequently the tests are only checking anything at all in debug builds:

The overall skip-if = os == "android" predates the migration to the RemoteSettings blocklist, e.g. test_gfxBlacklist_GTE_OK.js (the XML blocklist variant) (along with all tests in the directory) were disabled on Android in https://searchfox.org/mozilla-central/diff/d292ee73f107c441f68ae61730d8aed7c794dc14/toolkit/mozapps/extensions/test/xpcshell/xpcshell-shared.ini (7 oct 2014). The introduction of the skip-if did not mean that Android tests were running before, the mentioned xpcshell_android.ini file was introduced in bug 823703 and expanded later (without most of the tests from mozapps) (22 dec 2012). The test_gfxBlacklist_GTE_OK.js test was introduced in https://hg.mozilla.org/mozilla-central/rev/80c4d7317c29 (20 jan 2011).

I think that gfx blocklisting is entirely broken on Android because of the following changes:

After fixing that most of the affected tests started to pass. There are two more test failures, I'll look into that to see if it's something that I can fix.

Assignee: nobody → rob
Keywords: regression
OS: Unspecified → Android
Regressed by: 1274635
See Also: → 806369
Has Regression Range: --- → yes

The remote gfx blocklist for Android was broken since 2016 by
https://hg.mozilla.org/mozilla-central/rev/af65533093de as explained at
https://bugzilla.mozilla.org/show_bug.cgi?id=1714673#c2

This patch fixes the issue and re-enables the disabled tests.

test_gfxBlacklist_Equal_DriverNew.js fails on Android because the
meaning of "All" was changed by bug 1274635 to mean "All Windows"
instead of "All OSs", while the test relied on mock data in
test_gfxBlacklist.json that contains entries for "os": "All".

To fix the failures, add entries to test_gfxBlacklist.json based on
these existing "os": "All" entries, with driver version ranges chosen
to as closely match the original entry (greater/lower/equal compared to
6 (from gfxInfo.spoofDriverVersion("6")) in the test file).

The values for spoofVendorID/spoofDeviceID should match entries in
test_gfxBlacklist_AllOS.json. These entries were added in
https://hg.mozilla.org/mozilla-central/rev/21263fbfdcc2 (bug 1274152) but
inadvertently not updated (because the test was disabled).
Fixing that enables the test to use the expected entries.

Several test expectations were no longer correct for Android and have
been updated as well.

Status: NEW → ASSIGNED
Pushed by rob@robwu.nl:
https://hg.mozilla.org/integration/autoland/rev/acf24b0a8c0c
Fix gfx blocklist support on Android r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/65846ff92d38
Fix and re-enable test_gfxBlacklist_Equal_DriverNew.js on Android r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/51a6ac083fbe
Fix and re-enable test_gfxBlacklist_Version.js on Android r=jrmuizel
See Also: → 1715788
Severity: -- → N/A
Priority: -- → P1
Status: ASSIGNED → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → 91 Branch

The patch landed in nightly and beta is affected.
:robwu, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Flags: needinfo?(rob)
Flags: needinfo?(rob)
You need to log in before you can comment on or make changes to this bug.