Closed Bug 1714673 Opened 3 years ago Closed 3 years ago

toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_*.js test failures on Android


(Toolkit :: Blocklist Implementation, defect, P1)




91 Branch
Tracking Status
firefox-esr78 --- wontfix
firefox89 --- wontfix
firefox90 --- wontfix
firefox91 --- fixed


(Reporter: robwu, Assigned: robwu)




(Keywords: regression)


(3 files)

When a RemoteSettings-based blocklist was introduced, all Android tests were skipped in bug 1257565.

In bug 1706391, I re-enabled most tests to maintain a degree of test coverage, bu the patch stack got backed out for failures unrelated to my changes.
By the looks of it, it seems that the gfx blocklist isn't working on Android (or the tests are broken). I'm relanding the patches with the tests disabled again as before, but it's probably worth investigating why the gfx blocklist tests are failing on Android.

From :

TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_DriverNew.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_DriverNew.js | checkBlacklist - [checkBlacklist : 66] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_OK.js | checkBlacklist - [checkBlacklist : 53] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_GTE_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_GTE_OK.js | checkBlacklist - [checkBlacklist : 53] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_No_Comparison.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_No_Comparison.js | checkBlacklist - [checkBlacklist : 49] 1 == 4
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_OK.js | checkBlacklist - [checkBlacklist : 54] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Version.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Version.js | checkBlocklist - [checkBlocklist : 60] 1 == 3
Return code: 1 

Oddly, I do recall running all tests (even with --verify) locally, in the x64 Android emulator (opt build, artifacts) without failures.

On try, the xpcshell tests on opt builds haven't failed.

So it might be a debug-only failure...

It's a debug-only failure because all test_gfxBlacklist* tests skip when nsIGfxInfoDebug is unavailable. Consequently the tests are only checking anything at all in debug builds:

The overall skip-if = os == "android" predates the migration to the RemoteSettings blocklist, e.g. test_gfxBlacklist_GTE_OK.js (the XML blocklist variant) (along with all tests in the directory) were disabled on Android in (7 oct 2014). The introduction of the skip-if did not mean that Android tests were running before, the mentioned xpcshell_android.ini file was introduced in bug 823703 and expanded later (without most of the tests from mozapps) (22 dec 2012). The test_gfxBlacklist_GTE_OK.js test was introduced in (20 jan 2011).

I think that gfx blocklisting is entirely broken on Android because of the following changes:

After fixing that most of the affected tests started to pass. There are two more test failures, I'll look into that to see if it's something that I can fix.

Assignee: nobody → rob
Keywords: regression
OS: Unspecified → Android
Regressed by: 1274635
See Also: → 806369
Has Regression Range: --- → yes

The remote gfx blocklist for Android was broken since 2016 by as explained at

This patch fixes the issue and re-enables the disabled tests.

test_gfxBlacklist_Equal_DriverNew.js fails on Android because the
meaning of "All" was changed by bug 1274635 to mean "All Windows"
instead of "All OSs", while the test relied on mock data in
test_gfxBlacklist.json that contains entries for "os": "All".

To fix the failures, add entries to test_gfxBlacklist.json based on
these existing "os": "All" entries, with driver version ranges chosen
to as closely match the original entry (greater/lower/equal compared to
6 (from gfxInfo.spoofDriverVersion("6")) in the test file).

The values for spoofVendorID/spoofDeviceID should match entries in
test_gfxBlacklist_AllOS.json. These entries were added in (bug 1274152) but
inadvertently not updated (because the test was disabled).
Fixing that enables the test to use the expected entries.

Several test expectations were no longer correct for Android and have
been updated as well.

Pushed by
Fix gfx blocklist support on Android r=jrmuizel
Fix and re-enable test_gfxBlacklist_Equal_DriverNew.js on Android r=jrmuizel
Fix and re-enable test_gfxBlacklist_Version.js on Android r=jrmuizel
See Also: → 1715788
Severity: -- → N/A
Priority: -- → P1
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 91 Branch

The patch landed in nightly and beta is affected.
:robwu, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Flags: needinfo?(rob)
Flags: needinfo?(rob)
You need to log in before you can comment on or make changes to this bug.