toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_*.js test failures on Android
Categories
(Toolkit :: Blocklist Implementation, defect, P1)
Tracking
()
People
(Reporter: robwu, Assigned: robwu)
References
(Regression)
Details
(Keywords: regression)
Attachments
(3 files)
When a RemoteSettings-based blocklist was introduced, all Android tests were skipped in bug 1257565.
In bug 1706391, I re-enabled most tests to maintain a degree of test coverage, bu the patch stack got backed out for failures unrelated to my changes.
By the looks of it, it seems that the gfx blocklist isn't working on Android (or the tests are broken). I'm relanding the patches with the tests disabled again as before, but it's probably worth investigating why the gfx blocklist tests are failing on Android.
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_DriverNew.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_DriverNew.js | checkBlacklist - [checkBlacklist : 66] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Equal_OK.js | checkBlacklist - [checkBlacklist : 53] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_GTE_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_GTE_OK.js | checkBlacklist - [checkBlacklist : 53] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_No_Comparison.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_No_Comparison.js | checkBlacklist - [checkBlacklist : 49] 1 == 4
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_OK.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_OK.js | checkBlacklist - [checkBlacklist : 54] 1 == 3
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Version.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_Version.js | checkBlocklist - [checkBlocklist : 60] 1 == 3
Return code: 1
|
Assignee | |
Comment 1•3 years ago
|
||
Oddly, I do recall running all tests (even with --verify) locally, in the x64 Android emulator (opt build, artifacts) without failures.
On try, the xpcshell tests on opt builds haven't failed.
So it might be a debug-only failure...
|
|
Assignee | |
Comment 2•3 years ago
|
||
It's a debug-only failure because all test_gfxBlacklist* tests skip when nsIGfxInfoDebug is unavailable. Consequently the tests are only checking anything at all in debug builds:
- "We can't do anything if we can't spoof the stuff we need.", e.g. at https://searchfox.org/mozilla-central/rev/c1ec9ecbbc7eac698923ffd18c8594aa3e2e9da0/toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_gfxBlacklist_GTE_OK.js#13-17
#ifdef DEBUG- https://searchfox.org/mozilla-central/rev/c1ec9ecbbc7eac698923ffd18c8594aa3e2e9da0/widget/GfxInfoBase.h#35-38
The overall skip-if = os == "android" predates the migration to the RemoteSettings blocklist, e.g. test_gfxBlacklist_GTE_OK.js (the XML blocklist variant) (along with all tests in the directory) were disabled on Android in https://searchfox.org/mozilla-central/diff/d292ee73f107c441f68ae61730d8aed7c794dc14/toolkit/mozapps/extensions/test/xpcshell/xpcshell-shared.ini (7 oct 2014). The introduction of the skip-if did not mean that Android tests were running before, the mentioned xpcshell_android.ini file was introduced in bug 823703 and expanded later (without most of the tests from mozapps) (22 dec 2012). The test_gfxBlacklist_GTE_OK.js test was introduced in https://hg.mozilla.org/mozilla-central/rev/80c4d7317c29 (20 jan 2011).
I think that gfx blocklisting is entirely broken on Android because of the following changes:
- 2 nov 2012: https://hg.mozilla.org/mozilla-central/rev/f710f9f6f7e#l3.63 changed the
osfromDRIVER_OS_ANDROID(i.e.OperatingSystem::Androidthese days) tomOS, which was uninitialized, i.e. equivalent to "unknown" (these days:OperatingSystem::Unknown)- The change landed without unit tests for Android: https://bugzilla.mozilla.org/show_bug.cgi?id=806369#c25
- 8 jun 2016: https://hg.mozilla.org/mozilla-central/rev/af65533093de changed the logic, to require the OS of the blacklist entry to match. Except for macOS and Windows, where "unknown" is treated as matching.
- .... but since Android unconditionally specified "unknown", the blacklist isn't working any more on Android.
- this went undetected because unit tests for the blocklist tests were never run on Android CI.
After fixing that most of the affected tests started to pass. There are two more test failures, I'll look into that to see if it's something that I can fix.
|
||
Updated•3 years ago
|
|
|
Assignee | |
Comment 3•3 years ago
|
||
The remote gfx blocklist for Android was broken since 2016 by
https://hg.mozilla.org/mozilla-central/rev/af65533093de as explained at
https://bugzilla.mozilla.org/show_bug.cgi?id=1714673#c2
This patch fixes the issue and re-enables the disabled tests.
|
|
Assignee | |
Comment 4•3 years ago
|
||
test_gfxBlacklist_Equal_DriverNew.js fails on Android because the
meaning of "All" was changed by bug 1274635 to mean "All Windows"
instead of "All OSs", while the test relied on mock data in
test_gfxBlacklist.json that contains entries for "os": "All".
To fix the failures, add entries to test_gfxBlacklist.json based on
these existing "os": "All" entries, with driver version ranges chosen
to as closely match the original entry (greater/lower/equal compared to
6 (from gfxInfo.spoofDriverVersion("6")) in the test file).
|
|
Assignee | |
Comment 5•3 years ago
•
|
||
The values for spoofVendorID/spoofDeviceID should match entries in
test_gfxBlacklist_AllOS.json. These entries were added in
https://hg.mozilla.org/mozilla-central/rev/21263fbfdcc2 (bug 1274152) but
inadvertently not updated (because the test was disabled).
Fixing that enables the test to use the expected entries.
Several test expectations were no longer correct for Android and have
been updated as well.
|
||
Updated•3 years ago
|
Pushed by rob@robwu.nl: https://hg.mozilla.org/integration/autoland/rev/acf24b0a8c0c Fix gfx blocklist support on Android r=jrmuizel https://hg.mozilla.org/integration/autoland/rev/65846ff92d38 Fix and re-enable test_gfxBlacklist_Equal_DriverNew.js on Android r=jrmuizel https://hg.mozilla.org/integration/autoland/rev/51a6ac083fbe Fix and re-enable test_gfxBlacklist_Version.js on Android r=jrmuizel
|
||
Updated•3 years ago
|
|
||
Comment 8•3 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/acf24b0a8c0c
https://hg.mozilla.org/mozilla-central/rev/65846ff92d38
https://hg.mozilla.org/mozilla-central/rev/51a6ac083fbe
|
||
Updated•3 years ago
|
|
||
Comment 9•3 years ago
|
||
The patch landed in nightly and beta is affected.
:robwu, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.
For more information, please visit auto_nag documentation.
|
|
Assignee | |
Updated•3 years ago
|
Description
•