2045122 - Update extension tests to handle new Nova style rules

Status: RESOLVED FIXED

(WebExtensions :: Untriaged, task, P2)

Description

[Stephen Thompson [:sthompson]]

:mossop ran a try run with browser.nova.enabled to see whether there were any test issues hiding from us since Nova is off by default.

I think roughly 13 of the 81 test failures are coming from extensions tests that depend on CSS rules which change when Nova is enabled. The full list of failures across the codebase is at https://tests.firefox.dev/try.html?rev=813998049ff1835af3a60def7853df3d8be83924, with a filtered list of specific /extensions/ tests that fail asserts against style values here:

• toolkit/components/extensions/test/browser/browser_ext_themes_alpha_accentcolor.js
• toolkit/components/extensions/test/browser/browser_ext_themes_autocomplete_popup.js
• toolkit/components/extensions/test/browser/browser_ext_themes_chromeparity.js
• toolkit/components/extensions/test/browser/browser_ext_themes_dynamic_updates.js
• toolkit/components/extensions/test/browser/browser_ext_themes_lwtsupport.js
• toolkit/components/extensions/test/browser/browser_ext_themes_sanitization.js
• toolkit/components/extensions/test/browser/browser_ext_themes_separators.js
• toolkit/components/extensions/test/browser/browser_ext_themes_sidebars.js
• toolkit/components/extensions/test/browser/browser_ext_themes_toolbar_fields.js
• toolkit/components/extensions/test/browser/browser_ext_themes_toolbar_fields_focus.js
• toolkit/components/extensions/test/browser/browser_ext_themes_toolbarbutton_icons.js
• browser/components/extensions/test/browser/browser_ext_management.js
• browser/components/extensions/test/browser/browser_ext_popup_corners.js

We'd like to get these Nova-specific test failures addressed before the planned 155 Nightly pref flip to make Nova enabled by default. We don't want to have to burden the sheriffs with permafails that we know about.

Comment 1

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: Bug 2045122 - Adapt browser_ext_themes_autocomplete_popup.js to pass with and without Nova styles enabled. r=willdurand!

Comment 2

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: Bug 2045122 - Adapt browser_ext_themes_dynamic_updates.js to pass with and without Nova styles enabled. r=willdurand!

Comment 3

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: Bug 2045122 - Adapt browser_ext_themes_separators.js to pass with and without Nova styles enabled. r=willdurand!

Comment 4

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: WIP: Bug 2045122 - Fix missing sidebar custom theme outline color with Nova enabled.

Comment 5

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: WIP: Bug 2045122 - Missing urlbar custom theme border and outline focused color with Nova enabled.

Comment 6

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: WIP: Bug 2045122 - extension-popup-panel body border-radius should match panel border-radius when Nova is enabled.

Comment 7

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: Bug 2045122 - Adapt browser_ext_themes_multiple_backgrounds.js to pass with Nova enabled and disabled. r=willdurand!

Comment 8

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: WIP: Bug 2045122 - Run toolkit-level webextensions desktop frontend tests with and without Nova styles enabled.

Similarly to how Bug 2030183 added mochitest browser manifests to run urlbar tests with and without
Nova styles enabled, this patch adds browserNova.toml files to each webextensions desktop frontend
browser test subdirectories.

Each one sets browser.nova.enabled and imports the main browser.toml in that
subdirectory.

A new webextensions-frontend-nova tag has been also added to make it easier to run tests with Nova enabled.
Examples:

./mach mochitest -f browser --tag webextensions-frontend-nova toolkit/components/extensions/test/browser
./mach mochitest -f browser --tag webextensions-frontend-nova toolkit/components/extensions/test/browser/browser_ext_themes_autocomplete_popup.js

NOTE: we should consider doing the same also for browser-level webextensions mochitest-browser tests, but the browser-level
browser.toml manifest takes too long and so it may be better to do that on top of Bug 2026593 and cherry-pick the toml
manifests that are actually providing relevant test coverage.

Comment 9

[Luca Greco [:rpl] [:luca] [:lgreco]]

I haven't been able to reproduce locally failures from a few on the tests mentioned in comment 0, in particular the following ones:

• toolkit/components/extensions/test/browser/browser_ext_themes_alpha_accentcolor.js
• toolkit/components/extensions/test/browser/browser_ext_themes_chromeparity.js
• toolkit/components/extensions/test/browser/browser_ext_themes_lwtsupport.js
• toolkit/components/extensions/test/browser/browser_ext_themes_sanitization.js
• toolkit/components/extensions/test/browser/browser_ext_themes_toolbarbutton_icons.js
• browser/components/extensions/test/browser/browser_ext_management.js

But as a first pass I only investigate the ones that are being hit on macOS, I'll give it another look on my Linux and Windows systems as part of a second pass, but in the meantime I've also pushed the stack to try along with a browserNova.toml manifest that would re-run the toolkit-level webextensions mochitest-browser tests (which are largely "custom themes"-related) on the CI and so I'll also look at what my push to try may still be hitting.

I have also looked into another failure that wasn't listed in comment 0 (browser_ext_themes_multiple_backgrounds.js, which I guess may have been started as a side-effect of Bug 2036647 changes or from changes to the default theme built on top of that given that it was related to the gradient background-image used when Nova and the default theme are both in use).

A few test failures looked like hit due to actual gaps/differences in the theme properties use when Nova is enabled (in particular the ones attached from Bug 2045122 comment 4, Bug 2045122 comment 5 and Bug 2045122 comment 6) which will require a design discussion/decision.

Comment 10

[Phabricator Automation]

Comment on attachment 9595237 [details]
WIP: Bug 2045122 - Fix missing sidebar custom theme outline color with Nova enabled.

Revision D305351 was moved to bug 2054481. Setting attachment 9595237 [details] to obsolete.

Comment 11

[Phabricator Automation]

Comment on attachment 9595238 [details]
WIP: Bug 2045122 - Missing urlbar custom theme border and outline focused color with Nova enabled.

Revision D305352 was moved to bug 2054492. Setting attachment 9595238 [details] to obsolete.

Comment 12

[Phabricator Automation]

Comment on attachment 9595239 [details]
WIP: Bug 2045122 - extension-popup-panel body border-radius should match panel border-radius when Nova is enabled.

Revision D305353 was moved to bug 2054499. Setting attachment 9595239 [details] to obsolete.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/03dfef0d2ac2
https://hg.mozilla.org/integration/autoland/rev/087cbe98f45f
Adapt browser_ext_themes_autocomplete_popup.js to pass with and without Nova styles enabled. r=willdurand
https://github.com/mozilla-firefox/firefox/commit/a4ed463b60b9
https://hg.mozilla.org/integration/autoland/rev/fa5c177349f0
Adapt browser_ext_themes_dynamic_updates.js to pass with and without Nova styles enabled. r=willdurand
https://github.com/mozilla-firefox/firefox/commit/4eff807e9d40
https://hg.mozilla.org/integration/autoland/rev/acf1d03b5435
Adapt browser_ext_themes_separators.js to pass with and without Nova styles enabled. r=willdurand
https://github.com/mozilla-firefox/firefox/commit/a7bc76ef11a4
https://hg.mozilla.org/integration/autoland/rev/089b0a7ad13a
Adapt browser_ext_themes_multiple_backgrounds.js to pass with Nova enabled and disabled. r=willdurand

Comment 14

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/087cbe98f45f
https://hg.mozilla.org/mozilla-central/rev/fa5c177349f0
https://hg.mozilla.org/mozilla-central/rev/acf1d03b5435
https://hg.mozilla.org/mozilla-central/rev/089b0a7ad13a