1974933 - Allow tests to opt out of compare-preferences

Status: RESOLVED INVALID

(Testing :: General, enhancement)

Description

[Julian Descottes [:jdescottes]]

See motivations at https://bugzilla.mozilla.org/show_bug.cgi?id=1816549#c9

Some tests are tricky to fix for --compare-preferences, and I think we should add some API to allow to opt out from this check for a given test so that test-verify can still be used with it.

Comment 1

[Julian Descottes [:jdescottes]]

Attachment: Bug 1974933 - Allow to disable preferences comparison in browser mochitests

Comment 2

[Rob Wu [:robwu]]

There is still discussion ongoing in bug 1816549 about the desired behavior. Pending the outcome of that discussion, the patch here is on hold.

The current proposal in this bug is to allow individual tests to opt out of --compare-preferences behavior, motivated by the fact that TV (--verify) test jobs are currently failing for many DevTools tests. If you are interested in resolving TV failures specifically (with less concern about --compare-preferences), then I would recommend different approaches.

--compare-preferences is meant to detect unexpected preference changes by individual test files, because these can unknowingly affect the test outcome of the next test file in the run. For general preferences unrelated to the specific test, an exception should be added to the ignorePrefs.json file If a test expects a pref to change, it should clean it up itself. If multiple tests within the same test manifest (in a directory) are expected to flip many preferences that are knowingly not affecting other tests in the directory, a broad solution is to reset all affected preferences, e.g. as you did in bug 1974700.

Comment 3

[Julian Descottes [:jdescottes]]

Agreed, I will keep the patch in changes planned until we know what is the decision with TV and compare-preferences.

or general preferences unrelated to the specific test, an exception should be added to the ignorePrefs.json file If a test expects a pref to change, it should clean it up itself.

I considered the options and did test cleanup as much as possible. But some devtools tests will indirectly trigger preferences completely unrelated to devtools by using the browser (opening a page, triggering a download etc...). I don't want to add them to ignorePrefs.json because I don't really know much about those preferences and folks maintaining them might disagree. I could just have an ignore list for each of those tests and cleanup prefs manually at the end but that's fragile. I have already seen yesterday new bugs filed because of preferences introduced to implement changes unrelated to devtools.

motivated by the fact that TV (--verify) test jobs are currently failing for many DevTools tests.

Quick clarification, my motivation is not to address the bulk of DevTools failures. I already landed patches which fix 99% of the issues in /devtools. But TV is an important tool for me when investigating and fixing intermittent failures, and so the few tests which fall in the category described above, I wanted a solution to be able to keep using test-verify.

Comment 4

[Julian Descottes [:jdescottes]]

Since we're resetting preferences for now I don't plan to keep working on this.