1816549 - [browser-chrome] add support to log preferences that are not reset after a test finishes

Status: RESOLVED FIXED

(Testing :: Mochitest, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

just like bug 1783248 will do for mochitest-plain, this will need to modify the browser-chrome harness.

Comment 1

[Rob Wu [:robwu]]

I tried using ./mach test with --compare-preferences, and was suprised to see that it did not report failures, even for tests that persist pref changes past the end of a test execution. Turns out that it was not implemented when support for --compare-preferences landed in bug 1783248. I'm adding a patch with an implementation.

Although the implementation in bug 1783248 resets preferences between each test, I am not going to do that here, because that could potentially affect the behavior of existing tests (e.g. if one test file depends on an earlier pref change, against best practices).

Comment 2

[Rob Wu [:robwu]]

Attachment: Bug 1816549 - Support --compare-preferences with browser chrome tests

The --compare-preferences flag to mach test (mach mochitest) is
recognized for browser tests, but its implementation was missing.

This patch implements --compare-preferences for browser chrome tests.

Example:

$ ./mach test toolkit/components/antitracking/test/browser/browser_storageAccess_userActivation.js --compare-preferences

ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/toolkit/components/antitracking/test/browser/browser_storageAccess_userActivation.js | changed preference: dom.user_activation.transient.timeout

The above test output shows that the after the end of the test, a pref
was permanently modified, because it uses Services.prefs.setIntPref to
change a preference instead of SpecialPowers.pushPrefEnv, at:
https://searchfox.org/mozilla-central/rev/c25dbe453ff9ca10f2c6bdfb873893c515a29826/toolkit/components/antitracking/test/browser/browser_storageAccess_userActivation.js#83-86

Comment 3

[:Gijs (he/him)]

How badly do things fail if we make this the default behaviour instead of an opt-in flag - could we do a trypush and find out? (I'm not sure I understand why this was originally added as an opt-in flag for mochitest-plain...)

Comment 4

[Rob Wu [:robwu]]

(In reply to :Gijs (he/him) from comment #3)

How badly do things fail if we make this the default behaviour instead of an opt-in flag - could we do a trypush and find out? (I'm not sure I understand why this was originally added as an opt-in flag for mochitest-plain...)

Try push: ./mach try fuzzy -q "'mochitest" results in 56446 failures (https://treeherder.mozilla.org/jobs?repo=try&revision=3cb255849869c5f91d905d393eb99fb728520715).

Forcing --compare-preferences to true is therefore not feasible.
The plain mochitest version of the logic (from bug 1783248) unconditionally resets the preferences to the baseline at the end of the test, which I intentionally omitted from my patch to minimize side effects and test regressions.

If we're confident that resetting prefs between each browser test is the right thing to do, we can give that a try in a follow-up, by unconditionally resetting the prefs between each test file execution. But even before triggering the try push I already imagined that browser chrome tests are more likely triggering state changes that affect browser state (besides prefs, UI, customizable state, etc), which are less likely with plain mochitests, which is why I did not implement resetting of prefs between test file runs. It might be more feasible to improve the test quality by opting in to stricter compare-preferences checks per test manifest.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/c0da75d67bc4
https://hg.mozilla.org/integration/autoland/rev/b6ee79997f38
Support --compare-preferences with browser chrome tests r=jmaher

Comment 6

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/b6ee79997f38

Comment 7

[Nicolas Chevobbe [:nchevobbe]]

Rob, does this mean that pushes that are adding/updating tests would be backed out if some prefs are not reset (since usually any patch adding/modifying test will trigger a TV job, which, if I'm correct, sets the --compare-preferences flag) ?
I'm asking because DevTools is storing quite a few things in prefs that we don't clean up, probably in all our tests (for example devtools.everOpened).
It's already very visible/noisy if you're running a test locally with --verify.

Comment 8

[Rob Wu [:robwu]]

(In reply to Nicolas Chevobbe [:nchevobbe] from comment #7)

Rob, does this mean that pushes that are adding/updating tests would be backed out if some prefs are not reset (since usually any patch adding/modifying test will trigger a TV job, which, if I'm correct, sets the --compare-preferences flag) ?

TV jobs are tier2 and do not result in backout.

I'm asking because DevTools is storing quite a few things in prefs that we don't clean up, probably in all our tests (for example devtools.everOpened).
It's already very visible/noisy if you're running a test locally with --verify.

If this is really that prominent across a large number of devtools tests, you could consider adding a setup task + cleanup handler in a head file that resets the relevant preferences. Another option (if a pref value is not expected to affect test outcomes in any way) is to add a pref to the ignorePrefs.json file. See linked bugs at bug 1783248 for examples of reported (TV) failures and the follow-up.

Comment 9

[Julian Descottes [:jdescottes]]

(In reply to Rob Wu [:robwu] from comment #4)

(In reply to :Gijs (he/him) from comment #3)

How badly do things fail if we make this the default behaviour instead of an opt-in flag - could we do a trypush and find out? (I'm not sure I understand why this was originally added as an opt-in flag for mochitest-plain...)

Try push: ./mach try fuzzy -q "'mochitest" results in 56446 failures (https://treeherder.mozilla.org/jobs?repo=try&revision=3cb255849869c5f91d905d393eb99fb728520715).

Sorry to push back, but should we keep this on by default for test-verify?

I am trying to green up devtools tests in Bug 1974700. I'm close but there are a bunch of tests where the preferences are outside of devtools' control, and more importantly where the list of changed preferences is not stable. For instance, one test opens about:preferences which seems to flip some services.sync preferences, but not always the same ones.

My pitch is that I'm relying on test-verify to investigate intermittent failures. I don't know how many individual tests are affected by the 50000+ failures from the try push mentioned above but that means test-verify can no longer be used for any of those unless we fix the test first. Which might be trivial, or might mean rewriting the test. That also means a TV bug will be opened the next time any of those tests is modified. Those bugs are already a low quality signal - they are usually incorrectly flagged as regressions of the bug which triggered the TV run.

I will file a follow up to allow individual tests to opt out of the comparePrefs checks (I think that's a must-have for some tests). But still wanted to raise the issue, maybe we should roll this out more progressively?

Comment 10

[Mark Banner (:standard8)]

Assuming that we're keeping this on for test-verify, or some kind of roll-out, please can we have announcements to dev-platform and firefox-dev?

This is a change in how we need to write mochitests, and it isn't obvious when you're running in non-verify mode (I suspect few developers run in verify mode normally). Although patches won't get backed out, additional bugs will get filed as regressions and developers will then get NIed, increasing the workload. Hence, developers should be warned that a change like this has happened, and teams might also want to go ahead and fix the existing issues in their areas like devtools is doing.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

In a perfect world every test that is tier-1 would pass test-verify. Also test-verify would do a lot more to ensure the test is independent.

When implementing this for mochitest-plain, we ended up with a ignorePrefs.json as these were changing on other timers or unrelated to tests.

Is there a way to add the prefs that devtools is adjusting to this file and then move forward? We even have limited support for *, which might solve the random sync prefs (although that might be a bug with sync?)

Probably the best way to roll this out is:

• announce a date ~2 weeks in the future (maybe July 15th? , meanwhile turn off by default)
• ensure all concerns are addressed leading up to that, ideally allowing people to opt in via try pushes and ./mach commands so this is something that can be fixed for teams that might have problematic tests.
• flip the switch.

unrelated, but similar to test-verify, I am working on re-annotating all the mochitests that will not work. Only starting with running standalone, but will make additional passes for repeat, chaos, and now preferences.

Comment 12

[Julian Descottes [:jdescottes]]

Even after fixing regular devtools preferences, we are getting bugs filed such as:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1975005
• https://bugzilla.mozilla.org/show_bug.cgi?id=1975007

which both mention privacy.trackingprotection.allow_list.hasMigratedCategoryPrefs which I can't find in the codebase. After a bit of digging it seems to come from Bug 1974449.

Comment 13

[Julian Descottes [:jdescottes]]

Is there a way to add the prefs that devtools is adjusting to this file and then move forward?

I implemented a logic which simply clears preferences at the end of the test, but limited to devtools preferences. I could just do it for all preferences, but in that case, that should really be the behavior of the test harness.

IMO this feature is not helping to find issues in the implementation, but issues in tests which don't cleanup prefs correctly. So if the test harness would be responsible for cleaning up most (all?) of the prefs modified during the test, that would be even more beneficial.

In a perfect world every test that is tier-1 would pass test-verify. Also test-verify would do a lot more to ensure the test is independent.

It would be better if this validation was performed earlier than test-verify, which only runs when a test is updated.
I think the end goal should be to enable this by default on all jobs, so that issues are detected very early. But prior to that there should be an effort to green up the tree, or to have an easy way to disable it in various test suites and let suite owners fix the tests when they have the bandwidth to do that.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

of the 50,000+ failures, I think it is misleading as there are mochitest-plain failures mixed in there which ends up being many of them.

I imagine some analysis could narrow this down to <1000 unique tests with probably on average have ~2 preferences each after adding some to the ignorePrefs.json file.

While this isn't trivial, it is probably realistic to achieve. There is no magic tooling for this, but it is something that can be achieved with a few hours of hacking.

Comment 15

[Julian Descottes [:jdescottes]]

Possibly, although there are already more than 1000 devtools tests failing :)

Do we agree that running this by default (ie not only on test-verify) is preferable? I think delaying the checks until test-verify is unnecessary and will lead to more issues down the road.

If we agree with this, maybe we can turn this off for now and start fixing the tests?

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

I have partial data (queries from the treeherder database- a limit of 40 errors/task).

devtools, there are 20 frequent prefs in 200 files:

devtools.everOpened
devtools.debugger.prefs-schema-version
devtools.toolbox.selectedTool
devtools.toolsidebar-width.inspector.splitsidebar
devtools.toolsidebar-height.inspector
devtools.toolsidebar-width.inspector
devtools.performance.popup.intro-displayed
devtools.debugger.pending-selected-location
devtools.netmonitor.columnsData
devtools.netmonitor.msg.visibleColumns
dom.ipc.processPrelaunch.enabled
sidebar.old-sidebar.has-used
devtools.debugger.log-event-breakpoints
devtools.aboutdebugging.collapsibilities.processes
devtools.aboutdebugging.tmpExtDirPath
devtools.inspector.activeSidebar
devtools.inspector.selectedSidebar
devtools.netmonitor.panes-search-width
devtools.netmonitor.panes-search-height
browser.download.lastDir

for browser-chrome there are 810 tests which use 248 prefs:

captivedetect.portalRecheckDelayMS
dom.security.https_only_mode_ever_enabled
doh-rollout.disable-heuristics
browser.toolbars.bookmarks.visibility
browser.uiCustomization.horizontalTabsBackup
sidebar.new-sidebar.has-used
browser.uiCustomization.navBarWhenVerticalTabs
services.sync.declinedEngines
services.sync.nextSync
privacy.clearOnShutdown_v2.formdata
privacy.sanitize.clearOnShutdown.hasMigratedToNewPrefs3
services.sync.globalScore
services.sync.clients.lastSync
browser.preferences.experimental.hidden
app.update.background.rolledout
extensions.ui.mlmodel.hidden
browser.engagement.sidebar-button.has-used
extensions.ui.dictionary.hidden
extensions.ui.locale.hidden
extensions.ui.extension.hidden
extensions.pendingOperations
extensions.ui.lastCategory
extensions.ui.sitepermission.hidden
sidebar.old-sidebar.has-used
app.normandy.user_id
identity.fxaccounts.toolbar.syncSetup.panelAccessed
messaging-system.profile.messagingProfileId
browser.engagement.fxa-toolbar-menu-button.has-used
identity.fxaccounts.toolbar.pxiToolbarEnabled
toolkit.osKeyStore.unofficialBuildOnlyLogin
browser.policies.alternatePath
media.decoder-doctor.testReportId.formats
media.decoder-doctor.MediaPlatformDecoderNotFound.formats
media.decoder-doctor.testReportId.button-clicked
media.decoder-doctor.MediaWMFNeeded.formats
media.decoder-doctor.MediaCannotInitializePulseAudio.formats
media.decoder-doctor.MediaUnsupportedLibavcodec.formats
media.videocontrols.picture-in-picture.video-toggle.has-used
browser.firefox-view.view-count
browser.urlbar.recentsearches.lastDefaultChanged
browser.uiCustomization.horizontalTabstrip
sidebar.visibility
sidebar.verticalTabs
extensions.webextensions.uuids
sidebar.main.tools
toolkit.tabbox.switchByScrolling
accessibility.typeaheadfind.flashBar
intl.locale.requested
browser.download.dir
browser.download.folderList
browser.policies.applied
browser.download.lastDir
browser.engagement.downloads-button.has-used
security.dialog_enable_delay
browser.download.save_converter_index
pdfjs.enableAltText
pdfjs.enableAltTextForEnglish
urlclassifier.phishTable
urlclassifier.malwareTable
urlclassifier.blockedTable
browser.protections_panel.infoMessage.seen
sidebar.backupState
browser.newtabpage.activity-stream.feeds.topsites
browser.newtabpage.activity-stream.discoverystream.endpointSpocsClear
browser.newtabpage.activity-stream.feeds.system.topsites
trailhead.firstrun.didSeeAboutWelcome
browser.profiles.profile-name.updated
browser.theme.content-theme
extensions.activeThemeID
browser.theme.toolbar-theme
privacy.sanitize.pending
privacy.clearOnShutdown_v2.browsingHistoryAndDownloads
privacy.clearOnShutdown_v2.cache
privacy.clearHistory.siteSettings
privacy.sanitize.cpd.hasMigratedToNewPrefs3
privacy.clearHistory.formdata
privacy.clearHistory.cookiesAndStorage
privacy.sanitize.timeSpan
privacy.cpd.cookies
privacy.cpd.sessions
privacy.cpd.cache
privacy.cpd.history
privacy.cpd.formdata
privacy.cpd.siteSettings
privacy.cpd.downloads
security.tls.version.max
datareporting.policy.dataSubmissionPolicyNotifiedTime
datareporting.policy.dataSubmissionPolicyAcceptedVersion
browser.preonboarding.enabled
security.data_uri.block_toplevel_data_uri_navigations
browser.bookmarks.editDialog.confirmationHintShowCount
browser.engagement.library-button.has-used
datareporting.dau.cachedUsageProfileGroupID
datareporting.dau.cachedUsageProfileID
app.update.lastUpdateTime.rs-experiment-loader-timer
distribution.version
distribution.about
distribution.id
network.trr.uri
app.update.interval
security.ssl3.deprecated.rsa_des_ede3_sha
browser.tabs.inTitlebar
browser.toolbarbuttons.introduced.sidebar-button
browser.engagement.home-button.has-removed
browser.search.totalSearches
permissions.desktop-notification.notNow.enabled
dom.webnotifications.requireuserinteraction
screenshots.browser.component.last-screenshot-method
screenshots.browser.component.last-saved-method
mochitest7-track-simple
mochitest6-track-simple
security.notification_enable_delay
browser.contentblocking.report.hide_vpn_banner
app.update.lastUpdateTime.telemetry_untrustedmodules_ping
browser.search.widget.lastUsed
browser.urlbar.placeholderName
browser.newtabpage.activity-stream.trendingSearch.defaultSearchEngine
browser.aboutwelcome.didSeeFinalScreen
test2.pref1
test1.pref2
test1.pref1
logging.config.output_type
sidebar.revamp
browser.engagement.ctrlTab.has-used
browser.toolbarbuttons.introduced.test-messing-with-default-placements-new-pref
ui.key.menuAccessKeyFocuses
screenshots.browser.component.enabled
identity.fxaccounts.remote.root
browser.newtabpage.activity-stream.discoverystream.config
browser.newtabpage.activity-stream.discoverystream.personalization.override
identity.fxaccounts.account.device.name
browser.backup.scheduled.last-backup-timestamp
browser.backup.scheduled.last-backup-file
browser.backup.location
browser.ml.linkPreview.onboardingTimes
dom.security.https_first
intl.accept_languages
privacy.trackingprotection.consentmanager.skip.pbmode.enabled
privacy.fingerprintingProtection
network.http.referer.disallowCrossSiteRelaxingDefault.top_navigation
privacy.trackingprotection.emailtracking.enabled
privacy.query_stripping.enabled.pbmode
privacy.query_stripping.enabled
privacy.annotate_channels.strict_list.enabled
privacy.trackingprotection.socialtracking.enabled
browser.contentblocking.cfr-milestone.milestone-achieved
urlclassifier.features.socialtracking.annotate.blacklistTables
urlclassifier.features.socialtracking.blacklistTables
urlclassifier.trackingTable
privacy.trackingprotection.enabled
urlclassifier.trackingAnnotationWhitelistTable
urlclassifier.trackingAnnotationTable
urlclassifier.trackingWhitelistTable
urlclassifier.features.emailtracking.blocklistTables
ui.caretBlinkTime
devtools.everOpened
devtools.toolsidebar-width.inspector.splitsidebar
devtools.debugger.prefs-schema-version
devtools.toolbox.footer.height
devtools.toolsidebar-height.inspector
devtools.performance.recording.entries
devtools.toolbox.selectedTool
devtools.performance.recording.features
devtools.toolbox.previousHost
devtools.toolsidebar-width.inspector
devtools.performance.recording.threads
devtools.netmonitor.msg.visibleColumns
media.navigator.permission.fake
extensions.install.requireBuiltInCerts
xpinstall.whitelist.required
signon.rememberSignons
browser.startup.page
app.support.baseURL
browser.newtabpage.activity-stream.support.url
browser.region.update.region
browser.region.update.updated
browser.region.update.first-seen
places.interactions.customBlocklist
browser.sessionstore.resume_from_crash
browser.tabs.firefox-view.ui-state.recentlyclosed.open
browser.tabs.firefox-view.ui-state.opentabs.open
browser.tabs.firefox-view.importHistory.dismissed
browser.tabs.firefox-view.ui-state.opentabs.sort-option
app.update.lastUpdateTime.content-relevancy-timer
app.update.background.enabled
browser.contentblocking.cfr-milestone.milestone-shown-time
browser.startup.homepage_override.once
browser.search.region
distribution.iniFile.exists.appversion
distribution.iniFile.exists.value
distribution.mozilla-MSIX.bookmarksProcessed
browser.shell.mostRecentDefaultPromptSeen
app.update.disable_button.showUpdateHistory
pdfjs.enabledCache.state
pref.downloads.disable_button.edit_actions
security.mixed_content.upgrade_display_content
privacy.purge_trackers.date_in_cookie_database
browser.engagement.home-button.has-used
browser.sessionstore.upgradeBackup.latestBuildID
privacy.purge_trackers.last_purge
browser.sessionstore.restore_on_demand
browser.sessionstore.idleDelay
test.existingPref
.testing.does-not-exist
test.mismatchPref
app.normandy.startupExperimentPrefs.extensions.recommendations.privacyPolicyUrl
app.update.lastUpdateTime.recipe-client-addon-run
app.normandy.run_interval_seconds
identity.fxaccounts.auth.uri
services.sync.lastTabFetch
app.update.lastUpdateTime.suggest-ingest
browser.urlbar.events.disableSuggest.maxSecondsFromLastSearch
services.settings.server
browser.urlbar.tabToSearch.onboard.interactionsLeft
datareporting.healthreport.uploadEnabled
browser.laterrun.bookkeeping.updateAppliedTime
media.webspeech.synth.test
browser.search.serpEventTelemetryCategorization.regionEnabled
messaging-system-action.test.multi.pref
messaging-system-action.showEmbeddedImport
security.client_auth_certificate_default_remember_setting
testing.allowed-prefs.some-char-pref
browser.newtabpage.activity-stream.feeds.section.highlights
browser.newtabpage.blocked
browser.newtabpage.activity-stream.asrouter.providers.cfr
browser.newtabpage.activity-stream.discoverystream.thumbsUpDown.enabled
browser.newtabpage.activity-stream.discoverystream.topicLabels.enabled
browser.newtabpage.activity-stream.weather.temperatureUnits
browser.region.network.url
browser.shell.checkDefaultBrowser
geo.provider.network.url
geo.prompt.testing
geo.prompt.testing.allow
toolkit.telemetry.user_characteristics_ping.uuid
browser.newtabpage.activity-stream.feeds.section.topstories.options
browser.newtabpage.activity-stream.system.showSponsored
browser.newtabpage.activity-stream.improvesearch.topSiteSearchShortcuts.searchEngines
network.process.enabled
reader.color_scheme
reader.custom_colors.visited-links
reader.custom_colors.selection-highlight
reader.custom_colors.background
reader.custom_colors.foreground
reader.custom_colors.unvisited-links
reader.character_spacing
reader.word_spacing
browser.bookmarks.editDialog.firstEditField
network.lna.block_trackers
extensions.webcompat-reporter.enabled
browser.startup.homepage
ui.new-webcompat-reporter.reason-dropdown.randomized
signon.firefoxRelay.feature

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

keep in mind the max_errors that treeherder parses could mean 2-3x the number in reality.

Comment 18

[Mark Banner (:standard8)]

I'm wondering if we reset the prefs between tests (like for mochitest-plain), if it would cause less failures overall - though that might cause other tests to fail?

In looking at some of the search tests, various prefers were touched by a couple of tests, but then had follow-on effects on other tests in the directory.

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

if things are reset between tests and some tests start failing- this is a good thing- the majority of those will be tests that are making assumptions based on a bad browser state. we should see what the failures really look like. I suspect it will be a short list.

Comment 20

[Rob Wu [:robwu]]

I implemented --compare-preferences support because it is opt-in, and TV failures are tier 2. In the rollout of bug 1783248 two years ago, there were some failures at first, but that died out once the affected tests fixed up their tests. My try push in comment 4 shows that there are still many plain mochitests that would still fail if run on TV.

EDIT: Although we print TEST-UNEXPECTED-FAIL (which appears in the log), the TV job is not interrupted, so the only negative effect is the filing of bug reports for TV failures.
A clear negative side effect of force-enabling the comparePrefs option in TV is that devs don't get the benefit from the other test coverage in TV, because the test fails early. This issue could be countered by moving the comparePrefs option to the last step of test verification (steps declared here), basically introducing another run that sets the comparePrefs option to True for that run. To those who really want to have --compare-preferences behavior early in TV, we could ensure that --compare-preferences is supported, so that someone can run the following locally: ./mach test path/to/mochitest.js --compare-preferences --verify

While delaying the comparePrefs check until the end of TV would help with maximizing the value out of TV jobs, it doesn't magically fix the issue of bugs getting filed due to a TV failure caused by an unexpected preference change. Fixing this properly would be more time consuming, but I see several options that do not alter the test behavior:

• Add a blanket exception for all knowingly failing prefs and make an effort to shrink the set of exceptions over time. With the existing mechanisms, this means adding the prefs from comment 16 to ignorePrefs.json.
• Changing the hard failure to a mere warning, to be printed at the end of test verification. When run locally with --verify, developers can see the warnings, but it won't cause failures locally nor in CI.
• Setting comparePrefs to False by default when the test flavor is a browser test (or equivalently, only set it when it is a plain mochitest, because the --compare-preferences feature is not implemented for other test types IIUC). That restores the TV behavior to before my patch.

I'm intentionally listing options that do not alter test behavior, because my objective here was implementing --compare-preferences, without taking on the task of fixing tons of other tests across the whole code base.

:jmaher, what do you think that we should do here?

Comment 21

[Florian Quèze [:florian]]

(In reply to Julian Descottes [:jdescottes] from comment #15)

Do we agree that running this by default (ie not only on test-verify) is preferable?

I think there's general agreement on this and what remains to discuss is: how can we smoothly get there?

My proposal would be:

• enable checking for changed preference by default, not just in test-verify mode, but as a first step make it only a warning, or an EXPECTED-FAIL. This way it wouldn't disturb any in progress patches, but would become visible sooner than later to engineers running tests locally.
• stop failing TV jobs for changed preferences, until we are ready to make this a failure by default too.
• analyze the set of warnings produced in a few try pushes, and use that to:
  • add to the ignorePrefs.json file the prefs that are fine to leave behind. Typically this would be prefs set by features after they are used for the first time.
  • use registerCleanupFunction in the head.js file of test manifests where many tests leave the same preferences behind.
  • for prefs that are left behind by very few tests, use a semi-automated process to modify the test files to cleanup these prefs. This could mean replacing Services.prefs.set*Pref calls with SpecialPowers.pushPrefEnv calls, or adding a cleanup function using registerCleanupFunction.

I'm willing to help to cleanup the tests.

Comment 22

[Mike Kaply [:mkaply]]

I'm unclear as to what to do here to fix my tests.

My prefs are set in the toml files:

prefs = [
"app.update.disabledForTesting=false",
"browser.policies.alternatePath='<test-root>/browser/components/enterprisepolicies/tests/browser/disable_app_update/config_disable_app_update.json'",
]

and they are causing this error to show up in all my tests.

Comment 23

[Rob Wu [:robwu]]

(Clearing needinfo; bug 1975980 will reduce noise from existing failures in TV / --verify tasks)