Closed Bug 1975980 Opened 4 months ago Closed 4 months ago

Reset preferences between all browser-chrome mochitests

Tracking

(firefox142 fixed)

Status:

RESOLVED FIXED

Milestone:

142 Branch

Tracking Flags:

Tracking

Status

firefox142

---

fixed

People

(Reporter: florian, Assigned: florian)

References

(Depends on 1 open bug, Blocks 3 open bugs)

Details

Attachments

(11 files, 2 obsolete files)

Bug 1975980 - accept for now that we can't reset prefs that were modified on the default branch, r=jmaher. 4 months ago Florian Quèze [:florian] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1975980 - report pref changes as expected failures, upload a modifiedPrefs.json artifacts with all the pref changes of the job, r=jmaher. 4 months ago Florian Quèze [:florian] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1975980 - add preferences modified by many tests to testing/mochitest/ignorePrefs.json, r=jmaher. 4 months ago Florian Quèze [:florian] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1975980 - report uncaught rejections caused by preference observers firing when resetting prefs after a test, r=jmaher. 4 months ago Florian Quèze [:florian] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1975980 - fix test failures due to the dom.security.https_first preference being inconsistently set, r=jmaher. 4 months ago Florian Quèze [:florian] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1975980 - fix browser_newtab_header.js, r=jmaher. 4 months ago Florian Quèze [:florian] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1975980 - clear observed prefs when uninitializing SelectableProfileService to avoid uncaught exception when some observed prefs are automatically reset at the end of tests, r=#profiles-reviewers. 4 months ago Florian Quèze [:florian] 48 bytes, text/x-phabricator-request		Details \| Review
node.js script to fetch all the modifiedPrefs.json artifacts from a push and generate a report 4 months ago Florian Quèze [:florian] 4.92 KB, application/x-javascript		Details
Test Preferences Report.html 4 months ago Florian Quèze [:florian] 1.33 MB, text/html		Details
Bug 1975980 - annotate as fail-if the tests that were passing in M-vt only because another test was setting sidebar.verticalTabs to true, r=jmaher. 4 months ago Florian Quèze [:florian] 48 bytes, text/x-phabricator-request		Details \| Review
node.js script to fetch all the modifiedPrefs.json artifacts from a push and generate a report 4 months ago Florian Quèze [:florian] 4.91 KB, application/x-javascript		Details
node.js script to fetch all the modifiedPrefs.json artifacts from a push and generate a report 4 months ago Florian Quèze [:florian] 5.72 KB, application/x-javascript		Details
Preferences changed by tests in push 1683450.html 4 months ago Florian Quèze [:florian] 456.97 KB, text/html		Details

Florian Quèze [:florian]

Assignee

Description

•

4 months ago

This is a pre-requisite to be able to make test-verify (TV) tier 1.

Florian Quèze [:florian]

Assignee

Comment 1

•

4 months ago

Attached file Bug 1975980 - accept for now that we can't reset prefs that were modified on the default branch, r=jmaher. — Details

Florian Quèze [:florian]

Assignee

Comment 2

•

4 months ago

Attached file Bug 1975980 - report pref changes as expected failures, upload a modifiedPrefs.json artifacts with all the pref changes of the job, r=jmaher. — Details

Florian Quèze [:florian]

Assignee

Comment 3

•

4 months ago

Attached file Bug 1975980 - add preferences modified by many tests to testing/mochitest/ignorePrefs.json, r=jmaher. — Details

Florian Quèze [:florian]

Assignee

Comment 4

•

4 months ago

Attached file Bug 1975980 - report uncaught rejections caused by preference observers firing when resetting prefs after a test, r=jmaher. — Details

Florian Quèze [:florian]

Assignee

Comment 5

•

4 months ago

Attached file Bug 1975980 - fix test failures due to the dom.security.https_first preference being inconsistently set, r=jmaher. — Details

Florian Quèze [:florian]

Assignee

Comment 6

•

4 months ago

Attached file Bug 1975980 - fix browser_newtab_header.js, r=jmaher. — Details

Florian Quèze [:florian]

Assignee

Comment 7

•

4 months ago

Attached file Bug 1975980 - clear observed prefs when uninitializing SelectableProfileService to avoid uncaught exception when some observed prefs are automatically reset at the end of tests, r=#profiles-reviewers. — Details

Rob Wu [:robwu]

Comment 8

•

4 months ago

Why is this work limited to browser-chrome?

Although plain mochitests already reset between prefs, they'd still report TV failures if a preference was not cleaned up. You can see plain mochitest failures in the try push that force-enables the comparePrefs option (--compare-preferences) in https://bugzilla.mozilla.org/show_bug.cgi?id=1816549#c4

Here are the results from the plain mochitest jobs on Linux 18.04 tsan. Some may be test-specific, others (e.g. browser.sessionstore.upgradeBackup.latestBuildID) look like something that should be in ignorePrefs.json, for any mochitest kind:

1
TEST-UNEXPECTED-FAIL | /tests/testing/mochitest/tests/Harness_sanity/test_SpecialPowersPushPrefEnv.html | changed preference: test.cleanup

2
TEST-UNEXPECTED-FAIL | /tests/dom/push/test/test_has_permissions.html | changed preference: dom.push.userAgentID
TEST-UNEXPECTED-FAIL | /tests/dom/push/test/test_multiple_register.html | changed preference: dom.push.userAgentID
TEST-UNEXPECTED-FAIL | /tests/dom/push/test/test_multiple_register_different_scope.html | changed preference: dom.push.userAgentID
TEST-UNEXPECTED-FAIL | /tests/dom/push/test/test_permissions.html | changed preference: dom.push.userAgentID
TEST-UNEXPECTED-FAIL | /tests/dom/push/test/test_register.html | changed preference: dom.push.userAgentID
TEST-UNEXPECTED-FAIL | /tests/dom/push/test/test_subscription_change.html | changed preference: dom.push.userAgentID
TEST-UNEXPECTED-FAIL | /tests/dom/push/test/test_try_registering_offline_disabled.html | changed preference: dom.push.userAgentID
TEST-UNEXPECTED-FAIL | /tests/dom/push/test/test_unregister.html | changed preference: dom.push.userAgentID
TEST-UNEXPECTED-FAIL | /tests/dom/push/test/test_data.html | changed preference: dom.push.userAgentID

3
TEST-UNEXPECTED-FAIL | /tests/dom/quota/test/mochitest/test_storage_manager_persist_deny.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID
TEST-UNEXPECTED-FAIL | /tests/gfx/tests/mochitest/test_font_whitelist.html | changed preference: font.fixme.hack
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/extensions/test/mochitest/test_ext_protocolHandlers.html | changed preference: extensions.webextensions.uuids
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/extensions/test/mochitest/test_ext_request_urlClassification.html | changed preference: mochitest7-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/extensions/test/mochitest/test_ext_request_urlClassification.html | changed preference: mochitest6-track-simple

4
TEST-UNEXPECTED-FAIL | /tests/image/test/mochitest/test_bug496292.html | changed preference: image.http.accept

5
TEST-UNEXPECTED-FAIL | /tests/dom/security/featurepolicy/test/mochitest/test_xslt.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/extensions/test/mochitest/test_ext_storage_cleanup.html | changed preference: extensions.webextensions.uuids
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/extensions/test/mochitest/test_ext_protocolHandlers.html | changed preference: extensions.webextensions.uuids
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/extensions/test/mochitest/test_ext_request_urlClassification.html | changed preference: mochitest7-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/extensions/test/mochitest/test_ext_request_urlClassification.html | changed preference: mochitest6-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/extensions/test/mochitest/test_ext_storage_manager_capabilities.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID

7
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/cors/test_CrossSiteXHR.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/passwordmgr/test/mochitest/test_prompt_async.html | changed preference: prompts.authentication_dialog_abuse_limit
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/passwordmgr/test/mochitest/test_LoginManagerContent_passwordEditedOrGenerated.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID

8
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/csp/test_ping.html | changed preference: browser.send_pings
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/pdfjs/test/test_pdf_file_in_iframe.html | changed preference: pdfjs.enableAltText
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/pdfjs/test/test_pdf_file_in_iframe.html | changed preference: pdfjs.enableAltTextForEnglish
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/pdfjs/test/test_pdf_file_in_object.html | changed preference: pdfjs.enableAltText
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/pdfjs/test/test_pdf_file_in_object.html | changed preference: pdfjs.enableAltTextForEnglish

9
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_allow_opening_data_pdf.html | changed preference: pdfjs.enableAltText
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_allow_opening_data_pdf.html | changed preference: pdfjs.enableAltTextForEnglish
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_block_toplevel_data_img_navigation.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_block_toplevel_data_navigation.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_bug1450853.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_bug1660452_http.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_cache_split.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_contentpolicytype_targeted_link_iframe.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_gpc.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_gpc.html | changed preference: privacy.globalprivacycontrol.was_ever_enabled
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_meta_referrer.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_nosniff.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_nosniff_navigation.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_same_site_cookies_about.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_same_site_cookies_cross_origin_context.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_same_site_cookies_from_script.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_same_site_cookies_iframe.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_same_site_cookies_laxByDefault.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_same_site_cookies_redirect.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_same_site_cookies_subrequest.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_same_site_cookies_toplevel_nav.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_same_site_cookies_toplevel_set_cookie.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/general/test_xfo_error_page.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/websocket/tests/test_websocket_mixed_content_blob.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID

10
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/https-first/test_redirect_upgrade.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/satchel/test/test_password_autocomplete.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID

11
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/https-only/test_fragment.html | changed preference: dom.security.https_only_mode_ever_enabled
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/https-only/test_http_background_auth_request.html | changed preference: dom.security.https_only_mode_ever_enabled
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/https-only/test_http_background_request.html | changed preference: dom.security.https_only_mode_ever_enabled
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/https-only/test_insecure_reload.html | changed preference: dom.security.https_only_mode_ever_enabled
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/https-only/test_user_suggestion_box.html | changed preference: dom.security.https_only_mode_ever_enabled
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/https-only/test_redirect_upgrade.html | changed preference: dom.security.https_only_mode_ever_enabled
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/https-only/test_resource_upgrade.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID
TEST-UNEXPECTED-FAIL | /tests/dom/security/test/https-only/test_resource_upgrade.html | changed preference: dom.security.https_only_mode_ever_enabled
TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_subworkers_suspended.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/url-classifier/tests/mochitest/test_cryptomining.html | changed preference: mochitest7-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/url-classifier/tests/mochitest/test_cryptomining.html | changed preference: mochitest6-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/url-classifier/tests/mochitest/test_emailtracking.html | changed preference: mochitest7-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/url-classifier/tests/mochitest/test_emailtracking.html | changed preference: mochitest6-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/url-classifier/tests/mochitest/test_emailtracking.html | changed preference: urlclassifier.features.emailtracking.blocklistTables
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/url-classifier/tests/mochitest/test_fingerprinting.html | changed preference: mochitest7-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/url-classifier/tests/mochitest/test_fingerprinting.html | changed preference: mochitest6-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/url-classifier/tests/mochitest/test_socialtracking.html | changed preference: mochitest7-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/url-classifier/tests/mochitest/test_socialtracking.html | changed preference: mochitest6-track-simple
TEST-UNEXPECTED-FAIL | /tests/toolkit/components/url-classifier/tests/mochitest/test_socialtracking.html | changed preference: urlclassifier.features.socialtracking.blacklistTables

12
TEST-UNEXPECTED-FAIL | /tests/dom/worklet/tests/test_audioWorklet_options.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID

13
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_bug1499169.html | changed preference: pdfjs.enableAltText
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_bug1499169.html | changed preference: pdfjs.enableAltTextForEnglish
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_data_uri.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_delazification_strategy.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_document.all_iteration.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_document.all_unqualified.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_document_constructor.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_document_importNode_document.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_document_wireframe.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_domparser_null_char.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_domparsing.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_domwindowutils.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_drag_drop_shadow_crossing_selection.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_element.matches.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_elementTraversal.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_element_closest.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_embed_xorigin_document.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_encodeToStringWithMaxLength.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_encodeToStringWithRequiresReinitAfterOutput.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_eventsource_event_listener_leaks.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_eventsourceservice_basic.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_eventsourceservice_reconnect_error.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_eventsourceservice_status_error.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_eventsourceservice_worker.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_explicit_user_agent.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_find.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_find_bug1601118.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_find_bug1654683.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_find_nac.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_focus_design_mode.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_focus_display_none_xorigin_iframe.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_focus_keyboard_event.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_focus_radio.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_focus_scroll_padding_tab.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_focus_scrollable_fieldset.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_focus_scrollable_input.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_focus_shadow_dom.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_focus_shadow_dom_root.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_getAttribute_after_createAttribute.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations
TEST-UNEXPECTED-FAIL | /tests/dom/base/test/test_getElementById.html | changed preference: security.data_uri.block_toplevel_data_uri_navigations

14
TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/fetch/test_fetch_cors.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID

15
TEST-UNEXPECTED-FAIL | /tests/dom/notification/test/mochitest/test_notification_crossorigin_iframe.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID

16
TEST-UNEXPECTED-FAIL | /tests/dom/serviceworkers/test/test_serviceworker_interfaces.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID
TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/general/test_interfaces_secureContext.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID

17
TEST-UNEXPECTED-FAIL | /tests/dom/geolocation/test/mochitest/test_cancelCurrent.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID

18
TEST-UNEXPECTED-FAIL | /tests/dom/serviceworkers/test/test_fetch_event_with_thirdpartypref.html | changed preference: privacy.bounceTrackingProtection.hasMigratedUserActivationData
TEST-UNEXPECTED-FAIL | /tests/dom/serviceworkers/test/test_navigationPreload_disable_crash.html | changed preference: browser.sessionstore.upgradeBackup.latestBuildID
TEST-UNEXPECTED-FAIL | /tests/netwerk/test/mochitests/test_pdf.html | changed preference: pdfjs.enableAltText
TEST-UNEXPECTED-FAIL | /tests/netwerk/test/mochitests/test_pdf.html | changed preference: pdfjs.enableAltTextForEnglish
TEST-UNEXPECTED-FAIL | /tests/netwerk/test/mochitests/test_rel_preconnect.html | changed preference: network.http.debug-observations

19
TEST-UNEXPECTED-FAIL | /tests/uriloader/exthandler/tests/mochitest/test_invalidCharFileExtension.xhtml | changed preference: browser.download.dir
TEST-UNEXPECTED-FAIL | /tests/uriloader/exthandler/tests/mochitest/test_invalidCharFileExtension.xhtml | changed preference: browser.download.folderList
TEST-UNEXPECTED-FAIL | /tests/uriloader/exthandler/tests/mochitest/test_nullCharFile.xhtml | changed preference: browser.download.dir
TEST-UNEXPECTED-FAIL | /tests/uriloader/exthandler/tests/mochitest/test_nullCharFile.xhtml | changed preference: browser.download.folderList
TEST-UNEXPECTED-FAIL | /tests/uriloader/exthandler/tests/mochitest/test_unsafeBidiChars.xhtml | changed preference: browser.download.dir
TEST-UNEXPECTED-FAIL | /tests/uriloader/exthandler/tests/mochitest/test_unsafeBidiChars.xhtml | changed preference: browser.download.folderList

Florian Quèze [:florian]

Assignee

Comment 9

•

4 months ago

(In reply to Rob Wu [:robwu] from comment #8)

Why is this work limited to browser-chrome?

Because I needed to start somewhere, and most of the complains I have seen recently are related to TV failures reported for browser-chrome tests.

Although plain mochitests already reset between prefs, they'd still report TV failures if a preference was not cleaned up.

So do you think I should change mochitest-plain tests to report failures as TEST-EXPECTED-FAIL in TV (when --compare-preferences is not set) to be consistent with what I'm doing here for browser-chrome tests?

Rob Wu [:robwu]

Comment 10

•

4 months ago

(In reply to Florian Quèze [:florian] from comment #9)

(In reply to Rob Wu [:robwu] from comment #8)

Why is this work limited to browser-chrome?

Because I needed to start somewhere, and most of the complains I have seen recently are related to TV failures reported for browser-chrome tests.

Sounds reasonable. I was curious about the context of this bug, and when I looked at the parent meta, I only saw browser-chrome and not plain mochitests as a dependency. That made me wonder whether it was overlooked or a todo without bug.

Although plain mochitests already reset between prefs, they'd still report TV failures if a preference was not cleaned up.

So do you think I should change mochitest-plain tests to report failures as TEST-EXPECTED-FAIL in TV (when --compare-preferences is not set) to be consistent with what I'm doing here for browser-chrome tests?

The concerns with browser-chrome failures in TV are equally valid with plain mochitests, and that they would be a blocker to "TV as tier1" (bug 1975728). Therefore I'd expect similar logic in test behavior in plain mochitests and browser-chrome tests. This is just informal feedback, I don't have any authority on this matter.

Florian Quèze [:florian]

Assignee

Comment 11

•

4 months ago

Attached file node.js script to fetch all the modifiedPrefs.json artifacts from a push and generate a report (obsolete) — Details

Florian Quèze [:florian]

Assignee

Comment 12

•

4 months ago

Attached file Test Preferences Report.html — Details

Report generated by the script in attachment 9499050 [details] before adding to the ignorePrefs.json file (attachment 9498919 [details]), showing 327 prefs modified by 1375 tests.

Florian Quèze [:florian]

Assignee

Updated

•

4 months ago

Depends on: 1976110

Florian Quèze [:florian]

Assignee

Comment 13

•

4 months ago

Attached file Bug 1975980 - annotate as fail-if the tests that were passing in M-vt only because another test was setting sidebar.verticalTabs to true, r=jmaher. — Details

Florian Quèze [:florian]

Assignee

Updated

•

4 months ago

Keywords: leave-open

Pulsebot

Comment 14

•

4 months ago

Pushed by fqueze@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/e4c55fb77b63 https://hg.mozilla.org/integration/autoland/rev/d3d1f335ca7f fix test failures due to the dom.security.https_first preference being inconsistently set, r=jmaher,necko-reviewers,sunil. https://github.com/mozilla-firefox/firefox/commit/c523c77f6346 https://hg.mozilla.org/integration/autoland/rev/e2c3fe9dd340 fix browser_newtab_header.js, r=jmaher,home-newtab-reviewers,mconley. https://github.com/mozilla-firefox/firefox/commit/f00df01e3c6c https://hg.mozilla.org/integration/autoland/rev/54529ef18b57 clear observed prefs when uninitializing SelectableProfileService to avoid uncaught exception when some observed prefs are automatically reset at the end of tests, r=profiles-reviewers,niklas.

Cristina Horotan [:chorotan]

Comment 15

•

4 months ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/d3d1f335ca7f
https://hg.mozilla.org/mozilla-central/rev/e2c3fe9dd340
https://hg.mozilla.org/mozilla-central/rev/54529ef18b57

Pulsebot

Comment 16

•

4 months ago

Pushed by fqueze@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/01aa71b2567c https://hg.mozilla.org/integration/autoland/rev/6c241901a11f accept for now that we can't reset prefs that were modified on the default branch, r=jmaher. https://github.com/mozilla-firefox/firefox/commit/fbdf128d83d0 https://hg.mozilla.org/integration/autoland/rev/17cabe229561 add preferences modified by many tests to testing/mochitest/ignorePrefs.json, r=jmaher. https://github.com/mozilla-firefox/firefox/commit/a97c6e69da80 https://hg.mozilla.org/integration/autoland/rev/8d7a829ae5af report uncaught rejections caused by preference observers firing when resetting prefs after a test, r=jmaher.

Atila Butkovits

Comment 17

•

4 months ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/6c241901a11f
https://hg.mozilla.org/mozilla-central/rev/17cabe229561
https://hg.mozilla.org/mozilla-central/rev/8d7a829ae5af

Florian Quèze [:florian]

Assignee

Comment 18

•

4 months ago

Attached file node.js script to fetch all the modifiedPrefs.json artifacts from a push and generate a report (obsolete) — Details

Attachment #9499050 - Attachment is obsolete: true

Rob Wu [:robwu]

Updated

•

4 months ago

Blocks: 1976067

Pulsebot

Comment 19

•

4 months ago

Pushed by fqueze@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/01fbea5b1558 https://hg.mozilla.org/integration/autoland/rev/e8a505a17462 annotate as fail-if the tests that were passing in M-vt only because another test was setting sidebar.verticalTabs to true, r=jmaher,tabbrowser-reviewers,nsharpley.

Pulsebot

Comment 20

•

4 months ago

Pushed by sstanca@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/838b213def54 https://hg.mozilla.org/integration/autoland/rev/24935e432860 Revert "Bug 1975980 - annotate as fail-if the tests that were passing in M-vt only because another test was setting sidebar.verticalTabs to true, r=jmaher,tabbrowser-reviewers,nsharpley." for causing mochitests failures in browser_tabCloseProbes.js.

Serban Stanca [:SerbanS]

Comment 21

•

4 months ago

Reverted this because it was causing mochitests failures in browser_tabCloseProbes.js.

Revert link
Push with failures
Failure Log
Failure line: TEST-UNEXPECTED-PASS | browser/components/tabbrowser/test/browser/tabs/browser_tabCloseProbes.js | We expect at least one assertion to fail because this test file is marked as fail-if in the manifest. -

Flags: needinfo?(florian)

Florian Quèze [:florian]

Assignee

Comment 22

•

4 months ago

(In reply to Serban Stanca [:SerbanS] from comment #21)

Reverted this because it was causing mochitests failures in browser_tabCloseProbes.js.

Failure line: TEST-UNEXPECTED-PASS | browser/components/tabbrowser/test/browser/tabs/browser_tabCloseProbes.js | We expect at least one assertion to fail because this test file is marked as fail-if in the manifest. -

Oops, because I annotated failures as fail-if rather than skip-if, I can't land D256365 without D256279.

Flags: needinfo?(florian)

Pulsebot

Comment 23

•

4 months ago

Pushed by fqueze@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/a44d92cab1f7 https://hg.mozilla.org/integration/autoland/rev/43162516f4cd annotate as fail-if the tests that were passing in M-vt only because another test was setting sidebar.verticalTabs to true, r=jmaher,tabbrowser-reviewers,nsharpley. https://github.com/mozilla-firefox/firefox/commit/5accfd747d86 https://hg.mozilla.org/integration/autoland/rev/1ae3bc6159b4 report pref changes as expected failures, upload a modifiedPrefs.json artifacts with all the pref changes of the job, r=jmaher.

Florian Quèze [:florian]

Assignee

Updated

•

4 months ago

Keywords: leave-open

Norisz Fay [:noriszfay]

Comment 24

•

4 months ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/43162516f4cd
https://hg.mozilla.org/mozilla-central/rev/1ae3bc6159b4

Status: ASSIGNED → RESOLVED

Closed: 4 months ago

status-firefox142: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → 142 Branch

Kai Engert [:KaiE:]

Updated

•

4 months ago

Regressions: 1977567

Florian Quèze [:florian]

Assignee

Comment 25

•

4 months ago

Attached file node.js script to fetch all the modifiedPrefs.json artifacts from a push and generate a report — Details

Attachment #9499725 - Attachment is obsolete: true

Florian Quèze [:florian]

Assignee

Comment 26

•

4 months ago

Attached file Preferences changed by tests in push 1683450.html — Details

I'm attaching the report generated by attachment 9500934 [details] for the mozilla-central push that includes the changesets in comment 24.

It shows 316 prefs modified by 572 tests.

Rob Wu [:robwu]

Updated

•

4 months ago

Blocks: 1974991

Julian Descottes [:jdescottes]

Updated

•

4 months ago

Updated

•

4 months ago

Component: General → Mochitest

Florian Quèze [:florian]

Assignee

Updated

•

4 months ago

Blocks: 1978278

Erik Nordin [:nordzilla]

Updated

•

4 months ago

Blocks: 1978441

Erik Nordin [:nordzilla]

Updated

•

4 months ago

Updated

•

4 months ago

Regressions: 1977611

Florian Quèze [:florian]

Assignee

Updated

•

4 months ago

Regressions: 1979209

Mathew Hodson

Updated

•

3 months ago

Regressions: 1977605

You need to log in before you can comment on or make changes to this bug.