1816449 - Add exception list for https-first with remote settings

Status: RESOLVED FIXED

(Core :: DOM: Security, enhancement, P1)

Description

[Frederik Braun [:freddy]]

HTTPS-First allows for fallbacks when a website does not provide HTTPS. However, there is a low number of websites that support HTTPS but do not work over HTTPS (prime example is http://www.bom.gov.au/). Therefore, we need an opt-out.
Our Product team argues that a browser button is a poor choice because then we will put the onus on the end user, who already has a (somewhat undesirable) "unbreak this website please" button with the Tracking Protection. Having two buttons that serve a similar flow would be unfortunate.

Therefore, we want to work with a list of websites that should not be upgraded.

Tim Huang suggests using remote settings collectio on the server end and implementing a client in Firefox to consume this. The list itself could live on GitHub and be maintained as a separate repo.

Further pointers (from Tim)

• https://github.com/mozilla/cookie-banner-rules-list
• PartitioningExceptionListService

Comment 1

[Malte Jürgens [:maltejur]]

Attachment: WIP: Bug 1816449 - Add pref to exempt domains from HTTPS-Only/First r?freddyb

Comment 2

[Malte Jürgens [:maltejur]]

To add a exemption list for HTTPS-Only/-First, we have decided a while ago that between remote settings, modifying the webcompat addon, and custom prefs, prefs are the best choice. They are the easiest to implement and can be updated quickly between releases if a site breaks because of HTTPS-Only/-First. So I have attached a prototype patch that adds four new prefs that implement an allowlist. They work as following:

dom.security.https_only_mode.exempt_domains.managed - Comma-separated list of eTLD+1 domains which should be ignored by HTTPS-Only. This pref is where the Mozilla-managed exemptions should be stored and updated. This pref should not be edited by users, unless they do not want to receive updates for exemptions.

dom.security.https_only_mode.exempt_domains.user_overrides - Similar comma-separated list as the first pref, although this is meant for users to individually override exemptions from the managed pref, or add their own exemptions. The entries of this list can optionally be prefixed with a - to indicate that a exemption should be removed instead of added. This list has priority over the first one.

dom.security.https_first.exempt_domains.managed and dom.security.https_first.exempt_domains.user_overrides - Similar to first two, although this time for HTTPS-First

An example of how these prefs can be set could be:

dom.security.https_first.exempt_domains.managed = a.example.com
(a.example.com is broken with HTTPS-First, so it is included by default in this pref)
dom.security.https_first.exempt_domains.user_overrides = -a.example.com,b.example.org
(The user does not want the default exemption, so they remove it again. They also want a exemption for b.example.org though, so they add it here)

---

It is also important to mention that there already is a mechanism for regular users to add exemptions via permissions that are completely unrelated to these prefs. They can be found in the HTTPS-Only section of the Privacy & Security settings, or in the site identity panel (lock icon) if there has been any interaction with HTTPS-Only/-First. These permissions can not easily be set though by default by us, and more importantly, they can not be updated quickly. This is why we want to add these prefs in addition to the current permission.

Comment 3

[Malte Jürgens [:maltejur]]

Mike, I was told that this is probably an interesting option for enterprise users, for example if any internal sites behave weirdly with HTTPS-Only/-First. I could imagine that it makes sense to add an enterprise policy which can also accept similar positive and negative overrides as the user_overrides pref. That policy could than be mapped to a locked third pref dom.security.https_only_mode.exempt_domains.enterprise_overrides.

Implementing that is probably best suited for a follow-up bug. But I already wanted to ask about your opinion now, because if you still have any fundamental concerns about these prefs, they can be best addressed now.

Comment 4

[Mike Kaply [:mkaply]]

Having the same things configured between prefs and permissions seems like a foot gun.

That being said, if I did this via policy, I would probably use the existing permissions mechanisms not the prefs (as it's much easier to manage adding to the permissions stuff with policy).

Comment 5

[Malte Jürgens [:maltejur]]

I see what you mean and had the same thought myself. The problem with permissions right now is that afaik we have no good way of adding "default" permissions for specific sites. But since you are also concerned about adding a extra pref, and I head a similar comment from pbz
on Slack, it may be a better idea to add some sort of Remote Settings capability for default permissions in the permissions manager. That would be more work than what would be required for the prefs, but it would also mean the same mechanism can be used for other permissions down the line.

If we would go that route, I could imagine that remote settings can always provide a list of default exceptions, and the user can then override those via the existing permissions ui (see attached mockup).

Comment 6

[Malte Jürgens [:maltejur]]

Attachment: Permission Manager Mockup

Comment 7

[Malte Jürgens [:maltejur]]

Attachment: Bug 1816449 - Allow https-only permissions to be set via remote settings in the permission manager r?pbz!,#permissions-reviewers!

Comment 8

[Malte Jürgens [:maltejur]]

Attachment: WIP: Bug 1816449 - Hide defaults perms behind a checkbox in permission manager settings UI

Comment 9

[Malte Jürgens [:maltejur]]

Attachment: Bug 1816449 - Part 8: Allow HTTPS-First exceptions to be set through the RemotePermissionService r?pbz!,#permissions-reviewers!

Comment 10

[Frederik Braun [:freddy]]

Is this a duplicate of bug 1919558?

It seems patches 1-6 and 8 went to bug 1919558 and patch 7 went here?

Comment 11

[Frederik Braun [:freddy]]

Oh never mind. This is about the HTTPS-First specific bit. Bug 1919558 is making that generic over all existing permissions, which is - by the way - really nice to have :)

Comment 12

[Malte Jürgens [:maltejur]]

Attachment: Bug 1816449 - Part 7: Add test for remote permission service r?pbz!,#permissions-reviewers!

Comment 13

[Phabricator Automation]

Comment on attachment 9429854 [details]
Bug 1816449 - Part 7: Add test for remote permission service r?pbz!,#permissions-reviewers!

Revision D225049 was moved to bug 1919558. Setting attachment 9429854 [details] to obsolete.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/69d226857a48
Part 8: Allow HTTPS-First exceptions to be set through the RemotePermissionService r=pbz,permissions-reviewers

Comment 15

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e8d842959e09
Backed out 8 changesets (bug 1816449, bug 1919558) for causing xpcshell and mochitest failures

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5f7eab80a48b
Part 8: Allow HTTPS-First exceptions to be set through the RemotePermissionService r=pbz,permissions-reviewers

Comment 17

[Atila Butkovits]

Backed out for causing failures at test_open_and_databases.js

Backout link: https://hg.mozilla.org/integration/autoland/rev/37f09713c73175d0a40718b24cf5a7e65f83d707

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=RteG7K48TNKMZA7Lp9hDUQ.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=5f7eab80a48b637274e1d991da3c51bbfbd4c82f

Failure log: https://treeherder.mozilla.org/logviewer?job_id=481837969&repo=autoland&lineNumber=3613

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a194f41588e9
Part 8: Allow HTTPS-First exceptions to be set through the RemotePermissionService r=pbz,permissions-reviewers

Comment 19

[Sandor Molnar[:smolnar]]

Backed out for causing mochitest failures @ test_refresh_firefox.py

Backout link: https://hg.mozilla.org/integration/autoland/rev/03586c970444deda68081dd83122b13ae535f66d

Push with failures

Failure log -> TEST-UNEXPECTED-ERROR | browser/components/migration/tests/marionette/test_refresh_firefox.py TestFirefoxRefresh.testFxANoSync

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e1c5bea6ace4
Part 8: Allow HTTPS-First exceptions to be set through the RemotePermissionService r=pbz,permissions-reviewers

Comment 21

[amarc]

https://hg.mozilla.org/mozilla-central/rev/e1c5bea6ace4