1515913 - Allowed Cookies from the Exceptions list are not saved after restart

Status: RESOLVED FIXED

(Toolkit :: Data Sanitization, defect, P2)

Description

[Rares Doghi, Desktop QA]

[Affected versions]
Firefox 66.0a1

[Affected platforms]
Ubuntu 16.04, Windows 7/10, Mac OS 10.13

[Steps to reproduce]:
1. Open the Firefox browser and reach the Manage Cookies and Site Data panel in Options.
2. Check the Manage Data panel in order to see the existing cookies.
3. Click the Manage Permissions button and add http://www.bbc.com to the ALLOW list.
4. Mark the Delete Cookies and Site Data when Nightly is closed option as checked.
5. Reach bbc.com and Verify the Cookies Sub Panel from the Site information panel.
6. Check the Manage Data panel from Options in order to see the existing cookies stored from bbc.com.
7. Restart the Browser.


Expected Result:
When the user checks the "Delete Cookies and Site data when Nightly is closed" option, all cookies should be deleted except the ones from the domains added to the Exceptions list.

Actual Results:
All the Cookies are deleted, regardless of the domains added to the exceptions list.

Please see the Following recording for more details: https://streamable.com/ped5t

This issue was added because of Bug 1503217

Comment 1

[Andrea Marchesini [:baku]]

A couple of questions:

1. have you tried to avoid restarting using about:profiles? Asking because I cannot reproduce the issue when closing firefox normally.

2. many things has changed in the last month. Can you check again if the new cookie policy are involved here?

3. Have you tried to add https://www.bbc.com in the allow list? It should not change too much.

I cannot reproduce the issue with the latest nightly. Let me know if you can. Thanks!

Comment 2

[Rares Doghi, Desktop QA]

Hi, First things first this issue occurs regardless of how you restart the browser, I can reproduce it with a fresh profile every time on MAC as well as Windows.

2nd I'm not sure what do you mean by check the new cookie policy, I think it's still the same policy as before where its OK to delete all cookies from Manage Data as well as the cookies from Site X (allowed) as long as the "Delete cookies and site data when Nightly is closed" option is checked.

I think those permissions are simply a Gateway for the cookies to come in or not, as long as you have a website added to the whitelist with the "allowed" status, the cookies from that website will always come through but it does not mean that they are saved and stored forever even with the "Delete cookies and site data when Nightly is closed" option checked.

3rd I did add the https, as well as http domains to the allow list and the cookies are all deleted after restart. This issue if it is an issue still occurs in the latest nightly.

Please see the newly attached video.

Comment 3

[Rares Doghi, Desktop QA]

Attachment: cookies.mp4

Comment 5

[Johann Hofmann [:johannh]]

This has been reported again in bug 1524916, needinfo because we should really figure out how to fix it.

Comment 6

[Brad R]

Just for the record, I'll note my reproducible case (same as bug 1524916, but spelled out better). Confirmed broken in FF 65.0.1 (working in FF 64.0.2):

Generally, the ability to use the exception table as a cookie whitelist, to delete-on-exit all cookies except those from particular origin(s), is no longer available.

Config:

• network.cookie.cookieBehavior: 4
• network.cookie.lifetimePolicy: 2 (expire at end of session)
• privacy.clearOnShutdown.cookies: false (do not wipe all cookies on exit, use lifetime policy and exception table)

In about:config, "Exceptions - Cookies and Site Data"

• https://accounts.google.com ALLOW

To reproduce:

• Log in to Google, using an account that requires 2FA, and tick the checkbox "Don't ask again on this computer"
• (notice that a cookie for accounts.google.com has been saved)
• Exit and restart Firefox.
• (notice that the cookie for accounts.google.com no longer exists)
• Log in to Google again, notice that Google asks again for 2FA.

Expected behavior:

• On exit, cookies with origin at or below accounts.google.com will be preserved according to their server lifetime policy (i.e. we expect ALLOW to be equal to an override to lifetimePolicy=0, as in FF 64)
• On exit, all other cookies are expired, and removed prior to subsequent startup.

Actual behavior:

• The cookies we explicitly requested to ALLOW are unexpectedly deleted prior to subsequent startup.
• In this particular use case, Google now requires 2FA on every login attempt.

Workaround:

• Downgrade FF to 64.0.2. [If anyone can suggest a workaround for FF 65, I'd love to hear it!]

Comment 7

[Andrea Marchesini [:baku]]

The issue is this:

1. we have cookies from accounts.google.com and from www.google.com. We should delete only the www.google.com's ones.

2. Sanitizer.jsm uses the ClearDataService to clean up the corresponding principal:
   https://searchfox.org/mozilla-central/rev/fbb251448feb7276f9b1d0a88f9c0cb1cd144ce4/browser/modules/Sanitizer.jsm#905

3. nsICookieManager doesn't support the remove by principal and ClearSiteService uses removeCookiesWithOriginAttributes() instead: https://searchfox.org/mozilla-central/rev/fbb251448feb7276f9b1d0a88f9c0cb1cd144ce4/toolkit/components/cleardata/ClearDataService.jsm#43

4. nsCookieService::RemoveCookiesWithOriginAttributes uses the basedomain to remove cookies:
   https://searchfox.org/mozilla-central/rev/fbb251448feb7276f9b1d0a88f9c0cb1cd144ce4/netwerk/cookie/nsCookieService.cpp#4718-4722

5. This means that it calls nsCookieService::RemoveCookiesWithOriginAttributes() with 'google.com' and accounts.google.com and www.google.com do have the same base domain:
   https://searchfox.org/mozilla-central/rev/fbb251448feb7276f9b1d0a88f9c0cb1cd144ce4/netwerk/cookie/nsCookieService.cpp#4686-4689

6. We end up removing accounts.google.com cookies.

ehsan, is there any particular reason to remove by basedomain instead of by host or by principal?

Comment 8

[(no longer active)]

Looks like this call site may depend on this: https://searchfox.org/mozilla-central/rev/b2d35912da5b2acecb0274eb113777d344ffb75e/browser/components/enterprisepolicies/Policies.jsm#220. This was added in bug 1440578, it is unclear to me what exact behaviour this code wants to depend on but I hesitate to change it.

Perhaps Felipe knows?

Can we add support for clearing by principal to nsICookieManager (in step 3)?

Comment 9

[:Felipe Gomes (needinfo for replies!)]

If I go to Cookies and Site Data > Manage Permissions and add "https://google.com" to the block list, will that also block accounts.google.com? What about if I used "https://www.google.com", is it any different?

The desired behavior is to do a one-time clear of anything that that input would block.

Comment 10

[(no longer active)]

(In reply to :Felipe Gomes (needinfo me!) from comment #9)

If I go to Cookies and Site Data > Manage Permissions and add "https://google.com" to the block list, will that also block accounts.google.com?

Yes.

What about if I used "https://www.google.com", is it any different?

No, if you added www.google.com to the block list, accounts.google.com wouldn't get blocked.

The desired behavior is to do a one-time clear of anything that that input would block.

So I think based on this, we should preserve the existing behaviour of the code. Thanks!

Comment 11

[wyoghe0gkzamotk]

tl;dr: When when network.cookie.lifetimePolicy=2, the subdomain (mail.protonmail.com)'s cookies get deleted even though both the subdomain and the domain (protonmail.com) are in the exception list (both as http and https)

With Firefox nightly (r465920+.3d5cd10cb1b2+-1) compiled locally,
selected "Delete cookies and site data when Nightly is closed" (aka about:config network.cookie.lifetimePolicy=2 )
I add only two "Allow" (not "Allow Session") hosts:
protonmail.com
mail.protonmail.com
(that adds 4 items in the list: the http and https version of the above)
Now, I log into protonmail, check the cookies(host and number of cookies from Preferences - page needs reload), then restart browser (but without having the protonmail website in focus, so it won't get loaded on startup), check the cookies and I see ONLY the protonmail.com cookies kept!
In other words, the mail.protonmail.com cookies are gone, the protonmail.com ones are kept after browser restart.
To fix this, I've to set network.cookie.lifetimePolicy to 3 or 0. Now both are kept.

However, something else (or protonmail's fault?) must be at hand, because I cannot keep the protonmail session logged in even with the above fix, after a browser restart, the previous mail.protonmail.com cookies are seemingly ignored and I'm presented with protonmail login screen and thus 2 more cookies get created each time, looking at the cookie count for mail.protonmail.com in preferences. But, this is probably a different issue, and I've tried fiddling with other settings to no avail, but I don't even remember which and what tens of combinations I've tried. What's more, even though everything else works in protonmail, I cannot open/read any emails and there are no errors on console! But this is unlikely related to any cookies, imho, must be something else I've set in about:config .

Comment 12

[wyoghe0gkzamotk]

tl;dr: this comment is unrelated to the current issue, and I'm confirming that the quoted part below(which is present in my previous comment) is also unrelated.

(In reply to wyoghe0gkzamotk from comment #11)

However, something else (or protonmail's fault?) must be at hand, because I cannot keep the protonmail session logged in even with the above fix, after a browser restart, the previous mail.protonmail.com cookies are seemingly ignored and I'm presented with protonmail login screen and thus 2 more cookies get created each time, looking at the cookie count for mail.protonmail.com in preferences. But, this is probably a different issue, and I've tried fiddling with other settings to no avail, but I don't even remember which and what tens of combinations I've tried. What's more, even though everything else works in protonmail, I cannot open/read any emails and there are no errors on console! But this is unlikely related to any cookies, imho, must be something else I've set in about:config .

Regarding this specific part only: it's protonmail's fault: their support team confirmed that protonmail purposefully discards the active session upon browser close, for security reasons. I'm guessing the cookies are marked session only. This came as a surprise to me because I misunderstood what they were trying to say with their note in (their) Settings-Security : "Unless you explicitly logout or change your password, sessions can last for up to 6 months. Sessions expire after 2 weeks of inactivity." But I guess they were only talking about their server-side sessions tracking; the client side(browser) cuts the cord upon browser shutdown (and restart too? unless restart temporarily keeps active session cookies? is there even a way to Restart firefox?) so any such sessions that the server keeps track of are then useless afterwards.

Comment 13

[(no longer active)]

baku, based on comment 7 do you have enough information to take this? Thanks!

Comment 14

[Andrea Marchesini [:baku]]

Attachment: Bug 1515913 - Implement nsICookieService.removeCookiesFromRootDomain, r?ehsan

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2fea4528362e
Implement nsICookieService.removeCookiesFromRootDomain, r=Ehsan

Comment 16

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/2fea4528362e

Comment 17

[u636583]

I've tested
changeset: 469606:a83cab75b00d
tag: tip
parent: 469591:be3f40425b52
parent: 469605:3bd02d619749
date: Tue Apr 16 07:05:54 2019 +0300
summary: Merge inbound to mozilla-central a=merge

which does seem to have changeset: 469478:2fea4528362e as seen by hg log

However, I'm losing 2 of the 8 cookies(so 6 cookies left after browser restart) on bugzilla.mozilla.org and I'm thus logged out,
and losing all cookies on bugzilla.redhat.com (thus I'm logged out of there too)
but, no cookies are lost on sourceware.org and bugs.webkit.org and thus I'm still logged in on these two.

This, of course, happens when Delete cookies and site data when Nightly is closed is selected, with the following Manage Permissions list (typed in below manually by me - in case you see any typos):

http://bugs.webkit.org  Allow
https://bugzilla.redhat.com  Allow
http://bugzilla.redhat.com  Allow
https://sourceware.org  Allow
http://sourceware.org  Allow
https://bugs.webkit.org  Allow
https://bugzilla.mozilla.org  Allow
http://bugzilla.mozilla.org  Allow
https://github.com  Allow for Session
http://github.com  Allow for Session
http://youtube.com  Block
http://instagram.com  Block
https://www.instagram.com  Block
https://youtube.com  Block
http://www.youtube.com  Block
https://instagram.com  Block
http://www.instagram.com  Block

When Delete cookies and site data when Nightly is closed isn't selected, I'm still logged in on b.m.o(and redhat) and there are 8 b.m.o cookies after restart(prior to focusing the b.m.o. tab for loading, as always)

So, unless I'm doing it wrong(tm), this issue isn't fixed? Any advice?

Comment 18

[u636583]

1. rename or delete /usr/lib/firefox/browser/defaults/preferences/vendor.js
2. firefox -P
3. make new profile, [Start Nightly]
4. go to: about:preferences#privacy then click button [Manage Data] under heading Cookies and Site Data
   you see addons.mozilla.org 5 cookies Last Used 29 seconds ago
   (note that for me addons.mozilla.org cannot be resolved to anything using any DNS eg. ping-ing it won't work either)
   press [Cancel] button
5. close the tab https://www.mozilla.org/privacy/firefox/
   (for me it failed to load due to www.mozilla.org not resolving DNS-wise)
6. load these each in a new tab:

https://bugzilla.mozilla.org
https://bugzilla.redhat.com
https://bugs.webkit.org
https://github.com/

7. after they fully loaded, close all 4 tabs opened in step 6
8. refresh the first tab which is about:preferences#privacy (refresh needed to see updated list)
   click button [Manage Data]
   you see:
   github.com, 7 cookies, 310 bytes storage, last used: 27 seconds ago
   addons.mozilla.org, 5 cookies, , last used: 2 minutes ago
   bugzilla.mozilla.org, 2 cookies, , last used: 44 seconds ago
   bugs.webkit.org, 1 cookies,, last used: 37 seconds ago
   bugzilla.redhat.com, 1 cookies, last used: 41 seconds ago
   click [Cancel]
9. press Ctrl+Q to quit browser
10. start browser with the same profile as before via firefox -P
11. go to about:preferences#privacy again
12. click button [Manage Data]
    you see:
    github.com 5 cookies(yes 2 cookies are gone), 310 bytes Storage, Last Used: 2 minutes ago
    addons.mozilla.org, 5 cookies, , Last Used: 24 seconds ago (yes it gets updated on every browser start!)
    click [Cancel]
13. go to about:preferences#general in the same tab, select [✓]Restore Previous session (it was previously deselected)
14. repeat from step 6,
    on step 8 you see the same,
    step 11 is already open for you (due to auto restored),
    on step 12 you see all 5 sites with Last Used: 4 seconds ago
    and github.com has 7 cookies, not 5.
    click [Cancel]
15. click [Manage Permissions]
Allow:
    bugzilla.mozilla.org
bugzilla.redhat.com
bugs.webkit.org
github.com
    click [Save Changes]
    select [✓]Delete cookies and site data when Nightly is closed (it was previously deselected)
16. Ctrl+Q to quit browser
17. do step 10 through 12, you see:
    github.com, 5 cookies, 310 bytes storage, last used: 4 minutes ago (not updated)
    addons.mozilla.org, 5 cookies,, last used 3 seconds ago! (yup, updated)
    and nothing else!! (step 15 had no effect?)
    click [Cancel]
18. deselect []Restore previous session
19. do step 6 through 12 after which you'll see the same list as in step 12 or 17,

I can't make heads or tails of what's going on...

Comment 19

[u636583]

Attachment: screenshots.zip

(while [✓]Restore Previous session is selected)
I'm still losing my login session on some(but not all) sites in the exception list when [✓]Delete cookies and site data when Nightly is closed is selected.

tested:
changeset: 473543:d51f3432e142
date: Sat May 11 18:01:50 2019 +0000
repo: https://hg.mozilla.org/mozilla-central

See screenshots.

I lost my login session on these sites:
bugzilla.mozilla.org
bugzilla.xfce.org
www.phoronix.com

and was kept logged on on these sites:
bugzilla.kernel.org
wiki.archlinux.org
aur.archlinux.org
bugs.archlinux.org

(not all sites were tested, only the ones listed above)

All logins were kept, previously, when [ ]Delete cookies and site data when Nightly is closed was NOT selected. (which is what I normally use to workaround this issue)

Comment 20

[Rares Doghi, Desktop QA]

Hi Guys, This issue still occurs in Fx 68.0a1 (2019-05-14), I tried adding and allowing the Websites from Comment 18, https://bugzilla.mozilla.org
https://bugzilla.redhat.com
https://bugs.webkit.org
https://github.com/

then after reaching each one in a different tab the cookies were allowed but if I check the Delete cookies and site data when Nightly is closed option they are all deleted after restart except the 2 cookies from github and the default mozilla.org ones, Im not sure how to proceed with this issue, what is the correct behavior is but the main issue logged here still happens.

Comment 22

[Liz Henry (:lizzard) (relman/hg->git project)]

baku, can you look at this again since it seems to still be happening?

Comment 23

[nevenr]

Hi,

I use Firefox 67.0.4 (64-bit) on windows 10 and it works.

I mean cookies and site data survive Firefox restart for sites that are in exception list with status allow even when option "Delete cookies and site data when Firefox is closed" is turned on.

It is a bit strange that this is fixed now although you stated "status-firefox67: affected → wontfix"

Regards,
N.

Comment 24

[Brad R]

I'm running 67.0.4 on MacOs 10.14.5.

The test case in comment #6 still fails: When the whitelist is exactly ALLOW https://accounts.google.com, then on exit, all cookies (including those for accounts.google.com) are deleted. i.e. the bug is still present.

But if I ALLOW https://google.com, all cookies are deleted except those for google.com and all its subdomains — this is the expected/desired behavior.

So it seems half-fixed. It appears to still fail for entries that are deeper than TLD+1 origins.

Comment 25

[Andrea Marchesini [:baku]]

I'm unable to reproduce this issue. Can you tell me if the STR is different than this:

0. new profile.
1. Enable "Delete cookies and site data when Firefox is closed"
2. Set the exceptions for https://bugzilla.mozilla.org - ALLOW
3. visit the site https://bugzilla.mozilla.org and do the login (I use mozilla oauth - coud it be that I cannot reproduce it because of this?)
4. restart the browser.

Note that if you do 4 between 1 and 2 things can behave differently. I'm going to change this in a separate bug.

Do you think is it possible to have a test case? Something that is 100% reproducible but without all the complexity of a real website.

Comment 26

[Rares Doghi, Desktop QA]

Hi Andrea, I used your steps only I added a few more websites to the Allowed list and accessed them, you can see in the attached image the number of cookies i had before and after restart.

https://bugzilla.mozilla.org
https://bugzilla.redhat.com
https://bugs.webkit.org
https://github.com/

These are the websites I added at step 2 and visited on step 3 before the restart.

Please see the attached image.

Comment 27

[Rares Doghi, Desktop QA]

Attachment: Cookiess.png

Comment 28

[Liz Henry (:lizzard) (relman/hg->git project)]

Likely too late for a patch in 69, but we could still take one for 70.

Comment 29

[Johann Hofmann [:johannh]]

I think I fixed the remaining cases in bug 1571234, I'll try to add a test once that lands.

Comment 30

[Johann Hofmann [:johannh]]

Attachment: Bug 1515913 - Update sub-domain sanitize test to use SiteDataTestUtils and add additional test case. r=baku

The new test covers the edge case described here: https://bugzilla.mozilla.org/show_bug.cgi?id=1515913#c6

This was fixed earlier in bug 1571234.

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/22d3da2568c6
Update sub-domain sanitize test to use SiteDataTestUtils and add additional test case. r=baku

Comment 32

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/22d3da2568c6

Comment 33

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/fbc3c116cb12

Comment 34

[Ryan VanderMeulen [:RyanVM]]

If we want to include the new test coverage with the uplift of bug 1571234 to ESR68, it'll need a rebased patch due to some non-trivial conflicts in browser_cookiePermission_subDomains.js.

Comment 35

[Johann Hofmann [:johannh]]

I thought I would get to this but I'm just busy right now and will be away for a while soon. Giving up on this. I think we'll be good without that test on ESR. Thanks and sorry!