1619623 - First-party cookies are deleted on daily clean up

Status: VERIFIED FIXED

(Core :: Privacy: Anti-Tracking, defect, P1)

Description

[Simona Badau, Desktop QA]

Attachment: 2020-03-03_16h30_56.gif

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0
Build ID: 20200303095030

Prerequisites

• Go to about:config and set the preference "privacy.purge_trackers.enabled" to true.
• Make sure that in the Settings of the Web Console, "Enabled browser chrome and add-on debugging toolboxes" is checked.

Steps to reproduce

1. Open Firefox.
2. Navigate to one or two popular websites to create some cookies and site storage.
3. Go to about:preferences#privacy and observe the cookies from the "Manage Cookies and Site Data" dialog.
4. Open the Browser Console, type and enter `Services.obs.notifyObservers(null, "idle-daily");' to send the idle-daily notification manually.
5. Check again the cookies from the "Manage Cookies and Site Data" dialog.

Expected result

• In step 3, I see several cookies from the Disconnect Tracking Protection list:
  www.instagram.com
  www.google.com
  google.com
  google.ro
  instagram.com
  doubleclick.net
• In step 5, the third-party cookies from the Disconnect Tracking Protection list should be cleared.

Actual result

• In step 5, some of the first-party (like www.instagram.com or www.google.com) cookies are also cleared. Please note that the issue is intermittent, and several attempts might be required to see this behavior. If the issue is not reproducible from the first attempt, please repeat steps 4 and 5.
  For more details, please see the attached screencast.

Comment 1

[Johann Hofmann [:johannh]]

There may be a misunderstanding about what constitutes "first party" here, first party means that you have visited that site directly, e.g. when you enter "google.com" in your URL bar.

With that in mind, can you provide a bit more detailed STR? i.e what sites did you visit exactly?

Thanks :)

Comment 2

[Simona Badau, Desktop QA]

(In reply to Johann Hofmann [:johannh] from comment #1)

There may be a misunderstanding about what constitutes "first party" here, first party means that you have visited that site directly, e.g. when you enter "google.com" in your URL bar.

With that in mind, can you provide a bit more detailed STR? i.e what sites did you visit exactly?

Thanks :)

Yes, sure.

1. Open Firefox
2. Open a new tab and do a search (I personally searched for: "cameron and lauren hamilton from love is blind cheat")
   -> a Google search is performed at this point - https://www.google.com/search?client=firefox-b-d&q=cameron+and+lauren+hamilton+from+love+is+blind+cheat
3. Click on one of the results from www.cheatsheet.com
   -> https://www.cheatsheet.com/entertainment/love-is-blind-the-parts-of-lauren-and-camerons-love-story-you-didnt-see.html/ is opened
4. Scroll down and click on the Instagram photo
   -> a new Instagram tab is opened - https://www.instagram.com/need4lspeed/?utm_source=ig_embed
5. Close all the pages opened in the previous steps and check the "Manage Cookies and Site Data" dialog.
   -> cookies from the Disconnect Tracking Protection list are created:
   www.instagram.com
   www.google.com
   google.com
   google.ro
   instagram.com
   doubleclick.net
6. Open the Browser Console, type and enter `Services.obs.notifyObservers(null, "idle-daily");' to send the idle-daily notification manually.
7. Check the "Manage Cookies and Site Data" dialog again.
   -> doubleclick.net is cleared (no doubt this is a third-cookie)
   -> google.com, google.ro, instagram.com are also cleared - not sure how these cookies are treated since I did enter on Google (step 2) and Instagram (step 4)
   -> www.instagram.com and www.google.com remain in the manage data dialog (first-party cookies). Please note that I encountered several situations in wich one of these cookies gets cleared.
   Unfortunately, this is an intermittent issue and I'm not sure what triggered it. But, if I repeated step 6 - for sure one of the 2 first-party cookies that remained is deleted and it shouldn't since the command `Services.obs.notifyObservers(null, "idle-daily");' only sends the idle-daily notification manually (and the first-party cookie should only be cleared after 30 days with no user interaction).

Johann, if the steps I added are still not clear, please see the gif that I added in the description for this scenario. :)

Comment 3

[Simona Badau, Desktop QA]

I just encountered another scenario where the issue is reproducible without having to call `Services.obs.notifyObservers(null, "idle-daily");' a second time.

Steps to reproduce:

1. Open Firefox and set the preference "privacy.purge_trackers.enabled" to "true"
2. Open a new tab and navigate to https://www.amazon.com/, browse a little trough the products.
3. Close all the pages opened in the previous steps and check the "Manage Cookies and Site Data" dialog.
   -> The following cookies from the Disconnect Tracking Protection can be seen:
   www.amazon.com
   amazon.com
4. Open the Browser Console, type and enter `Services.obs.notifyObservers(null, "idle-daily");' to send the idle-daily notification manually.
5. Check the "Manage Cookies and Site Data" dialog again.
   -> www.amazon.com and amazon.com cookies are no longer on the list.

Comment 4

[Johann Hofmann [:johannh]]

Thanks for clarifying, and you're right, this is exposing two interesting flaws and is definitely blocking turning on purging on Nightly:

Visiting https://www.amazon.com it's setting cookies for www.amazon.com and .amazon.com, but we only set the user interaction permission for https://www.amazon.com. When the purging code tries to do the permission check we test by origin, meaning that cookies for .amazon.com are deleted. Unfortunately telling the permission manager to not use exactHostMatch doesn't work because we're trying to match a non-subdomain entry against a subdomain entry instead of the other way around, which doesn't work.

This is bad and we should probably fix it. We have a few options, I guess:

1. We could iterate through all permissions and test with hasRootDomain, but that sounds a bit slow if the user has lots of permissions (which isn't unlikely).
2. We could update the nsPermissionManager exactHostMatch check to include a way to check for subdomain permissions when being passed the base principal, but it probably can't be the default behavior of exactHostMatch so that could get complicated.
3. We could always store storageAccessAPI permissions on the base domain, i.e. strip sub-domains at creation time, which to me feels like it wouldn't break any existing consumers if we make sure to pass the exactHostMatch flag, but it's a bit of a risk of course and we would close avenues for more fine-grained checks of this permission in the future.
4. We could invent a new permission type storageAccessAPIBaseDomain or something like that, which explicitly only stores base domains.

I'm not particularly excited about any of these. Ehsan, do you have any thoughts?

Oh, I forgot: Why is the cookie for www.amazon.com also deleted even though there's an interaction permission for it? That's because the clear data service uses option 1) and clears permissions by base domain, causing subsequent permission checks for www.amazon.com to fail if .amazon.com was cleared first 🤦‍♀️

Comment 5

[(no longer active)]

(In reply to Johann Hofmann [:johannh] from comment #4)

Thanks for clarifying, and you're right, this is exposing two interesting flaws and is definitely blocking turning on purging on Nightly:

Visiting https://www.amazon.com it's setting cookies for www.amazon.com and .amazon.com, but we only set the user interaction permission for https://www.amazon.com. When the purging code tries to do the permission check we test by origin, meaning that cookies for .amazon.com are deleted. Unfortunately telling the permission manager to not use exactHostMatch doesn't work because we're trying to match a non-subdomain entry against a subdomain entry instead of the other way around, which doesn't work.

Sigh.

This is bad and we should probably fix it. We have a few options, I guess:

1. We could iterate through all permissions and test with hasRootDomain, but that sounds a bit slow if the user has lots of permissions (which isn't unlikely).

I think this may not be too bad. Firstly I don't think performance should be a major goal here, since this whole task is running in the background in an iterative way. If we find that something here is taking too much time we can break down the work into more granular chunks. But more importantly I think we'd need to iterate through the permissions only once to collect the corresponding host names into an array or something an then iterate through that array to perform our hasRootDomain checks.

Is there any reason not to go with this option?

2. We could update the nsPermissionManager exactHostMatch check to include a way to check for subdomain permissions when being passed the base principal, but it probably can't be the default behavior of exactHostMatch so that could get complicated.

Hmm, not sure how we would do this... Do you mean by enumerating all permissions or something like that? For example if you asked for https://amazon.com we could have permissions store for https://www.amazon.com, https://foo.amazon.com, https://really.long.origin.name.amazon.com, etc.

3. We could always store storageAccessAPI permissions on the base domain, i.e. strip sub-domains at creation time, which to me feels like it wouldn't break any existing consumers if we make sure to pass the exactHostMatch flag, but it's a bit of a risk of course and we would close avenues for more fine-grained checks of this permission in the future.

That would make us forget which host the user has actually interacted with, no? But more importantly, users already have these permissions in their profiles, even if we want to go this way we still need a solution to correctly deal with old data found in profiles.

4. We could invent a new permission type storageAccessAPIBaseDomain or something like that, which explicitly only stores base domains.

Ugh, that would be unfortunate...

I'm not particularly excited about any of these. Ehsan, do you have any thoughts?

Oh, I forgot: Why is the cookie for www.amazon.com also deleted even though there's an interaction permission for it? That's because the clear data service uses option 1) and clears permissions by base domain, causing subsequent permission checks for www.amazon.com to fail if .amazon.com was cleared first 🤦‍♀️

Hmm, that actually looks like a bug to me. I'm not quite sure why we delete permissions from non-exact hosts in deleteByHost whereas for other data which we delete by host it seems like we delete it from exact hosts (e.g. cookies). So it seems to me that we should probably switch that code to delete permissions from exact hosts only?

Comment 6

[Johann Hofmann [:johannh]]

Yeah, I agree, thinking about it more iterating over permissions is probably not such a bad idea. Let's do that then.

Hmm, that actually looks like a bug to me. I'm not quite sure why we delete permissions from non-exact hosts in deleteByHost whereas for other data which we delete by host it seems like we delete it from exact hosts (e.g. cookies). So it seems to me that we should probably switch that code to delete permissions from exact hosts only?

Yeah, it feels a little weird. Maybe worth a separate bug, though.

Comment 7

[Ethan Tseng [:ethan]]

Cookie purging is pref'd off on Firefox 75.

Comment 8

[Johann Hofmann [:johannh]]

Attachment: Bug 1619623 - Don't purge cookies with interaction permission for sites with the same base domain. r=ewright

Depends on D67947

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e1672b3231e9
Don't purge cookies with interaction permission for sites with the same base domain. r=ewright

Comment 10

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/e1672b3231e9

Comment 11

[Simona Badau, Desktop QA]

I can still reproduce this issue, first-party cookies are still deleted on daily clean-up.

These are the steps that I use:

1. Open Firefox and set the preference "privacy.purge_trackers.enabled" to "true"
2. Restart Firefox
3. Open a new tab and navigate to https://www.amazon.com/
4. Open a new tab and navigate to https://www.youtube.com/
5. Open a new tab and navigate to https://www.instagram.com/
6. Open a new tab and navigate to https://www.facebook.com/
7. Close all the tabs opened in sets 3-7
8. Check the "Manage Cookies and Site Data" dialog
9. Open the Browser Console, type and enter `Services.obs.notifyObservers(null, "idle-daily");' to send the idle-daily notification manually
10. Check the "Manage Cookies and Site Data" dialog again.

Actual results

• the first party cookies: www.amazon.com, amazon.com, and youtube.com are no longer in the Manage Data site dialog.
  For more details, please see the screencast.
  https://drive.google.com/file/d/1V80QVwKyn9b1f7JWVV07nEf3n_V_OP2a/view?usp=sharing

Johann, should I re-open this bug or file a new one?

Also, is it intended to get "undefined" in the browser console when running the snippet `Services.obs.notifyObservers(null, "idle-daily");'? Previous this fix we were getting a message like:
The Browser Console is opened and the command runs properly:
*** PurgeTrackerService: Purging trackers enabled, beginning batch.
*** PurgeTrackerService: Batch finished, queueing next batch.
*** PurgeTrackerService: Purging trackers enabled, beginning batch.
*** PurgeTrackerService: All cookie purging finished, resetting list until tomorrow.

Comment 12

[Johann Hofmann [:johannh]]

You don't seem to interact with those pages. Please click on something on amazon.com and try to reproduce this again :)

Also, is it intended to get "undefined" in the browser console when running the snippet `Services.obs.notifyObservers(null, "idle-daily");'? Previous this fix we were getting a message like:

You need to set privacy.purge_trackers.logging.level to all and restart the browser to make the logs appear. We'll try to improve that in the future.

Comment 13

[Simona Badau, Desktop QA]

(In reply to Johann Hofmann [:johannh] from comment #12)

You don't seem to interact with those pages. Please click on something on amazon.com and try to reproduce this again :)

Indeed, If I interact with the pages, the cookies are not deleted.

Also, is it intended to get "undefined" in the browser console when running the snippet `Services.obs.notifyObservers(null, "idle-daily");'? Previous this fix we were getting a message like:

You need to set privacy.purge_trackers.logging.level to all and restart the browser to make the logs appear. We'll try to improve that in the future.

Great. Thanks!

Comment 14

[Johann Hofmann [:johannh]]

Awesome, thanks for confirming!

Comment 15

[Simona Badau, Desktop QA]

I can confirm this issue is fixed on the latest Nightly 77.0a1, verified on Mac OS X 10.15, Windows 10 x64 and Ubuntu 18.04 x64.

Comment 16

[Simona Badau, Desktop QA]

Johann,

Considering the Purging Tracking Cookies feature is disabled on Firefox 76, can the tracking flag to firefox76 be set to disabled?

Thanks!

Comment 17

[Johann Hofmann [:johannh]]

Yes, thanks