1322113 - WebExtensions request does not contain cookies even for visited sites when 'from visited' setting is enabled, in Private Browsing mode

Status: RESOLVED FIXED

(Core :: Networking: Cookies, defect, P2)

Description

[Anatoly Sazanov]

Attachment: test_ext.zip

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0
Build ID: 20161201172143

Steps to reproduce:

- enable 'Always use private browsing mode' + 'Accept third-party cookies: from visited'
- load attached extension
- click browser action, it will take you to httpbin. Click again while you are on httpbin.


Actual results:

Cookie is not appended to requests from any WebExtension context (page/popup/background).
Cookie is there if you visit https://httpbin.org/cookies manually.
Cookie is appended if you select 'Accept third-party cookies: always'



Expected results:

Extension requests to page should contain cookies if they exist for this domain and page was visited.

Comment 2

[Anatoly Sazanov]

:kmag(In reply to Kris Maglione [:kmag] from comment #1)
> 
> *** This bug has been marked as a duplicate of bug 1319536 ***

I can still repro this bug in 53.0a1 (Win64 53.0.0.6198) with original scenario ('Only visited' option prevents cookies from being attached to background requests). Do I need to provide more details?

Comment 3

[Shane Caraveo (:mixedpuppy)]

I've confirmed that this report is still reproducible.  The test in bug 1319536 doesn't cover the 'Accept third-party cookies: from visited' use case.  

As well, when third party cookies is set to always, fetch is working but xhr does not work in some of the code paths in the test addon. 

Kris, thoughts on these two issues?

Comment 4

[Shane Caraveo (:mixedpuppy)]

Attachment: cookieBehavior

Patch to introduce test for 'Accept third-party cookies: from visited'

Comment 5

[The 8472]

There are some more limitations

* see bug 1295660 for referrer and origin header limitations of content-script XHR/fetch.
* background page XHR/fetch has no way to deal with containers or other origin attributes, only content scripts get the right principals, which requires a tab to be opened
* patching of the xray-wrappers[0] for xhr/fetch makes it difficult to access the page instances instead of the sandbox ones


[0] http://searchfox.org/mozilla-central/rev/30fcf167af036aeddf322de44a2fadd370acfd2f/toolkit/components/extensions/ExtensionContent.jsm#384-386

Comment 6

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

I am also getting this issue under similar circumstances. I have third party cookies set to "never", and a WebExtension that has permissions set for the target origin. When making fetch(targeturl, { mode: "cors", credentials: "include" }) requests from a background script however, only a session cookie is applied to the request, not the remaining cookies containing the auth tokens. It works fine when visiting the page in a browser tab. This happens to me when making requests to addons.mozilla.org, where the auth token cookie is also for addons.mozilla.org.

Let me know if this is better suited for a separate bug, but it seems there are a few cases regarding cookie settings that fit together well so I'm just leaving a comment for now.

Comment 7

[Andreas Wagner [:TheOne] [use NI]]

This breaks SDK add-ons using Downloads.jsm as well.

Comment 8

[Andreas Wagner [:TheOne] [use NI]]

51 and 52 are affected too.

Comment 9

[Shane Caraveo (:mixedpuppy)]

While I see the behavior from comment 5, I don't think they apply to the specific test in this bug (ie. the test addon attached by OP).  In this case, the origin/referrer have no affect as the test site just reflects those.  The issue is solely the cookie setting all vs. visited, where no cookies are sent if the preference is visited.

I also dont think this is a webextension bug, it feels lower level, perhaps something on the channel where cookies are added to the request.

Comment 10

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

(In reply to Shane Caraveo (:mixedpuppy) from comment #9)
> I also dont think this is a webextension bug, it feels lower level, perhaps
> something on the channel where cookies are added to the request.
Who can investigate this? It looks like  I would really like to set my third party cookie settings back to "Never", where this bug also occurs. Looking at current P1's it seems Kris already has a pretty full plate, and if this is a platform bug maybe we can get some help from other teams.

Comment 11

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Looking at bug 674397 which I randomly found the platform behavior is probably correct, but WebExtensions with sufficient permissions should be able to get around this limitation.

Comment 12

[Shane Caraveo (:mixedpuppy)]

I don't think webextensions should work around how the platform is supposed to work for cookies without the proper justification for that.  I would like clarification on whether "from visited" is working correctly.  Assuming it is, then we need to decide if webextensions should behave differently.  In any case, I think that is still going to be platform changes, not something in the webextensions layer.

Comment 13

[Andreas Wagner [:TheOne] [use NI]]

My SDK add-on strips credentials in XHRs to addons.mozilla.org if "from visited" is set, even though I obviously visit that website a lot.

Comment 15

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Shouldn't we move this to Core XHR if it can only be fixed there?

Comment 16

[Julien Cristau [:jcristau] (back April 22)]

Mass wontfix for bugs affecting firefox 52.

Comment 17

[Tim Guan-tin Chien [:timdream] (please needinfo)]

I've been trying to see if this bug can be fixed soon, but I understand there are other priorities based on affected add-ons.

To workaround this bug, user may add the target host in the cookie exception list, which set Firefox to always allow the cookies from the said origin.

Comment 18

[Steve Sobel]

> I don't think webextensions should work around how the platform is supposed to work for cookies without the proper justification for that.

I can respect this position, but as an extension developer it is problematic.

If the purpose of WebExtensions is, in part, to make seamless development of extensions that work across browsers, then this is basically an achilles heel for a lot of website-specific extensions. Chrome behaves this way and has drastically more users, thus leverage.

Reddit Enhancement Suite, for example, is completely crippled by this bug. Having to explain to roughly 250,000 users that they need to add a cookies exception is not an easy task especially given the varying level of technical experience/comfort that exists in a group that large.  At the very least, an interface for developers to explain this to users or build in a permissions list for 3rd party to the manifest (and yes, go ahead and prompt the user and explain about it on install) would go a long way towards making addons work seamlessly.

I'd really like to see some sort of a better solution than a wontfix. Optimally, I'd like it to "just work", but I can understand Mozilla's privacy-centric concerns here and think that at the very least, extension developers should have some means of getting users around this problem that's better than responding to loads of individual bug reports.

Thanks for your consideration.

Comment 19

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Grammarly is also affected by this. These hostnames need to be in the exception for the extension to work:

https://capi.grammarly.com/
https://auth.grammarly.com/
https://data.grammarly.com/
https://f-log-extension.grammarly.io/
https://app.grammarly.com/
https://www.grammarly.com/

Comment 21

[Liz Henry (:lizzard) (relman/hg->git project)]

I just got a question about this from a user on irc. Sounds like from the priority flag this is possible for the 57 timeframe, but not certain that we can get to it. If not, then maybe for 58.

Comment 22

[nightpool]

What's the current status of this? It's really important for any extension that enhances or hooks into a site's API to make requests on the user's behalf. We've had tons of users get tripped up by this since we've moved to WebExtensions and functionality-wise this is definitely a regression from the Addon SDK. Like RES, New XKit is entirely built around this behavior and wouldn't function without the ability to make 1st-party requests on the user's behalf.

To clarify, this is the policy we think makes sense here: "Content scripts that are embedded *on* a origin, should be able to make requests *as* that origin". I don't think this is "[working] around how the platform is supposed to work for cookies".

Comment 23

[Andy McKay]

If I understand the STR and comments correctly, this doesn't need a WebExtension to recreate, so based on comment 12 and the other comments moving over to Core. If this is the intended behavior from this configuration then we should probably document this.

Comment 24

[irmin.okic]

Since the status of this is "status-firefox59: --- → fix-optional", has this been fixed in version 59? If not are there any plans for bugfix versions to include the fix?

Comment 25

[ExE Boss]

Since this is different behaviour from Chrome, I’m adding it as a blocker to bug 1161828.

Comment 26

[cruelplatypus67]

Any update on this? some extensions depend on this you know.

Comment 27

[The 8472]

(In reply to cruelplatypus67 from comment #26)
> Any update on this? some extensions depend on this you know.

A solution has been implemented. You can find the relevant documentation at
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Content_scripts#XHR_and_Fetch

Comment 28

[The 8472]

Sorry. Forget my previous reply. I was confusing this issue with another one.

Comment 29

[Shane Caraveo (:mixedpuppy)]

This has probably sat untouched since it remained assigned (to me) when moved to core:networking. I'd like to get input from that team and see if we can move this forward.

Selena, can you help find someone to poke at this?

Comment 30

[Selena Deckelmann :selenamarie :selena]

:miket, :ddurst -- Can you both weigh in on the prioritization of this? From my viewpoint the end state of this seems confusing for a user in private browsing mode, but it appears that the Chrome extensions API supports it, making it a potentially ambitious webcompat/add-ons crossover.

Comment 31

[Selena Deckelmann :selenamarie :selena]

Arthur - interested in your take on this as well.

Comment 32

[David Durst [:ddurst]]

If the OP still holds (manually visiting the site appends the cookies, but requests from an extension context do not append the cookies), then I think we should P2 this, as the reasonable assertion is that the content blocking settings are respected, even when in private browsing mode. [Or, to put it another way, that content blocking wouldn't be different in some contexts when in private browsing mode.]

Comment 33

[Mike Taylor [:miketaylr]]

From the perspective of add-ons interop (which might lead people to think pages are broken), it seems reasonable to match Chrome here.

Comment 34

[Arthur Edelstein [:arthur]]

I agree with David and Mike that fixing this to match Chrome and meet the expectation of users seems the right approach.

Comment 35

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 36

[Dragana Damjanovic [:dragana]]

Can you take a look?

Comment 37

[(no longer active)]

I can definitely take a look. The first step for me is to understand what the broken behaviour is. I'm a bit stuck on that for now...

Shane, I'm going off of your test case in attachment 8827265 [details] [diff] [review]. In that test, I do not see where the bug is. We open http://mochi.test:8888/..../file_sample.html, it tries to set a cookie using document.cookie, that fails because cookieBehavior == 3 (since there are no existing cookies from mochi.test) and then the rest of the test fails.

Can you please clarify what is the exact set of steps that are expected to work, and why do we expect them to work? Thanks!

Comment 38

[(no longer active)]

Oh wait, I think I misunderstood the bug originally. This bug actually isn't related to web extensions at all. The implementation of the "from visited" cookie policy is just broken under private browsing mode it seems! It was just noticed with WebExtensions...

Comment 39

[(no longer active)]

Attachment: Bug 1322113 - Fix the implementation of the 'from visited' cookie policy in private browsing mode;

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8e3be55481b9
Fix the implementation of the 'from visited' cookie policy in private browsing mode; r=baku

Comment 41

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/8e3be55481b9