1540810 - Tracking protection breaks Fastly support (can break any site using Zendesk)

Status: RESOLVED WORKSFORME

(Web Compatibility :: Site Reports, defect, P2)

Description

[Ryan Feeley [:rfeeley]]

Visiting this page with TP on Standard results in a message that says "Cookies must be enabled in your browser to sign in. Click here to enable them."

https://support.invisionapp.com/hc/en-us/signin

I sent this via Report a Problem but apparently it's not checked?

Comment 1

[(no longer active)]

Adding invisionapp.zendesk.com to urlclassifier.trackingAnnotationSkipURLs will fix this breakage.

Comment 3

[(no longer active)]

There are kind of bigger problems with the zendesk embedded login module in my testing. When I try logging in using the link in comment 0, after entering my credentials the main page tries to embed an iframe pointing to https://invisionapp.zendesk.com/access/login, but that page is served with a x-frame-options: SAMEORIGIN header which prevents it from being embedded in the first place.

I believe this is more of a web compat bug than anything else really.

Comment 4

[Steven Englehardt [:englehardt]]

Stpeter to look into this.

Comment 5

[Peter Saint-Andre [:stpeter]]

Follow-up in progress.

Comment 6

[Peter Saint-Andre [:stpeter]]

Zendesk support pointed me to https://support.zendesk.com/hc/en-us/articles/203657536-Embedding-Zendesk-into-an-iframe-is-not-allowed - which seems to imply that perhaps Invision is incorrectly embedding zendesk content into an iframe?

Comment 8

[(no longer active)]

(In reply to Peter Saint-Andre [:stpeter] from comment #6)

Zendesk support pointed me to https://support.zendesk.com/hc/en-us/articles/203657536-Embedding-Zendesk-into-an-iframe-is-not-allowed - which seems to imply that perhaps Invision is incorrectly embedding zendesk content into an iframe?

Now that we have two example sites that do this, could we reach out to Zendesk and tell them that this response is not a useful response when sites clearly do not abide by their recommendations? Thanks!

Comment 9

[Julien Cristau [:jcristau]]

(In reply to :ehsan akhgari from comment #8)

(In reply to Peter Saint-Andre [:stpeter] from comment #6)

Zendesk support pointed me to https://support.zendesk.com/hc/en-us/articles/203657536-Embedding-Zendesk-into-an-iframe-is-not-allowed - which seems to imply that perhaps Invision is incorrectly embedding zendesk content into an iframe?

Now that we have two example sites that do this, could we reach out to Zendesk and tell them that this response is not a useful response when sites clearly do not abide by their recommendations? Thanks!

Especially since this seems to happen on sites that zendesk themselves host? (support.fastly.com CNAME fastly.zendesk.com)

Comment 10

[(no longer active)]

This doesn't happen on InVision any more...

Comment 11

[(no longer active)]

(In reply to Julien Cristau [:jcristau] from comment #9)

Especially since this seems to happen on sites that zendesk themselves host? (support.fastly.com CNAME fastly.zendesk.com)

Hmm, baku, Steven, do you think it would make sense to have a special heuristic in ETP to relax storage restrictions when the top-level origin's domain is a CNAME record pointing to the third-party tracker's domain, assuming that the top-level domain has, in that case, relinquished the control of running their website to the third-party tracker?

Comment 12

[(no longer active)]

Attachment: Bug 1540810 - Part 1: Consider CNAME eTLD+1 matches as first-parties when performing third-partiness checks

Comment 13

[Steven Englehardt [:englehardt]]

(In reply to :ehsan akhgari from comment #11)

(In reply to Julien Cristau [:jcristau] from comment #9)

Especially since this seems to happen on sites that zendesk themselves host? (support.fastly.com CNAME fastly.zendesk.com)

Hmm, baku, Steven, do you think it would make sense to have a special heuristic in ETP to relax storage restrictions when the top-level origin's domain is a CNAME record pointing to the third-party tracker's domain, assuming that the top-level domain has, in that case, relinquished the control of running their website to the third-party tracker?

Sorry it's taken me so long to respond that you already have a prototype patch. It's an interesting direction that we should discuss further. I'm concerned that, as implemented, this heuristic enables some attacks on ETP.

Consider that a site may maliciously set up CNAME records for the purpose of relaxing restrictions with various third parties. In theory, they can CNAME one subdomain for each partner. So tracker1.news.example CNAME'ed to news.tracker1.example, tracker2.news.example CNAMED'ed to news.tracker2.example, and so on. Then, when a user first visits (or on navigations) they do a quick top-level redirect:

• news.example
• --> tracker1.news.example
• --> tracker2.news.example
• --> tracker3.news.example
• --> ...
• --> trackerN.news.example
• --> news.example

On each step in the redirect they sync news.example's first-party cookie with the tracker's third-party cookie (allowed by the heuristic) and then later just share data associated with news.example's "first-party" cookie (for which every partner how has a mapping).

Note that this attack is currently possible without the heuristic, but just requires top-level redirects to third-party redirects. E.g.,

• news.example
• --> tracker1.example/cookie_sync.html?targets=tracker2.example,trackerN.example,news.example&uid=12345
• --> tracker2.example/cookie_sync.html?targets=trackerN.example,news.example&uid=12345
• --> ...
• --> trackerN.example/cookie_sync.html?targets=news.example&uid=12345
• --> news.example

Where uid is news.example's ID for the user. This requires the third parties to cooperate and not mess with each other's data, but you can also imagine a scenario where the redirects go back to news.example with each step. This type of workaround can be handled in the same way that WebKit handles it: https://webkit.org/blog/8311/intelligent-tracking-prevention-2-0/.

If we add such a heuristic, do we expect a webkit-style mitigation to work? If we see a top-level redirect through a first-party origin that is CNAME'ed to a known tracker, do we purge the state for the first party? Will that lead to more breakage?

Comment 14

[(no longer active)]

Hmm, let's first ensure that we're thinking of the same heuristic, since I suspect that what I'm proposing may be slightly different than what you have in mind. Here is the scenario in my proposal:

• top.example (first party)
  ** foo.tracker.example (third party)

When resolving top.example the browser notices that top.example resolves to a CNAME record pointing to bar.tracker.example. This is commonly used e.g. when top.example has handed off the task of hosting this website to bar.tracker.example.

The heuristic that I'm proposing is that under this situation, we'd consider the iframe as first-party.

Is that also what you had in mind?

Comment 15

[Steven Englehardt [:englehardt]]

Ehsan: yes it is, assuming by "we'd consider the iframe as first-party." you mean that foo.tracker.example has access to tracker.example's first-party cookie jar.

So in my example attack: top.example embeds an iframe from foo.tracker.example?uid=12345, where uid is the user ID assigned to that user by top.example. foo.tracker.example loads with access to its full cookie jar, and makes the link between its tracking cookie and the ID in the uid parameter. Normally, foo.tracker.example couldn't do this because it wouldn't have access to its own cookie jar. Then top.example reports back to tracker.example with uid for all future tracking-related events, and tracker.example happily continues to collect data that it can associate with its own cross-site identifier.

Comment 16

[(no longer active)]

(In reply to Steven Englehardt [:englehardt] from comment #15)

Ehsan: yes it is, assuming by "we'd consider the iframe as first-party." you mean that foo.tracker.example has access to tracker.example's first-party cookie jar.

No, I mean that for anti-tracking checks (or elsewhere in the browser where we determine whether content is third-party), we'd consider it as first-party. The cookie jar that the content access is determined based on the storage principal of its document.

So in my example attack: top.example embeds an iframe from foo.tracker.example?uid=12345, where uid is the user ID assigned to that user by top.example. foo.tracker.example loads with access to its full cookie jar, and makes the link between its tracking cookie and the ID in the uid parameter. Normally, foo.tracker.example couldn't do this because it wouldn't have access to its own cookie jar. Then top.example reports back to tracker.example with uid for all future tracking-related events, and tracker.example happily continues to collect data that it can associate with its own cross-site identifier.

But yes, I think I understand how this attack scenario could work.

I don't really think we should implement this heuristic without the abuse prevention mechanisms built-in, so let's try to see if we can figure that out. I think purging the storage of the first-party could be a reasonable mitigation. I'm wondering about one thing though, do we need to worry about storage mechanisms besides cookies here? It seems to me that the attack scenario you demonstrated would only work because cookies aren't origin bound, and any origin bound storage backend shouldn't be similarly affected?

Comment 17

[Steven Englehardt [:englehardt]]

(In reply to :ehsan akhgari from comment #16)

(In reply to Steven Englehardt [:englehardt] from comment #15)

Ehsan: yes it is, assuming by "we'd consider the iframe as first-party." you mean that foo.tracker.example has access to tracker.example's first-party cookie jar.

No, I mean that for anti-tracking checks (or elsewhere in the browser where we determine whether content is third-party), we'd consider it as first-party. The cookie jar that the content access is determined based on the storage principal of its document.

Hmm yeah, re-reading my comment I don't know what I was getting at. Your description is what I meant.

So in my example attack: top.example embeds an iframe from foo.tracker.example?uid=12345, where uid is the user ID assigned to that user by top.example. foo.tracker.example loads with access to its full cookie jar, and makes the link between its tracking cookie and the ID in the uid parameter. Normally, foo.tracker.example couldn't do this because it wouldn't have access to its own cookie jar. Then top.example reports back to tracker.example with uid for all future tracking-related events, and tracker.example happily continues to collect data that it can associate with its own cross-site identifier.

But yes, I think I understand how this attack scenario could work.

I don't really think we should implement this heuristic without the abuse prevention mechanisms built-in, so let's try to see if we can figure that out. I think purging the storage of the first-party could be a reasonable mitigation. I'm wondering about one thing though, do we need to worry about storage mechanisms besides cookies here? It seems to me that the attack scenario you demonstrated would only work because cookies aren't origin bound, and any origin bound storage backend shouldn't be similarly affected?

No, I think we need to purge all storage for top.example and tracker.example.

Let's think through the constraints:

1. The first party will necessarily have to CNAME a subdomain of their main domain. They could CNAME top.example, but then they could sync with at most one third-party before needing to use subdomains and would then be handing over hosting of their base site to the tracker.
2. Everything except the first party's cookies are scoped to the subdomains. So foo.top.example has different DOM storage than bar.top.example. Cookies are associated with top.example. Let's say they have a cookie named uid scoped to top.example which includes a unique identifier.
3. The CNAME record count point either to a subdomain of the tracker (e.g., foo.tracker.example) or the base domain (e.g., tracker.example). The tracker can carry out the attack described in Comment 15 in either case.

With that in mind, the first party could prime all of their "partner" subdomains with uid by loading each subdomain in an iframe when the user first visits top.example. E.g., top.example includes the following iframes:

• tracker1.top.example?uid=12345
• tracker2.top.example?uid=12345
• and so on.
  Each iframe has code to read uid from the query string of document.location.href and pushes it down into DOM storage. Now, tracker1.top.example has a localStorage item with key uid and value 12345 (same for tracker2.top.example and so on).

The first party carries out the redirect attack described in Comment 13. Now tracker.example associates its first-party tracking cookie (tracker_id=XYZ scoped to tracker.example) with uid=12345 which the tracker knows corresponds to their partner top.example. tracker_id=XYZ is associated with a large number of IDs other publishers similar to top.example.

Now, we detect the redirects and purge ONLY cookies for top.example and for tracker.example. The next time the users visits top.example, the publisher again embeds iframes from tracker1.top.example and so on. These iframes postMessage back up the uid=12345 value out of DOM storage and tracker.example sets it back as a first-party cookie scoped to tracker.example. It can now repeat the attacks in Comment 13 and Comment 15 to re-establish the identifier with tracker.example. So now its new tracking cookie tracker_id=ABC is linked to uid=12345.

The beauty of this attack is that the tracker doesn't need to re-sync with all partners, only top.example. Once it has made that connection it can transitively re-link all of its previous partner IDs. i.e., tracker_id=ABC == uid=12345 for top.example --> tracker_id=ABC == tracker_id=XYZ (using the link with uid=12345 established before the purge) --> connections to all past partners.

Comment 18

[(no longer active)]

Status update: I started to brush the dust off from my patches here, and realized that the recent work on document channel has completely broken my patches in the mean time (they still work if document channel is preffed off.) I've been working on fixing things up again since yesterday...

Comment 19

[(no longer active)]

(In reply to Steven Englehardt [:englehardt] from comment #17)

The beauty of this attack is that the tracker doesn't need to re-sync with all partners, only top.example. Once it has made that connection it can transitively re-link all of its previous partner IDs. i.e., tracker_id=ABC == uid=12345 for top.example --> tracker_id=ABC == tracker_id=XYZ (using the link with uid=12345 established before the purge) --> connections to all past partners.

Yes, you're right about this attack.

I think the most reliable way to prevent this attack from happening is to partition 3rd party storage/communication when choosing to grant storage access based on this heuristic. For example, using the names used in comment 13, when the user is browsing news.example, the cookie jar granted to tracker1.news.example (CNAME'd to news.tracker1.example) will be tied to the news.example top-level domain. In the attack scenario described, when the attacker redirects the top-level context from news.example to tracker1.news.example, the tracker1.news.example top-level page will be unable to see the partitioned cookie jar of this domain under news.example, which breaks the attack scenario.

I have a heuristic implemented based on this idea, cleaning up the patches to post for review...

Comment 20

[(no longer active)]

Attachment: Bug 1540810 - Part 1: Add a cookie settings API for partitioning third-party content;

Comment 21

[(no longer active)]

Attachment: Bug 1540810 - Part 2: Add an AntiTrackingCommon API for notifying a document that third-party content is now partitioned from its HTTP channel,

This API would allow us to notify a generic channel object about the
fact that the storage for third-party content should be partitioned
dynamically from now on, and communicate this with the document the
channel belongs to, irrespective of whether the notification is emitted
from the parent or the content process.

This will ensure that future channels created from the browsing context
to which this document belongs will honour the partitioned storage
semantics.

Comment 22

[(no longer active)]

Attachment: Bug 1540810 - Part 3: Add a Necko API for retrieving the canonical host name of channels, and keeping tracking of the top-level channel's canonical host name;

These Necko APIs can be used for keeping track of the top-level
document's canonical host name (CNAME), as queried from the DNS
subsystem. Additionally it will provide access to the canonical host
name of the channel URI, which will be used in future anti-tracking
interventions (see bug 1592727 for example).

For the heuristic implemented in this bug, we only really need the
top-level document's canonical host name information.

Comment 23

[(no longer active)]

Attachment: Bug 1540810 - Part 4: Promote third-party sub-resouces from domains that are CNAME'd to be same-eTLD+1 as the top-level document's origin to be considered as first-party sub-resources;

This is the implementation of the main heuristic that fixes the breakage
reported in this bug. With parts 1-4 applied, we support successful
login to ZenDesk support forums.

Note that this heuristic is landing disabled by default on late
beta/release builds, and will be turned on after further testing in a
future bug.

Comment 24

[(no longer active)]

Attachment: Bug 1540810 - Part 5: Partition 3rd party storage when the canonical host name heuristic was used;

This patch makes the heuristic added in the previous part safe to
deploy, by ensuring that the storage access granted is always a
partitioned storage grant, tied to the top-level document's eTLD+1.

This uses the dynamic FPI infrastructure to dynamically turn on FPI only
for the third-party subresources that the heuristic is being applied to.

Comment 25

[(no longer active)]

Attachment: Bug 1540810 - Part 6: Add a test to verify that storage access is successfully granted using the heuristic added in part 4, and it is a partitioned storage access based on the heuristic in part 5;

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d2df12c9c496
Part 1: Add a cookie settings API for partitioning third-party content; r=baku

Comment 27

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/d2df12c9c496

Comment 28

[Steven Englehardt [:englehardt]]

Zendesk is still broken on https://support.leanplum.com/hc/en-us/signin?return_to=https%3A%2F%2Fsupport.leanplum.com%2Fhc%2Fen-us%2Frequests. If you attempt to log in with ETP you will receive an X-Frame-Options error. Note that Zendesk has also started using the window.open heuristic to get cookie access (see Bug 1636289), but users who ignore their warning will still see an error.

Comment 29

[Steven Englehardt [:englehardt]]

Attachment: Screenshot from 2020-08-21 16-52-00.png

It looks like Zendesk has adopted the Storage Access API, but only on Safari. Our implementation is similar (and in fact, it's less restrictive), so I'm hopeful they can support Firefox as well. Check out this help page and the comments (posting the archived version since it doesn't look like this page's content is static).

If I spoof a Safari user agent string in Firefox I do see the Safari experience (shown in attachment).

Initially the "Continue" link includes the following click handler:

function onclick(event) {
  window.open("/auth/v2/login/set_cookie", "_blank")
}

That opens a tab from zendesk.com that says "Recent versions of Safari have new cookie requirements to improve security. To continue, you must allow cookies." and another "Continue" button. When the user clicks "Continue" the tab is closed and they return to the original tab. This is required in Safari because an origin that wants to call the Storage Access API must have received user interaction as a first party in the past. Firefox does not have such a requirement.

Back in the original tab, the "Continue" button now has the following click handler:

function(e) {
  document.requestStorageAccess().then(function() {
    location.reload(), console.log("request storage access GRANTED iframe reloaded")
  }, function() {
    console.log("request storage access fails")
  }), e.preventDefault()
}

In Firefox, the call to requestStorageAccess() will succeed automatically if the third-party origin (zendesk.com in this case) has received access on less than 5 sites. Otherwise, the user will be shown a prompt and the call will succeed or fail depending on the user's actions.

Unfortunately the flow doesn't work in Firefox with a spoofed user agent string because the "Sign in" button remains grayed out. Thus, a webcompat intervention probably isn't possible here. Zendesk could add support for Firefox by skipping the awkward zendesk.com new tab first step, and instead just using second click handler from the start.

Comment 30

[Steven Englehardt [:englehardt]]

Since there are Zendesk folks actively engaging with the discussion thread on that help page I've posted a comment. Hopefully we'll have better luck than we did with our other outreach attempts.

Comment 31

[Mike Taylor [:miketaylr]]

Hi Steven, given we've got another instance of this bug in Bug 1658257, is there anything else we can do here? It seems like this is something that could affect retention -- zendesk is used pretty heavily (especially in enterprise scenarios).

Comment 32

[Steven Englehardt [:englehardt]]

(In reply to Mike Taylor [:miketaylr] from comment #31)

Hi Steven, given we've got another instance of this bug in Bug 1658257, is there anything else we can do here? It seems like this is something that could affect retention -- zendesk is used pretty heavily (especially in enterprise scenarios).

One thing to note is that this breakage should only be seen when the "Level 2" blocklist is active, which is currently only in pre-release channels.

A zendesk product manager responded to my outreach about a week ago and said:

Hi Steven,

Thanks so much for reaching out! The team is investigating the Storage Access API for Firefox and Edge in the coming weeks so we'll definitely reach out if we run into anything unexpected with Firefox. Much appreciated. 

Cheers, Caroline

I'm not sure there's anything more we can do.

Comment 33

[Oana Arbuzov [:oanaarbuzov]]

The issue still occurs on my side.

Tested:
[Fixed] URL: https://support.invisionapp.com/hc/en-us/signin
[Reproducible] URL: https://support.imaginecurve.com/hc/en-gb/signin?mobile_site=true&return_to=https%3A%2F%2Fsupport.imaginecurve.com%2Fhc%2Fen-gb%3Fmobile_site%3Dtrue%26return_to%3D%252Fhc%252Frequests
[Reproducible] URL: https://enterprise-help.mozilla.com/access/unauthenticated

Tested with:
Browser / Version: Firefox Nightly 210128 (🦎 87.0a1-20210127093943)
Operating System: Samsung Galaxy S8 (Android 9) - 1440 x 2960 pixels, 18.5:9 ratio (~570 ppi density), Huawei P20 Lite (Android 8.0.0) - 1080 x 2280 pixels, 19:9 ratio (~432 ppi density)

Comment 34

[Oana Arbuzov [:oanaarbuzov]]

The issue still occurs on my side.

Tested:
[Fixed] URL: https://support.invisionapp.com/hc/en-us/signin
[Reproducible] URL: https://support.imaginecurve.com/hc/en-gb/signin?mobile_site=true&return_to=https%3A%2F%2Fsupport.imaginecurve.com%2Fhc%2Fen-gb%3Fmobile_site%3Dtrue%26return_to%3D%252Fhc%252Frequests
[Reproducible] URL: https://enterprise-help.mozilla.com/access/unauthenticated

Tested with:
Browser / Version: Firefox Nightly 210128 (🦎 87.0a1-20210127093943)
Operating System: Samsung Galaxy S8 (Android 9) - 1440 x 2960 pixels, 18.5:9 ratio (~570 ppi density), Huawei P20 Lite (Android 8.0.0) - 1080 x 2280 pixels, 19:9 ratio (~432 ppi density)

Comment 35

[Johann Hofmann [:johannh]]

I can't confirm, these sites work fine for me. Can you elaborate on what's going wrong for you?

Comment 36

[Oana Arbuzov [:oanaarbuzov]]

I get the following behavior:

• For URL: https://support.invisionapp.com/hc/en-us/signin I can dismiss "Cookies" popup (tapping "Got it" button), proceed to login and successfully sign in with ETP - Strict enabled.

• For URL: https://support.imaginecurve.com/hc/en-gb/signin?mobile_site=true&return_to=https%3A%2F%2Fsupport.imaginecurve.com%2Fhc%2Fen-gb%3Fmobile_site%3Dtrue%26return_to%3D%252Fhc%252Frequests, I can't dismiss "Cookies" popup by tapping "Continue" button with ETP - Strict or Standard enabled.
  Thus even if I type in the email & password I'm not able to sign in ("Sign in" is disabled).
  https://prnt.sc/yq5mpk
  Disabling ETP, I'm able to sign in.
  https://prnt.sc/yq6g52

• For https://enterprise-help.mozilla.com/access/unauthenticated I get the same behavior as above, no way to sign in with ETP - Strict or Standard.

Tested with:
Browser / Version: Firefox Nightly 210207 (🦎 87.0a1-20210203093146)
Operating System: Samsung Galaxy S8 (Android 9) - 1440 x 2960 pixels, 18.5:9 ratio (~570 ppi density), Huawei P20 Lite (Android 8.0.0) - 1080 x 2280 pixels, 19:9 ratio (~432 ppi density)

Comment 37

[Johann Hofmann [:johannh]]

Ooh yeah you're on Android, that's bug 1543720 :(

Comment 38

[Johann Hofmann [:johannh]]

If you try it on desktop (Windows/OSX/Linux) it should work fine.