1923500 - Firefox 131 Enhanced Tracking Protection major breakage of multiple banks Bill Pay services

Status: RESOLVED FIXED

(Core :: Privacy: Anti-Tracking, defect, P1)

Description

[Noah (NoahSUMO/SolidSnake) [:Noah]]

It appears ever since Firefox 131 was released we're getting a lot of users in SUMO reporting they can't access the Bill Pay portion of their banks site.
Of course, you don't need me to tell you that's very BAD. The affected banks seem to include various small credit union banks and big established banks like Bank of America, TD Bank and PNC Bank. As the last time I broke news this big, I'm not happy imagining thousands of Firefox users hitting this bug, getting angry, whipping out the pitchforks and marching down to support.mozilla.org to let us know that something important they use everyday broke yet again. As an aside, I think we're going to have to look at our QA practices here. Although QAing for all big banks and some small credit unions isn't easy sadly.

I believe the mighty RyanVM stumbled upon this bug back on Sept 12 in Bug 1918483. But I couldn't tell if he was on release, beta or nightly when he hit the bug. And I have seen a few scattered reports about broken Bill Pay as early as of August this year. I believe those were on credit union sites.

My guess is that Enhanced Tracking Protection is blocking a script or cross-site cookie all of these banks use to conduct Bill Pay. Only 2 bank users were able to make Bill Pay work again by lowering the ETP to Standard. One of them had it on Custom mode prior. The rest claim changing ETP settings do nothing. Even totally disabling ETP and still having broken Bill Pay.

Shooting up the flares to the great Relman team. Get the word out to whoever can help with this. :)

SUMO reports

8-1-24; earliest report so far
I am unable to use my bank bill pay with Mozilla as of 8/1/2024
https://support.mozilla.org/questions/1455863

National Bank of Middlebury (Vermont) user who solved Bill Pay breaking by reducing ETP to Standard:
https://support.mozilla.org/questions/1455863#answer-1679337
https://cw411.checkfreeweb.com/imm/BrowserPolicy/NoCookiesEnabled/14726
https://nbmvt.com/convenient-banking/mobile-online-banking/bill-pay/

9-1-24 - Unspecified bank with broken Bill Pay:
https://support.mozilla.org/questions/1461518

Santander Bank user [solved by reducing ETP to Standard] (almost lost this user we've had since 2003!):
https://support.mozilla.org/questions/1467195#answer-1679287
"And, to whom it may concern, I solved this most recent issue, finally, by clearing my custom settings and using "STANDARD" because the solutions the bank offered, along with my 2 customizations still did not work.
And FTR I have written my diatribe about this to the bank as well as posting here. I WANT my FF back!"

PNC Bank user [solved by reducing ETP to Standard]:
https://support.mozilla.org/questions/1467900#answer-1680150

Bank of America Bill Pay reports (list keeps growing):
https://support.mozilla.org/questions/1468068
https://support.mozilla.org/questions/1467853
https://support.mozilla.org/questions/1467420
https://connect.mozilla.org/t5/discussions/bank-of-america-s-bill-pay-does-not-work/td-p/73544

TD Bank Bill Pay reports:
https://support.mozilla.org/questions/1467865
"Since the recent update to Firefox 131, I can no longer use the Billpay option of my TD Bank account. I am able to log into my account without any problems and use all of the options except for Billpay. I want to add that I know I am not alone with this problem as I have seen several others with the same problem but with different banks. It is the Billpay option which no longer works."
https://support.mozilla.org/en-US/questions/1455863#answer-1680293
"With all add-ons disabled, Enhanced Tracking completely off, with the cache and data cleared this results in the same message as I and apparently many others are having.
I have had this bank account for many years and it is just with FF 131 on Windows 11 that the problem has cropped up. Another user stated that he went back to 130 and all was fine until he again updated to 131 and the problem returned."

Thoughts:

I don't know about you all but I think we should backout the offending patch. RyanVM might be able to do a regression range hunt to see which Nightly first introduced the issue but I have a feeling someone is going to circle back and say its due to "network.cookie.cookieBehavior.optInPartitioning" (bug 1918483 comment #3). Which I have no idea what that is. Yet another cookie isolation feature?

Comment 1

[Ryan VanderMeulen [:RyanVM]]

My issue was on Nightly and the culprit was a change that isn't currently riding the trains. Let's start with Tim to get a sense of what might have changed during the 131 release cycle. Taking a quick look at the release notes, I wonder if CHIPS might be involved if online billpay systems are doing funny things with cookies across related-but-not-the-same domains?
https://developer.mozilla.org/en-US/docs/Web/Privacy/Privacy_sandbox/Partitioned_cookies

Comment 2

[Maire Reavy [:mreavy]]

I have a bankofamerica account (checking and credit card), and I can't reproduce this problem using Fx 131 (Release) or Fx 133 (Nightly) and a clean profile.

Comment 3

[Neha Kochar [:neha]]

I tried TD bill pay with Fx 131 release with ETP-Strict and without it but couldn't repro the problem.

Comment 4

[Neha Kochar [:neha]]

We enabled CHIPS in Fx131 Release - https://bugzilla.mozilla.org/show_bug.cgi?id=1908160 so this could be related as ryanvm pointed out. Tim is the right person to look into this.

Comment 5

[Dan Mehic]

Team is looking into this with priority.

Comment 6

[Noah (NoahSUMO/SolidSnake) [:Noah]]

I think you nailed it Ryan. It's gotta be related to CHIPS. I saw that in the 131 release notes but it was listed so far down the page that I didn't expect it to cause any huge issues.

Thanks Maire and Neha for testing using your own bank accounts! I thought for sure it would be reproducible for you. Is it possible this is a study only activated for some users so that's why you had different results? I'm curious what happens if either one of you sets the pref network.cookie.cookieBehavior.optInPartitioning to True then tries to visit your Bill Pay section.

Thanks to Neha and Dan who also registered on SUMO & Connect to answer questions about this. I was about to respond to those. I didn't earlier as I was too tired. Thanks for looking into this with priority!

Comment 7

[Noah (NoahSUMO/SolidSnake) [:Noah]]

Also there is a part of me that is very concerned about Enhanced Tracking Protection's filter lists. I have no idea where they are so I can't inspect them and attempt some type of manual review. My suspicion is that whoever is curating these ETP lists is adding too many domains that are necessary for some sites to work properly.

Are we still getting our lists from Disconnect.me? Because, as one example, I have found they are causing payment failures on payment sites due to blocking captcha domains (recaptcha.net & hcaptcha.com). So I'm worried their mistakes could spill over onto us and cause our userbase huge headaches in the oddest of places.

So I'm just curious how we develop/curate and fine-tune the ETP lists. And do we have a team dedicated to manually reviewing those lists from time to time? And lastly, if we share any lists with Disconnect? By that I mean, does Disconnect create the bulk of the filter list then we add our own blocking rules to it? Thanks to anyone who might know.

Comment 8

[Noah (NoahSUMO/SolidSnake) [:Noah]]

My spidey-sense was triggered by this report from October 7:
E-trade website does not load many a times
https://support.mozilla.org/questions/1467963
On the bottom left, I see "Read c.evidon.com" with the hourglass in the tab.

Older report from the previous year, June 2023
https://support.mozilla.org/questions/1416165
Firefox starting Thursday 6/15 would not connect with Etrade saying "Waiting for nexus - ensighten.com"

Any chance that ETP blocks either of these domains? I feel like it might be something banks use as well.

Comment 9

[Tim Huang[:timhuang]]

We, the Privacy team, are responsible for reviewing and deploying the lists from Disconnect. Since we don't have our own tracker list, we rely on Disconnect's tracker lists to block trackers in ETP strict mode and private browsing windows. We report to them if we find significant website breakages due to their list, but the final decision is theirs.

The ETP list is public and can be found in this repo. And people can test domains using the about:url-classifier page.

Disconnect does care about the website breakages caused by their tracking lists, so they are pretty cautious about the domains added to their lists. They did thorough technical reviews on domains to ensure they are truly trackers and do privacy-invasive things to users.

Comment 10

[Tim Huang[:timhuang]]

(In reply to Noah (NoahSUMO/SolidSnake) [:Noah] from comment #8)

Any chance that ETP blocks either of these domains? I feel like it might be something banks use as well.

The c.evidon.com is not on the list, but the ensighten.com is in the ad tracker list. This means that we block ensighten.com in ETP strict mode and private windows. In ETP standard, we still allow ensighten.com to load, but we restrict its cookie access in third-party context.

Comment 11

[Noah (NoahSUMO/SolidSnake) [:Noah]]

Thanks Tim for the great answers on ETP's tracker lists! I've always wanted to know. I do need to escalate that issue about Disconnect's addon breaking payment sites due to blocking captcha domains. (https://support.mozilla.org/questions/1455325#answer-1665517)

Thanks for checking those 2 domains. I'll keep hunting until I get something solid from the affected users.
I did get 1 more successful fix by having a user switch to ETP Standard for Bank of America:
https://support.mozilla.org/questions/1468253#answer-1680451

Fingers crossed. It looks like Dan may have gotten confirmation that CHIPS broke Bill Pay for those bank sites:
https://support.mozilla.org/questions/1467865#answer-1680453
"Thank you for the help. Flipping the pref network.cookie.CHIPS.enabled to false immediately solved my problem and I was able to get into Billpay."

Comment 12

[Tim Huang[:timhuang]]

FWIW, the Disconnect's addon might have a different blocking behavior than ours. The recaptcha.net is on the content tracker list. This means we don't do anything to this domain in ETP standard mode. In ETP strict mode, we only block third-party cookie access and allow the domain to load.

The hcaptcha.com is not on any lists.

Comment 13

[Tim Huang[:timhuang]]

Attachment: Bug 1923500 - Sending CHIPS cookies in non-default cookieBehaviors. r?bvandersloot!

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ea5a9992fc92
Sending CHIPS cookies in non-default cookieBehaviors. r=bvandersloot,cookie-reviewers,edgul

Comment 15

[Benjamin VanderSloot [:bvandersloot]]

Attachment: Bug 1923500 - Sending CHIPS cookies in non-default cookieBehaviors.

Original Revision: https://phabricator.services.mozilla.com/D225097

Comment 16

[Benjamin VanderSloot [:bvandersloot]]

Attachment: Bug 1923500 - Sending CHIPS cookies in non-default cookieBehaviors.

Original Revision: https://phabricator.services.mozilla.com/D225097

Comment 17

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: continued incident, bank bill pay breakage
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: minimal
• Explanation of risk level: this is an improvement over the existing known buggy code
• String changes made/needed: n/a
• Is Android affected?: no

Comment 18

[Phabricator Automation]

release Uplift Approval Request

• User impact if declined: continued incident, bank bill pay breakage
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: minimal
• Explanation of risk level: this is an improvement over the existing known buggy code
• String changes made/needed: n/a
• Is Android affected?: no

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1908160

Comment 20

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/ea5a9992fc92

Comment 21

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/796763b3eb4c

Comment 22

[Dianna Smith [:diannaS]]

:timhuang looks like the release patch requires a rebased patch due to conflicts with bug 1475599 that is only in 132.

Comment 23

[Tim Huang[:timhuang]]

I've rebased the release patch. Dianna, please try it again.

Comment 24

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/cd3358c3531d