1909980 - Duo 2fa hits "403 forbidden" if your system clock is wrong, only in Firefox

Status: VERIFIED FIXED

(Core :: Networking: Cookies, defect, P3)

Description

[Daniel Holbert [:dholbert]]

STR (requires a MoCo LDAP account, or probably another Duo-enabled account would work)

1. Set your system clock to 10 hours in the future. (So for me it's 1pm locally; I set my clock to 11pm.)
2. Visit https://sso.mozilla.com/ in a fresh Firefox user profile (or in a private browsing window)
3. Type in your LDAP credentials and submit the form.

EXPECTED RESULTS:
I should be redirected to a working Duo 2fa landing page, sending a notification to my phone.

ACTUAL RESULTS:
I land on a Duo URL that just says 403 forbidden. If I look in my network log, I can also see a 403 response for a POST action that failed.

Chrome gives EXPECTED RESULTS.
Firefox gives ACTUAL RESULTS.

I happened to hit this today after rebooting from Ubuntu to Windows, which often does funny things with your system clock, if you haven't yet done the right workaround dance (I haven't yet).

I was stumped at the cause for a while until I landed on https://support.swarthmore.edu/support/solutions/articles/14000050405 which mentions checking your system clock. But googling for "duo 403" finds a handful of other reports where folks are talking about Duo specifically hitting 403 errors on Firefox, with Chrome/Edge working just fine (as is my experience). Some of those reports are probably folks hitting this exact issue, where Firefox is affected by an incorrect system clock in a way that Chromium apparently avoids.

Comment 1

[Daniel Holbert [:dholbert]]

Attachment: screenshot of 403 forbidden duo landing page

Comment 2

[Daniel Holbert [:dholbert]]

Note that this essentially prevents folks from using their 2fa-protected account (e.g. all Mozilla sites, in my case) until they realize that they need to fix their system clock.

Comment 3

[Daniel Holbert [:dholbert]]

I hit this in Firefox 130.0a1 nightly, but it also affects Firefox 128 release as well, BTW. I can reproduce on Windows as well as Ubuntu 22.04.

Looking at network devtools when perfomring the STR at the correct time vs. with system clock 10 hours in the future, the relevant difference happens with two requests to:

https://api-4b043da5.duosecurity.com/frame/frameless/v4/auth?sid=frameless-[...]

There's a GET request, followed by a POST request, to a URL like that. Looking at the headers for those requests, the cookie is substantially shorter when I do this with my system clock set to a time in the future!!

When I perform the STR with an incorrectly-configured system clock, both the GET and POST requests have a cookie header like this:

'Cookie: trc|...'

...where "..." is a 62-character mostly-capital-alphanumeric string (including a | and an =).

When I perform the STR with a correctly-configured system clock, both the GET and POST request have much longer cookie header like this:

'Cookie: sid|A="B=|C|D"; _xsrf|E="F=|G|H"; trc|I=J'

Here, each capital letter represents a UUID or another multi-character string that I've ellided in case it's got my own sensitive auth data. The whole header (including the word "Cookie") is 358 characters long.

Notably, there's a trc| section at the very end of this long cookie; that seems to match what I had above for the incorrect-system-clock scenario. That trc cookie snippet is the only thing that gets sent when I perform the STR with a clock configured to be in the future.

Comment 4

[Daniel Holbert [:dholbert]]

With my system clock 10h in the future, Chrome still somehow ends up sending a "long"/good cookie, like the lower one in comment 3, which is why this Just Works in Chrome (on Windows as well as Ubuntu, the platforms I've tested this on).

So, as a guess at a diagnosis here:

• Maybe Duo is setting a cookie where part of the cookie has some sort of short validity period (less than 10 hours, at least)
• and Firefox is incorrectly discarding that cookie immediately (or not including it on the request for whatever reason), because we think that the cookie is stale (which would maybe be accurate if 10 hours had actually elapsed, but they haven't elapsed)
• ...Whereas Chrome is either ignoring the cookie expiration, or noting that its validity-duration hasn't yet expired (at least from a time-delta perspective, if not an absolute-clock-time perspective)

I do see that Firefox and Chrome are both picking up my incorrect-system-clock via e.g. new Date() in the JS console, for what it's worth. So I think we can mostly-rule out the possibility of Chrome using some other sort of time-source separate from my system clock.

(I don't know a lot about how cookies & their lifetimes are managed, so I'm hand-waving a bit here.)

baku, could you take a look here perhaps? Skimming commit logs, I see you've been doing some cookie-age-related work recently in e.g. bug 1906914. :) (not that it would have caused this, since this isn't a regression -- but rather just that you've maybe got the relevant code/features paged into your head.)

Comment 5

[Andrea Marchesini [:baku]]

Chrome computes the expiration time by adding the delta between the local and server times. If your computer is 10 hours in the future, the cookie expiration time will be increased by 10 hours. As far as I know, this is not part of the cookie spec (rfc6265bis), but we should implement the same logic.

Comment 6

[Andrea Marchesini [:baku]]

Attachment: Bug 1909980 - Compare the cookie `expires` attribute with the server time, r?edgul

Comment 7

[Andrea Marchesini [:baku]]

Attachment: data review request

Comment 8

[[e-mail]]

Comment on attachment 9419209 [details]
data review request

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
​
Yes.
​
Is there a control mechanism that allows the user to turn the data collection on and off?
​
Yes. This collection can be controlled through the product's preferences.
​
If the request is for permanent data collection, is there someone who will monitor the data over time?
​
No. This collection will expire in firefox 132.
​
Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
​
Category 1, Technical.
​
Is the data collection request for default-on or default-off?
​
Default on for all channels.
​
Does the instrumentation include the addition of any new identifiers?
​
No.
​
Is the data collection covered by the existing Firefox privacy notice?
​
Yes.
​
Does the data collection use a third-party collection tool?
​
No.
​

Result: datareview+

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0a3cd5d9f229
Compare the cookie `expires` attribute with the server time, r=edgul,cookie-reviewers,valentin

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/606b39b7dbfe
Compare the cookie `expires` attribute with the server time, r=edgul,cookie-reviewers,valentin

Comment 11

[tszentpeteri]

Backed out for causing mochitest and wd failures related to cookies.

• Backout link
• Push with failures
• Failure Log for wd
• Failure Log for mochitest

[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - TEST-PASS | netwerk/cookie/test/mochitest/test_server_time.html | undefined assertion name 
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - Buffered messages finished
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - TEST-UNEXPECTED-FAIL | netwerk/cookie/test/mochitest/test_server_time.html | undefined assertion name - got "", expected "test2=foo"
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - SimpleTest.is@https://example.org/tests/SimpleTest/SimpleTest.js:509:14
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - testServerTime@https://example.org/tests/netwerk/cookie/test/mochitest/test_server_time.html?currentTestURL=netwerk%2Fcookie%2Ftest%2Fmochitest%2Ftest_server_time.html&closeWhenDone=1&showTestReport=false&expected=pass:55:7
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - async*@https://example.org/tests/netwerk/cookie/test/mochitest/test_server_time.html?currentTestURL=netwerk%2Fcookie%2Ftest%2Fmochitest%2Ftest_server_time.html&closeWhenDone=1&showTestReport=false&expected=pass:69:1
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - TEST-PASS | netwerk/cookie/test/mochitest/test_server_time.html | undefined assertion name 
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - TEST-PASS | netwerk/cookie/test/mochitest/test_server_time.html | undefined assertion name 
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - TEST-PASS | netwerk/cookie/test/mochitest/test_server_time.html | undefined assertion name 
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - Not taking screenshot here: see the one that was previously logged
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - TEST-UNEXPECTED-FAIL | netwerk/cookie/test/mochitest/test_server_time.html | undefined assertion name - got "", expected "test3=foo"
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - SimpleTest.is@https://example.org/tests/SimpleTest/SimpleTest.js:509:14
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - testServerTime@https://example.org/tests/netwerk/cookie/test/mochitest/test_server_time.html?currentTestURL=netwerk%2Fcookie%2Ftest%2Fmochitest%2Ftest_server_time.html&closeWhenDone=1&showTestReport=false&expected=pass:62:7
[task 2024-08-28T18:11:04.113Z] 18:11:04     INFO - async*@https://example.org/tests/netwerk/cookie/test/mochitest/test_server_time.html?currentTestURL=netwerk%2Fcookie%2Ftest%2Fmochitest%2Ftest_server_time.html&closeWhenDone=1&showTestReport=false&expected=pass:69:1
[task 2024-08-28T18:11:04.114Z] 18:11:04     INFO - GECKO(10143) | MEMORY STAT | vsize 2581MB | residentFast 136MB | heapAllocated 7MB
[task 2024-08-28T18:11:04.114Z] 18:11:04     INFO - GECKO(10143) | [Parent 10143, Main Thread] WARNING: IPC message 'PBrowser::Msg_WillChangeProcess' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
[task 2024-08-28T18:11:04.114Z] 18:11:04     INFO - TEST-OK | netwerk/cookie/test/mochitest/test_server_time.html | took 868ms
[task 2024-08-28T18:11:04.163Z] 18:11:04     INFO - TEST-START | netwerk/cookie/test/mochitest/test_sharedWorker.html

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/75fe4246040f
Compare the cookie `expires` attribute with the server time, r=edgul,cookie-reviewers,valentin

Comment 13

[tszentpeteri]

https://hg.mozilla.org/mozilla-central/rev/75fe4246040f

Comment 14

[edgul]

Release Note Request (optional, but appreciated)
[Why is this notable]: When receiving an expiry time for a cookie if the server and client clocks are out of sync a cookie may be considered expired to one and not the other. In particular this is an issue when the server gives a valid cookie that is invalid to the client, the cookie will be immediately thrown out by the client. Here we take the difference between client and server clocks and add that as a buffer to the expiry period to avoid this scenario altogether.
[Affects Firefox for Android]: Yes
[Suggested wording]: If we have the server time, we adjust the "expire" attribute value by adding the delta between the server and the local times. If the current time is set in the future, we consider valid cookies that are not expired for the server.
[Links (documentation, blog post, etc)]: N/A

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Added to the Fx132 relnotes.

Comment 16

[Bianca Hidecuti, Desktop Test Engineering [:bhidecuti]]

I am still able to reproduce the issue as described in Comment 0, using Firefox 132.0b5, on macOS 14.7, Ubuntu 22.04 and on Windows 11. I am redirected to a page that shows the "403: Forbidden" message.
However, I am no longer able to reproduce the issue in Firefox Nightly 133.0a1 (2024-10-10), using the previously mentioned platforms.
@baku, does the fix only apply to Nightly? Thanks!

Comment 17

[Andrea Marchesini [:baku]]

Correct. The fix is behind pref (network.cookie.useServerTime) and is enabled just in nightly.

Comment 18

[Bianca Hidecuti, Desktop Test Engineering [:bhidecuti]]

Thanks. I can confirm that it is no longer reproducing with Firefox 132.0b5, if the pref "network.cookie.useServerTime" is set to true.
Marking this verified as fixed.

Comment 19

[Daniel Holbert [:dholbert]]

(In reply to Andrea Marchesini [:baku] from comment #17)

Correct. The fix is behind pref (network.cookie.useServerTime) and is enabled just in nightly.

Two followup questions on this^ point:

(1) Right now we've got a release note about this for 132 (currently on beta channel); but given that this is turned off for release, does it still make sense to include this in the release notes? If we do keep it, probably we should mention the fact that this new behavior is behind this off-by-default pref...

(2) Is there a bug filed on lifting the nightly-only restriction here? Until we do that, release/beta users will still be affected by this bug, so it's worth being sure we don't forget about doing this. If the intent is to just let this bake and watch for any issues, then maybe we could tentatively remove the Nightly-only guard on mozilla-central (now v133) now or soonish, in the hopes that we can enable this with the 133 cycle barring any issues? (I don't really know this code well enough to understand the risks, though.)

Comment 20

[Andrea Marchesini [:baku]]

We enable this feature to release too in bug 1923872, which should be included in 132. Daniel, can you confirm?

Comment 21

[Daniel Holbert [:dholbert]]

That sounds good, though if you want it to be in 132, you'll need a beta uplift request on that bug - right now its patch only looks to have landed on central which is 133.

Comment 22

[Daniel Holbert [:dholbert]]

Here's the searchfox link for beta132, showing that the nightly guard is still there (as of now):
https://searchfox.org/mozilla-beta/source/modules/libpref/init/StaticPrefList.yaml#12206

(This matches the status flags on bug 1923872 which show it being fixed in 133, not yet fixed in 132)

Comment 23

[Andrea Marchesini [:baku]]

I think it's OK to move the rel-note to 133.

Comment 24

[Daniel Holbert [:dholbert]]

Ok. RyanVM, would you mind moving the remote from 132 to 133?

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Done