1880888 - CRLite files are not downloaded if push services are disabled

Status: UNCONFIRMED

(Firefox :: Remote Settings Client, defect)

Description

[portbackward]

Steps to reproduce:

CRLite requires Firefox Push Services to be enabled ("dom.push.enabled = true") to continously download and update CRLite revocation files. With the setting enabled it's reliable.

With "dom.push.enabled = false" it started to get unreliable.
I tested many different Firefox installations 24/7 and even after days of being opened no CRLite files were downloaded.

"dom.push.enabled = false" is disabled by many users to prevent a persistent AWS connection, when Push Services are not required. CRLite is a critical security feature and should not only reliably work when Push Services are enabled.

Actual results:

CRLite files located in the "security_state" folder are often not downloaded or downloaded with a delay of many days. In most of my testings the folder is empty when "dom.push.enabled = false"

So the feature is not reliable and mostly no revocation checks can be performed with disabled push services.

Expected results:

It should be possible to configure a poll interval which is regularly downloading CRLite files when "dom.push.enabled = false" with a standard poll interval set per default.

Also Firefox should immediately try to fetch CRLite files on first run to fetch CRLite files on first start.

Comment 1

[:Gijs (he/him)]

I don't see an obvious way the code relies on the push service. Mathieu, am I missing something?

Comment 2

[portbackward]

I reverse engineered the behavior and researched official Mozilla documents that describe CRLite.
My research is documented here: https://github.com/arkenfox/user.js/issues/1811
Based on that I'm 100% sure CRLite depends on Push Services.

Reverse engineered behavior with a clean Firefox install and following about:config attributes:

• "dom.push.enabled = false";
• "security.pki.crlite_mode = 2";
• " security.remote_settings.crlite_filters.enabled = true";
  With a clean install of Firefox 122.0.1 and 123.0 I'm able to confirm, that my installations are not receiving any CRLite files.
  I see no intend of Firefox to receive CRLite files with Push Services disabled. The user profile folder "security_state" is empty.

Reverse engineered behavior with a clean Firefox install and following about:config attributes:

• "dom.push.enabled = true";
• "security.pki.crlite_mode = 2";
• " security.remote_settings.crlite_filters.enabled = true";
  With a clean install of Firefox 122.0.1 and 123.0 I'm able to confirm, that my installations are receiving CRLite files successfully around 5-10 minutes after opening the browser first. When the files were downloaded I recognized a network connectivity In this case following files are downloaded to user profile folder "security_state":
• crlite.coverage
• crlite.enrollment
• crlite.filter
• data.safe.bin

Comment 3

[portbackward]

Attachment: Mozilla HighLevel Description of CRLite push behavior

Source: https://blog.mozilla.org/security/2020/12/01/crlite-part-4-infrastructure-design/
Per description fromMozilla CRLite downloads are dependent on Push Services enabled.
Per my testings I can confirm this, only with "dom.push.enabled = true" files are downloading.

Feature request:
The browser should attempt to download CRLite files also with "dom.push.enabled = false"

Comment 4

[Mathieu Leplatre [:leplatrem]]

Thank you for your report.

I am not familiar with the CRLite details, but from the Remote Settings standpoint I can say that:

• Remote Settings initiates synchronizations when it receives Push notifications (and short after startup)
• In parallel a daily timer is also in place
• Hence, when Push is disabled, synchronization is not going to be instantaneous but should still run every day. On startup, it's also possible that nsITimerCallback takes more time to kick in than the Push hello/register phase

Comment 5

[portbackward]

Thanks for your reply.

So far I have no installation where CRLite files are downloaded when dom.push.enabled is set to false.
I also checked old installations where Firefox was installed for months and used many times, also in this case no files were downloaded.

Per my testings with dom.push.enabled set to false CRLite files are not downloaded.
Would be great if the issue could further be checked by the experts :)

Comment 6

[:Gijs (he/him)]

Given all the detail on that public bug, keeping this security sensitive seems pointless.

Comment 7

[portbackward]

I see it as security sensitive, when Firefox is not informed about revoked certificates.
When a website like your banking site uses a revoked certificate and Firefox is not informed, the user is also not informed and continues to access the website. The whole point of CRLite is to warn the user about revoked certificates to stop proceeding.

Users which decided to disable Push Services for additional privacy reasons are excluded from receiving CRLite files, in that case revocation information of certificates is missing. In my point it's a download of four additional files, where the download could also be triggered from the client side in regular intervals.

Sure OCSP could be used, but it's not offered by every website and CRLite is the way to go performance and privacy-wise.

Comment 8

[:Gijs (he/him)]

(In reply to Mathieu Leplatre [:leplatrem] from comment #4)

• In parallel a daily timer is also in place
• Hence, when Push is disabled, synchronization is not going to be instantaneous but should still run every day. On startup, it's also possible that nsITimerCallback takes more time to kick in than the Push hello/register phase

It's an update-timer, right? Cf. https://searchfox.org/mozilla-central/rev/9982a76fe95c70f7c2dca2d60bd78e015313924d/services/settings/servicesSettings.manifest#7

So it should be 24h across restarts (ie the browser doesn't need to be continuously running for the timer to fire as soon as the browser is running after 24h have elapsed).

Reporter, what is the value of app.update.lastUpdateTime.services-settings-poll-changes in about:config for a profile that you've been using for a long time with no CRLite info? What about any prefs that start with services.settings.security-state. ?

Comment 9

[portbackward]

app.update.lastUpdateTime.services-settings-poll-changes values for three installs which didn't receive CRLite files:

1. Setup running for several months: 1708416389
2. Setup running for three days: 1708404981
3. Setup running for one day: 1708457430

services.settings.security-state. entries do not exist in about:config on all three systems.

Comment 10

[:Gijs (he/him)]

So I defer to Dana/jschanck and Mathieu here - I'm not sure who would be best to look into this so I'm needinfo-spreeing a little - sorry! Please redirect as appropriate. It does seem disconcerting if our fallback for pushless scenarios (potentially not limited to ones where users are disabling the service) is not working.

AFAICT one potential cause for the timer-based updates not working may be that:

1. the non-push based remote settings update happens via the update timer (every 24h). I think that this might happen early on startup (once the 24h have elapsed), except for cases where the 24h timer fires while the browser is already running.
2. the security settings are initialized off the "best effort" user idle tasks, which get kicked off once the user is idle for 20s or more, in CPU-idle periods, cf. https://searchfox.org/mozilla-central/rev/c26f7461fc2a51196b7f517c7f98a1e271dc9ec0/browser/components/BrowserGlue.sys.mjs#2470 .

If (2) happens after (1), then the code in RemoteSecuritySettings might not get its "poll ended" observer notifications, because they fire before (2) adds its observer for those, and so that code just won't run.

Even if this explanation is right, it's still surprising to me if the timer never fired while Firefox was already running, over many months of regular use (which I take to mean: at least once a day, for a sustained period). And if the timer did fire, I don't understand why at that point the remote security settings would not be initialized and do something. Unless the idle notification is never firing on the reporter's system and so we just don't ever initialize RemoteSecuritySettings nor run the other things in that BrowserGlue code? That would equally be worrying...

Comment 11

[portbackward]

Guess I have some good news.

I tested with a new install and following values (reproduced 3x times):
"services.settings.poll_interval" set to 1 (instead of 86400 which is the default value)
"dom.push.connection.enabled" set to false (this time dom.push.enabled is true but the AWS connection is still prohibited)

After that the CRLite files were downloaded approximately 15 minutes after my browser was opened.

Still irritated by this, since the setting "services.settings.poll_interval" should be based on seconds per my understanding.
The configured update interval differs a lot from the configured value.

Maybe there is an issue with the poll interval and a multiplicator behind that delays the update frequency, which isn't an issue when the value is set to 1 but extremely delays the download with the value set to 86400 (default)?

Any idea about this and a recommendation what value for "services.settings.poll_interval" would be best with Push disabled?

Comment 12

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Using a local build, I cleared my profile and started Firefox with dom.push.enabled set to false. After a minute or two of leaving the browser idle, I checked the new profile directory for the security_state folder, and it had been populated. So, as described, I can't reproduce this issue.

Comment 13

[portbackward]

I reproduced the issue many times and can't confirm the behavior Dana is mentioning.

@Dana:
Did you make sure to disable internet connectivity when the new install was started and dom.push.enabled was set to false?
I made sure internet connectivity is disabled during the first startup, since dom.push.enabled is set to true per default.
Adjusted the value then, closed my browser, enabled internet connectivity.

With dom.push.enabled set to false and the default services.settings.poll_interval of 86400 my Firefox installations don't receive CRLite files after many days. My workaround now is to set services.settings.poll_interval to value 1. In this case the files are downloaded at the first startup and approximately every 2 hours per day when the browser is opened.

Seems to me there is still an issue with dom.push.enabled set to false together with the default poll interval of 86400.

Comment 14

[Dana Keeler (she/her) (use needinfo) [:keeler]]

I started Firefox with a new profile with the pref set to false. It had never connected to the internet before.

Comment 15

[Mathieu Leplatre [:leplatrem]]

My workaround now is to set services.settings.poll_interval to value 1.

This might fix your problem, but it means that potentially your browser is going to check for new settings updates every second. You may want to increase it to a few minutes at least

Comment 16

[portbackward]

(In reply to Mathieu Leplatre [:leplatrem] from comment #15)

My workaround now is to set services.settings.poll_interval to value 1.

This might fix your problem, but it means that potentially your browser is going to check for new settings updates every second. You may want to increase it to a few minutes at least

This indeed fixes the problem.

What I'm curious about is, if the poll interval is really working how intended.
With the value set to 1, my CRLite files get updated every 1-2 hours.
With the value set to 2, it is delayed around 3-4 hours.
With the value set to 10800, I didn't see an update for many days, so I changed back the value to 1.

My understanding was the services.settings.poll_interval is based on seconds, but is this really the case or might it work differently when dom.push.enabled is set to false?

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:keeler, could you have a look please?

For more information, please visit BugBot documentation.

Comment 18

[Dana Keeler (she/her) (use needinfo) [:keeler]]

I'm still not convinced that a) this is a bug and b) it's a bug specific to CRLite, so I'm going to put this in the remote settings component.

Comment 19

[portbackward]

Hello, I further evaluated the behavior and want to comment back.

The poll interval of "set services.settings.poll_interval" seems to work as intended and is based on seconds.
Only if new updates for CRLite files exist, the files get downloaded.

Still, with the value set to 86400 and disabled Push Services, it mostly does not download any files.
It seems to count 86400 seconds down based on how long the browser is opened.
When the browser was opened only for a few hours each day, it might take weeks to download CRLite files.
On my setup I adjusted the value to 3600 and updates are downloaded on a regular basis.

My recommendation would be, that in future a setting is included, which enforces CRLite downloading when the browser is opened each time.
Only when it is already opened, frequent updates should be based on the poll interval set by "set services.settings.poll_interval".
This way you make sure that every user receives continious CRLite updates, even with Push Services disabled.

Maybe this use case is too specific, since your main user bases is using Push Services.
Would be great to see such an option and here your feedback.

Except from delayed download times CRLite seems to work absolutely great and is a big security and privacy feature.
Thanks to Mozilla team for this feature and it's development!

Comment 20

[portbackward]

Adding a comment to my previous post:
With disabled push services and a poll interval of 86400, CRLite files are also often inaccurate since they are not updated frequently.
In a worse scenario a revoked certificate is not recognized, since CRLite files were not updated for multiple days or weeks.
This can be a security issue, so besides adding a setting that triggers CRLite download each time the browser is opened, maybe the poll interval should be changed to a lower value.

Comment 21

[Krestofir]

I believe that CRL update should be on behalf of Mozilla Update Service not by browser itself.
I am sure that (and history proofs that ) SSL certificate cannot be revoked immidiately by the issuing authority because they have no abilties to track proper usage of it. Years may pass until someone will report leaked (stoled) private keys.
CRL is a feature which rarely helps.

Comment 22

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:leplatrem, could you have a look please?

For more information, please visit BugBot documentation.