Closed Bug 1866991 Opened 6 months ago Closed 4 months ago

App updates don't respect security.enterprise_roots.enabled when a redirect is involved (breaks Thunderbird updates)

Categories

(Toolkit :: Application Update, defect)

Product:
Toolkit
Component:
Application Update
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
124 Branch
Tracking Flags:
Tracking Status
firefox124 --- fixed

People

(Reporter: reqman, Assigned: bytesized)

References

Details

(Whiteboard: [fidedi-ope])

Attachments

(2 files)

update_messages.log
1.74 KB, text/plain
Details
Bug 1866991 - Don't use BadCertHandler in UpdateService r=nalexander!,dveditz!
48 bytes, text/x-phabricator-request
Details | Review

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0

Steps to reproduce:

My government branch now requires local clients to install a root CA, in order to inspect HTTPS traffic for malware, as well as DLP. If this root CA is not installed on a client system, or it is installed but the application does not utilize it, HTTPS access to the net is impossible.

Officially we use Firefox as the default browser and Thunderbird as the default email client. Once the above change took place and after including the root CA to the Windows system certificate store, we noticed that communication was not possible for the former, whereas the latter could not check for TB updates.

We found that by setting security.enterprise_roots.enabled to true, via a branch-wide GPO, the issue was resolved for Firefox. However, thunderbird is still unable to check for program updates. We can see that this option is present and locked in it, and enabled as well ("true").

Steps to reproduce in our environment:

  1. Open TB
  2. Go to help -> about Thunderbird

Actual results:

  1. In the dialog that opens, after a while a message (translated) appears: "Update check has failed"

Expected results:

  1. In the dialog that opens information about either the existence of a newer version, or that the one installed is current should be given
Component: Untriaged → Security

Is it also using some proxy or something else to intercept the traffic? Perhaps you haven't set that other pref for Thunderbird?

(In reply to Magnus Melin [:mkmelin] from comment #1)

Is it also using some proxy or something else to intercept the traffic? Perhaps you haven't set that other pref for Thunderbird?

If you are referring to what the gov IT is doing, then the answer is that they might, by they are definitely not sharing the details. On my branch side, no insrtuctions were given for a proxy, so they most likely are network transparent ones and beyond my control. Settings regarding proxy on my Firefox are set to "direct connection to the internet" and it works just fine after setting security.enterprise_roots.enabled to true.

After opening my issue here, I've found https://bugzilla.mozilla.org/show_bug.cgi?id=1668191 which essentially is the same. I'd close my own issue as a duplicate, but I'll leave it open a bit since it is about the current Thunderbird version and, well, I'm definitely around to provide logs if needed.

On the subject of logs, I'd be grateful if someone could share information on how to get TB logs from its update operations (most things I've seen about TB logging affect its core mail functionalities, ie POP/IMAP/SMTP etc).

If you set security.osclientcerts.autoload true, does it work? (But then S/MIME won't work properly)

(In reply to Magnus Melin [:mkmelin] from comment #4)

If you set security.osclientcerts.autoload true, does it work? (But then S/MIME won't work properly)

I've set it to true and after restarting TB the issue persists.

With all due respect to the (pro bono) developers, is there any chance to prioritize this due to the implicit security impact it has? We have around 2000 TB installations on our gov organization, for which the attack/exploitable surface will just get larger, due to the inability to effectively upgrade all installations.

On my own branch (~150 systems) TB was deployed via GPO, so I could mitigate for the time by installing a newer version via a GPO. This is not the case for all points, whereas installation should be made on a per system basis.

(Most likely, even if a fix comes, the re-installation will not be avoided; local re-install is the only way to go as things are).

Is it only checking for updates that do not work?
Do e.g. https web pages as the start page (see settings) load alright?

Nice catch: enabling the TB start page in settings does display a "Welcome to Thunderbird" page without a hitch! (Unless of course this info is hard-coded into TB and no actual downloading takes place?)

To help you help us, help you:
(In reply to Michail Pappas from comment #3)

On the subject of logs, I'd be grateful if someone could share information on how to get TB logs from its update operations (most things I've seen about TB logging affect its core mail functionalities, ie POP/IMAP/SMTP etc).

Perhaps setting app.update.log to true could help diagnose it.
The code is around here: https://searchfox.org/mozilla-central/rev/f961e5f2a22f4d41733545190892296e64c06858/toolkit/mozapps/update/AppUpdater.sys.mjs#221

See Also: → 1734992
Attached file update_messages.log —

Additional info: visiting the addons page in TB and selecting to search for a specific addon (say "snooze") from addons.thunderbird.net works perfectly! Additionally actually selecting to install one also seems to work without a hitch: addon is downloaded and installed just fine!

So this could affect solely update check and upgrade?

(In reply to Magnus Melin [:mkmelin] from comment #9)

Perhaps setting app.update.log to true could help diagnose it.
The code is around here: https://searchfox.org/mozilla-central/rev/f961e5f2a22f4d41733545190892296e64c06858/toolkit/mozapps/update/AppUpdater.sys.mjs#221

That's great! I'm attaching the update_messages.log file.

Summary: Thunderbird: setting of security.enterprise_roots.enabled is not respected → Thunderbird: setting of security.enterprise_roots.enabled is not respected when updating TB

Just adding that I can not decipher the contents of the update_messages.log file, not a coder here. This is in the devs ball park for resolution.

I really hope that upper management will not push for a switch to Outlook, my own TB installation is from the very first version of it :(

That update log doesn't look like what I'd expect it to... not sure what's wrong

I wonder if some network setting is different between you Firefox and Thunderbird installation. Here are some of them to check: https://searchfox.org/mozilla-central/source/modules/libpref/init/StaticPrefList.yaml#11988

(In reply to Magnus Melin [:mkmelin] from comment #12)

That update log doesn't look like what I'd expect it to... not sure what's wrong

I wonder if some network setting is different between you Firefox and Thunderbird installation. Here are some of them to check: https://searchfox.org/mozilla-central/source/modules/libpref/init/StaticPrefList.yaml#11988

That looks to be bug 1830368. That was fixed back in August though... after the 115 branch was made so it might need uplifting.

(In reply to Magnus Melin [:mkmelin] from comment #12)

That update log doesn't look like what I'd expect it to... not sure what's wrong

I wonder if some network setting is different between you Firefox and Thunderbird installation. Here are some of them to check: https://searchfox.org/mozilla-central/source/modules/libpref/init/StaticPrefList.yaml#11988

I took a peek at your feedback in the issue quoted by Rob Lemley. Took me a while but there seems that Tb has a debug log console built-in. With that console open and app.update.log set to true I've received the following, the error "Certificate issuer is not built-in" stands out:

AUS:AUM AppUpdater:check - currentState=STATE_IDLE
AUS:AUM AppUpdater:check - starting update check
AUS:SVC CheckerService:checkForUpdates - checkType: 2
AUS:SVC CheckerService:checkForUpdates - Making new check request for check id 2.
AUS:SVC CheckerService:getUpdateURL - checkType: 2
AUS:SVC CheckerService:getUpdateURL - update URL: https://aus.thunderbird.net/update/6/Thunderbird/115.5.0/20231120220011/WINNT_x86_64-msvc-x64/el/release/Windows_NT%2010.0.0.0.19045.3930%20(x64)/ISET:SSE4_2,MEM:16180/default/default/update.xml?force=1
AUS:SVC CheckerService:#updateCheck - sending request to: https://aus.thunderbird.net/update/6/Thunderbird/115.5.0/20231120220011/WINNT_x86_64-msvc-x64/el/release/Windows_NT%2010.0.0.0.19045.3930%20(x64)/ISET:SSE4_2,MEM:16180/default/default/update.xml?force=1
NS_ERROR_ABORT: Certificate issuer is not built-in. CertUtils.sys.mjs:177
  checkCert resource://gre/modules/CertUtils.sys.mjs:177
  asyncOnChannelRedirect resource://gre/modules/CertUtils.sys.mjs:205
AUS:SVC CheckerService:#updateCheck - request got 'load' event
AUS:SVC CheckerService:#updateCheck - request completed downloading document
AUS:SVC CheckerService:#updateCheck - there was a problem checking for updates. Exception: TypeError: request.responseXML is null
AUS:SVC getStatusTextFromCode - transfer error: Κακοδιατυπωμένο αρχείο ενημέρωσης XML (200), default code: 200
AUS:AUM AppUpdater:check - Update check failed; CHECKING_FAILED

I'm not sure what that implies. Thunderbird uses aus.thunderbird.net and Firefox uses aus5.mozilla.org. But how that impacts any imported root CAs...

:bytesized, any ideas?

Flags: needinfo?(bytesized)

Can you please try directly loading the update URL (https://aus.thunderbird.net/update/6/Thunderbird/115.5.0/20231120220011/WINNT_x86_64-msvc-x64/el/release/Windows_NT%2010.0.0.0.19045.3930%20(x64)/ISET:SSE4_2,MEM:16180/default/default/update.xml?force=1) via the browser location bar. Does it load successfully? What does it load?

Flags: needinfo?(bytesized) → needinfo?(reqman)

Not sure what you mean by "browser location bar" here, is it some TB bar?

The URL opens just fine in Firefox though, with the following contents:

<updates>
<update appVersion="115.6.1" buildID="20240105183125" detailsURL="https://live.thunderbird.net/thunderbird/releasenotes?locale=el&version=115.6.1&channel=release" displayVersion="115.6.1" type="minor">
<patch type="complete" URL="https://download.mozilla.org/?product=thunderbird-115.6.1-complete&os=win64&lang=el" hashFunction="sha512" hashValue="7ea017ecf46ac64b5af7d2276cde1d8bee5d281fb2021fcddfc0c7ea38ff2f4d2011efa65578ea623a12538b4fa574a3ff9d228b3eb9cbb325baaf27b9c6d749" size="65402787"/>
</update>
</updates>
Flags: needinfo?(reqman)

In Thunderbird, you can set that URL to the startpage (in settings)
Or, open the error console (Ctrl+Shift+J) and enter openContentTab("https://aus.thunderbird.net/update/6/Thunderbird/115.5.0/20231120220011/WINNT_x86_64-msvc-x64/el/release/Windows_NT%2010.0.0.0.19045.3930%20(x64)/ISET:SSE4_2,MEM:16180/default/default/update.xml?force=1"); - it should show an xml response in a Thunderbird tab.

(In reply to Magnus Melin [:mkmelin] from comment #18)

In Thunderbird, you can set that URL to the startpage (in settings)

Really feel dummy here :/

Or, open the error console (Ctrl+Shift+J) and enter openContentTab("https://aus.thunderbird.net/update/6/Thunderbird/115.5.0/20231120220011/WINNT_x86_64-msvc-x64/el/release/Windows_NT%2010.0.0.0.19045.3930%20(x64)/ISET:SSE4_2,MEM:16180/default/default/update.xml?force=1"); - it should show an xml response in a Thunderbird tab.

XML opens up just fine either way (console or start page).

Hmm, I looked around a bit, and it seems like we explicitly do not allow non-built-in certificates during update transfer. Off the top of my head, I don't know if there is a good reason for this requirement. I'll have to look into it.

It looks like this was once controllable by a pref, and we hardcoded it in Bug 1315505.

addon update check to versioncheck-bg.addons.thunderbird.net seems to be affected as well, with a reason the one :bytesized mentioned above (raised exception "Certificate issuer is not built-in"):

 1705469462126	addons.update-checker	WARN	Request failed: https://versioncheck-bg.addons.thunderbird.net/update/VersionCheck.php?reqVersion=2&id=SpellingGR-EN_US@kostaskatsaros&version=0.8.7.1webext&maxAppVersion=null&status=userEnabled&appID={3550f703-e582-4d05-9a08-453d09bdfdc6}&appVersion=115.5.0&appOS=WINNT&appABI=x86_64-msvc&locale=el&currentAppVersion=115.5.0&updateType=112&compatMode=strict - [Exception... "Certificate issuer is not built-in."  nsresult: "0x80004004 (NS_ERROR_ABORT)"  location: "JS frame :: resource://gre/modules/CertUtils.sys.mjs :: checkCert :: line 177"  data: no]

Oh. I think I need to revise what I said in my previous comment. I seem to have missed this bit from the bug description on my first read-through:

We found that by setting security.enterprise_roots.enabled to true, via a branch-wide GPO, the issue was resolved for Firefox. However, thunderbird is still unable to check for program updates.

That I am having a very hard time explaining. To be honest, based on what I mentioned in my last comment, I'm fairly surprised that security.enterprise_roots.enabled fixes this on Firefox.

But given that (a) it works on Firefox but not Thunderbird, (b) it sounds like, from Comment 8, security.enterprise_roots.enabled otherwise works on Thunderbird, and (c) AFAIK, Thunderbird and Firefox use the exact same code to update, I'm kinda stumped. I'll try to make some time to think more about the issue, but I'm not feeling optimistic at the moment.

The only thing I can think of is that the update code in Firefox and Thunderbird isn't the same version.
@Michail Pappas - Could you check with an up-to-date Firefox and an up-to-date Thunderbird, if you haven't already, and check if Firefox can still update properly and Thunderbird still can't?

Flags: needinfo?(reqman)

Oh, I do have one other idea for something to check. Maybe it's possible that I was correct in Comment 20, and some external reason is causing Firefox to behave differently from Thunderbird. But, this exact same code seems to be used in the Addons update checker, and it also has a preference that the reporter could control: extensions.update.requireBuiltInCerts.

@Michail Pappas - Can you still reproduce the issue from Comment 21 if you set extensions.update.requireBuiltInCerts to false?

(In reply to Magnus Melin [:mkmelin] from comment #4)

If you set security.osclientcerts.autoload true, does it work?

Client (user) certs are not involved in updates.

(In reply to Daniel Veditz [:dveditz] from comment #24)

Client (user) certs are not involved in updates.

What do you mean? It sounds like you are saying that update will not (should not?) use non-builtin HTTPS certificates, but the reporter seems to be saying that Firefox is updating successfully while being MITMed, so that doesn't seem right.

Flags: needinfo?(dveditz)

One difference I see is that Firefox updates go directly to aus5.mozilla.org, but Thunderbird loads aus.thunderbird.net and then that redirects to aus5.mozilla.org. I think this is the source of the difference.

In the addons case I see explicit calls to CertUtils.checkCert() in AddonUpdateChecker.sys.mjs and ProductAddonChecker.sys.mjs once the final content has been fetched, but I don't see anywhere that UpdateService.sys.mjs ever explicitly checks for built-in certs. BUT—the BadCertHandler does have an asyncOnChannelRedirect listener that will call checkCert() on each redirect automatically.
https://searchfox.org/mozilla-central/rev/15f758f06d97ee3c4e382b174836395442c38f71/toolkit/modules/CertUtils.sys.mjs#184

It doesn't make sense to skip the built-in cert check for the download but then check them on redirects. I'm guessing somewhere along the line the explicit checks were removed, but they never removed the now-vestigial BadCertHandler because it wasn't hurting anything—until you add a redirect into the mix.

Someone should definitely double-check this quick-hit since I haven't looked at this code in years; not really since the bug 1315505 era when I used to sit near Robert Strong in Mountain View. It would be worth bisecting to figure out when nsUpdateService.js/UpdateService.sys.mjs stopped calling checkCert() and see if my "forgot to unhook the BadCertHandler" theory is plausible. It doesn't make any sense to check redirects if you're not checking the final result. I'm pretty confident the built-in-cert checks were intentionally removed because it caused a lot of download failures for people who used anti-virus with a traffic-scanning MITM feature. The built-in-cert checks were a very poor "better than nothing" option used before we switched to signed-MAR updates.

Options:

The 1st and 3rd will require a Thunderbird update since the update URL isn't a pref anymore, in which case you might as well do the 3rd option since that both easier and more correct. I'm guessing "aus.thunderbird.net" is a load balancer rewrite rule and can't in practice serve meaningful content itself.

Meanwhile, the reporter might be able to change the update URL through group policy. It's still documented at https://mozilla.github.io/policy-templates/#appupdateurl, and while it's no longer a pref, it does look like updateService checks the policy directly. Obviously you'd need to change "Firefox" to "Thunderbird" in the recipes that use the product name. Assuming Thunderbird supports group policies.

I think you'd want to set the value to the same thing Firefox uses; it's the same request structure anyway:
"https://aus5.mozilla.org/update/6/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%SYSTEM_CAPABILITIES%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml" but you'd definitely have to test it. Literally use that string -- the product itself replaces the variable values for you.

(In reply to Robin Steuber (they/them) [:bytesized] from comment #25)

If you set security.osclientcerts.autoload true, does it work?

Client (user) certs are not involved in updates.

What do you mean? It sounds like you are saying that update will not (should not?) use non-builtin HTTPS certificates,

Magnus suggested changing a pref related to client certs which will have no impact whatsoever on an issue involving server certs. Two different beasts.

Flags: needinfo?(dveditz)

(In reply to Robin Steuber (they/them) [:bytesized] from comment #22)

The only thing I can think of is that the update code in Firefox and Thunderbird isn't the same version.
@Michail Pappas - Could you check with an up-to-date Firefox and an up-to-date Thunderbird, if you haven't already, and check if Firefox can still update properly and Thunderbird still can't?

Firefox works just fine, TB does not. I'm administering 130 clients with the same setup and our software asset tracking shows that most clients are at the latest firefox version, whereas TB is stuck at the version right before the government made the network MITM change.(In reply to Robin Steuber (they/them) [:bytesized] from comment #23)

Oh, I do have one other idea for something to check. Maybe it's possible that I was correct in Comment 20, and some external reason is causing Firefox to behave differently from Thunderbird. But, this exact same code seems to be used in the Addons update checker, and it also has a preference that the reporter could control: extensions.update.requireBuiltInCerts.

@Michail Pappas - Can you still reproduce the issue from Comment 21 if you set extensions.update.requireBuiltInCerts to false?

I think that there was no such variable in config, so I created a boolean one, set it to false and restarted TB. This time addons were updated just fine (but TB update still does not work)! Is this behaviour a bug or a feature?

Off-topic a bit, but do you know if this setting is configurable via adm/admx templates to set it via GPO to all of my clients?

(In reply to Daniel Veditz [:dveditz] from comment #26)

Someone should definitely double-check this quick-hit since I haven't looked at this code in years; not really since the bug 1315505 era when I used to sit near Robert Strong in Mountain View. It would be worth bisecting to figure out when nsUpdateService.js/UpdateService.sys.mjs stopped calling checkCert() and see if my "forgot to unhook the BadCertHandler" theory is plausible. It doesn't make any sense to check redirects if you're not checking the final result. I'm pretty confident the built-in-cert checks were intentionally removed because it caused a lot of download failures for people who used anti-virus with a traffic-scanning MITM feature. The built-in-cert checks were a very poor "better than nothing" option used before we switched to signed-MAR updates.

Your reference to AV MITM traffic-scanners initially confused me: we have been using ESET Endpoint for year, which does include TLS filtering, but has been working without a hitch for years. Then I recalled that the difference is that locally installed AV software (in constrast to network UTM boxes) usually install their own root CA directly to the application (TB/Firefox in this case, as well as to the Windows store), so no issues there...

Options:

I think you'd want to set the value to the same thing Firefox uses; it's the same request structure anyway:
"https://aus5.mozilla.org/update/6/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%SYSTEM_CAPABILITIES%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml" but you'd definitely have to test it. Literally use that string -- the product itself replaces the variable values for you.

I've got a mixed bag of results by doing so: I've modified my TB GPO to set a custom URL as per above. On my own rig doing a gpupdate to get the new policy worked like a charm, the system immediately downloaded the next version. I've checked this on a couple of other systems: worked on one, did not work or another. But since it does work, it's my job to fix (for the time being).

Only question is the one above: ie, should I also unset extensions.update.requireBuiltInCerts to my clients, in addition to pushing the customURL, or does doing the latter fix extension update as well?

Flags: needinfo?(reqman)

Thanks for checking the update URL: the results confirm my theory that the redirect is the problem. I don't know why it doesn't work in all cases but there could be a second unrelated bug affecting those machines (or maybe GPO is disabled or failing on some machines?)

@Michail Pappas - Can you still reproduce the issue from Comment 21 if you set extensions.update.requireBuiltInCerts to false?

I think that there was no such variable in config, so I created a boolean one, set it to false and restarted TB. This time addons were updated just fine (but TB update still does not work)! Is this behaviour a bug or a feature?

Not sure what part your question is asking is a bug or feature. It is intentional that this pref doesn't exist in about:config; unless you create one the default value is set in code. It is intentional that this extension update pref does not affect app updates: separate mechanisms, with content signing added at different times.

Off-topic a bit, but do you know if this setting is configurable via adm/admx templates to set it via GPO to all of my clients?

I don't believe you can set this pref via GPO. The confusing thing to me is that I think the default value should already be false! Does Thunderbird not require signed addons? TIL, Thunderbird does not require signed extensions: bug 1727113. That means the default value for that pref is true (unlike Firefox) and makes sense why setting it would help.

https://searchfox.org/mozilla-central/rev/9d88be0557a5bc39d3da288f9aff71414baa303e/toolkit/mozapps/extensions/internal/AddonSettings.sys.mjs#51-53,63-69

Excuse me while I go uninstall all my Thunderbird addons and disable update checks...

Please file a separate bug about extension updates not working in this situation. It will require different changes in different places in the code, and potentially a lot of arguing about how to fix the extension part. The APP-update part looks very simple and uncontroversial so let's stick to that in this bug. Hopefully you have the "New/Clone" button at the top of the page here. If so create a bug that "... depends on this bug". If you can't one of us can do it, but it's likely best if you're the bug reporter

(In reply to Daniel Veditz [:dveditz] from comment #26)

One difference I see is that Firefox updates go directly to aus5.mozilla.org, but Thunderbird loads aus.thunderbird.net and then that redirects to aus5.mozilla.org. I think this is the source of the difference.

Ahhhh, good catch. So it sounds like this bug is in the Toolkit::Application update code, not Thunderbird code. I'll go ahead and move this to the appropriate component.

Options:

The 1st and 3rd will require a Thunderbird update since the update URL isn't a pref anymore, in which case you might as well do the 3rd option since that both easier and more correct. I'm guessing "aus.thunderbird.net" is a load balancer rewrite rule and can't in practice serve meaningful content itself.

I agree. I think the third option is preferable. I'll see about writing a patch.

(In reply to Michail Pappas from comment #28)

I think that there was no such variable in config, so I created a boolean one, set it to false and restarted TB. This time addons were updated just fine (but TB update still does not work)! Is this behaviour a bug or a feature?

I mean, I would say that the existence of the pref is a feature, but the fact that it doesn't work without setting it in this case is a bug. But technically speaking that is probably for the addons people to decide.

Off-topic a bit, but do you know if this setting is configurable via adm/admx templates to set it via GPO to all of my clients?

There is a policy to set semi-arbitrary prefs. It looks like extensions. in on the "allowed prefix list", so you should be able to set extensions.update.requireBuiltInCerts that way if you want to. I would still recommend filing a bug with the addons team though, if you haven't already.

Only question is the one above: ie, should I also unset extensions.update.requireBuiltInCerts to my clients, in addition to pushing the customURL, or does doing the latter fix extension update as well?

Ideally, we will fix on our end rather than requiring a custom update URL, but it should be fine to use the custom update URL in the meantime. It only controls Application Updates though, not extension updates. Which will also be true of the patch I am about to write.

Assignee: nobody → bytesized
Component: Security → Application Update
Product: Thunderbird → Toolkit
Version: Thunderbird 115 → unspecified

aus.thunderbird.net is some sort of cloudfront service that just logs and redirects. Helps gather some platform and locale stats.

First let me thank you for helping me here, FOSS support can be that awesome!

In reply to Daniel Veditz [:dveditz] from comment #29)

Thanks for checking the update URL: the results confirm my theory that the redirect is the problem. I don't know why it doesn't work in all cases but there could be a second unrelated bug affecting those machines (or maybe GPO is disabled or failing on some machines?)

Getting old -> getting less careful. You were spot on that this was GPO-related: I simply did not reboot (my policy was on the system part of the GPO). Oil's well then.

@Michail Pappas - Can you still reproduce the issue from Comment 21 if you set extensions.update.requireBuiltInCerts to false?

I think that there was no such variable in config, so I created a boolean one, set it to false and restarted TB. This time addons were updated just fine (but TB update still does not work)! Is this behaviour a bug or a feature?

Not sure what part your question is asking is a bug or feature. It is intentional that this pref doesn't exist in about:config; unless you create one the default value is set in code. It is intentional that this extension update pref does not affect app updates: separate mechanisms, with content signing added at different times.

Thanks, that was what I was asking.

(In reply to Robin Steuber (they/them) [:bytesized] from comment #31)

(In reply to Michail Pappas from comment #28)

I think that there was no such variable in config, so I created a boolean one, set it to false and restarted TB. This time addons were updated just fine (but TB update still does not work)! Is this behaviour a bug or a feature?

I mean, I would say that the existence of the pref is a feature, but the fact that it doesn't work without setting it in this case is a bug. But technically speaking that is probably for the addons people to decide.

So, the appdate issue will be handled here by you Robin, whereas you'd like me to open an issue about the extensions not updating. I'll give it a try.

Off-topic a bit, but do you know if this setting is configurable via adm/admx templates to set it via GPO to all of my clients?

There is a policy to set semi-arbitrary prefs. It looks like extensions. in on the "allowed prefix list", so you should be able to set extensions.update.requireBuiltInCerts that way if you want to. I would still recommend filing a bug with the addons team though, if you haven't already.

I'm not certain what should I write in the GPO. I mean, there's an option to enable it (doh) and a box to write down something (in JSON from the looks of it) but I'm clueless to the exact syntax. No worries, most (if not all) of my users have no extensions installed, so it's no biggie security-wise for me -> I'll leave things as they are for now.

Only question is the one above: ie, should I also unset extensions.update.requireBuiltInCerts to my clients, in addition to pushing the customURL, or does doing the latter fix extension update as well?

Ideally, we will fix on our end rather than requiring a custom update URL, but it should be fine to use the custom update URL in the meantime. It only controls Application Updates though, not extension updates. Which will also be true of the patch I am about to write.

My question at the time (which was resolved by Daniel above) was whether fixing the customURL would fix the extension update. Which it does not, they are separate issue as I now understand.

Tried to create https://bugzilla.mozilla.org/show_bug.cgi?id=1875428 not sure if I wrote that one up ok, feel free to fix my bug report.

Whiteboard: [fidedi-ope]
See Also: → 1875428

(In reply to Daniel Veditz [:dveditz] from comment #26)

Someone should definitely double-check this quick-hit since I haven't looked at this code in years; not really since the bug 1315505 era when I used to sit near Robert Strong in Mountain View. It would be worth bisecting to figure out when nsUpdateService.js/UpdateService.sys.mjs stopped calling checkCert() and see if my "forgot to unhook the BadCertHandler" theory is plausible. It doesn't make any sense to check redirects if you're not checking the final result. I'm pretty confident the built-in-cert checks were intentionally removed because it caused a lot of download failures for people who used anti-virus with a traffic-scanning MITM feature. The built-in-cert checks were a very poor "better than nothing" option used before we switched to signed-MAR updates.

I've tracked this down. It looks like Bug 1182352 was supposed to remove transport certificate checking, but missed the BadCertHandler call. Specifically, this is where we removed the checkCert() call. It looks to me like it is clearly the intention that update not check transport certificates.

I should have a patch up for this shortly.

It appears that the current usage of BadCertHandler was added back before update MARs were signed (allowing us to rely on the MAR certificate regardless of the security characteristics of the transport mechanism). After MAR signing was added, we removed the checkCert() call in Bug 1182352 since we no longer needed it. Presumably, the BadCertHandler call should have been removed at the same time, but it must have been overlooked.

Updated the summary to reflect the underlying generic bug we are fixing. It's breaking Thunderbird updates today, but it could affect Firefox in the future. For example, if we switch cloud providers we might roll out aus6.mozilla.org and then set up a redirect from aus5 to handle old clients. We must have done something like that when retiring aus3 and aus4.

Summary: Thunderbird: setting of security.enterprise_roots.enabled is not respected when updating TB → App updates don't respect security.enterprise_roots.enabled when a redirect is involved (breaks Thunderbird updates)
Pushed by rsteuber@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2c23f3159f5a
Don't use BadCertHandler in UpdateService r=nalexander,dveditz

https://hg.mozilla.org/mozilla-central/rev/2c23f3159f5a

Status: UNCONFIRMED → RESOLVED
Closed: 4 months ago
status-firefox124: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 124 Branch
Duplicate of this bug: 1880145
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: