Open Bug 1602166 Opened 5 years ago Updated 18 days ago

Implement OAuth 2.0 Dynamic Client Registration Protocol

Categories

(Thunderbird :: Account Manager, enhancement, P2)

Product:
Thunderbird
Component:
Account Manager
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: unicorn.consulting, Unassigned)

References

(Blocks 1 open bug)

Details

Rfc7591 has been out there for a while now. Implement the dynamic client registration so Thunderbird account wizard can determine is the server supports oAuth and register the client instead of having a manual process with a JS file.
https://tools.ietf.org/html/rfc7591

Priority: -- → P2

Recent versions of Dovecot are now OAuth2 capable, so that pretty much every IMAP server deployer out there can enable OAuth2 support.

By not implementing Dyn Client Reg, Thunderbird is effectively blocking uptake of OAuth in mail contexts.

It is not any more acceptable to try and work with a manual whitelist of servers!

(In reply to Stefan Winter from comment #2)

Recent versions of Dovecot are now OAuth2 capable, so that pretty much every IMAP server deployer out there can enable OAuth2 support.

By not implementing Dyn Client Reg, Thunderbird is effectively blocking uptake of OAuth in mail contexts.

It is not any more acceptable to try and work with a manual whitelist of servers!

Ditto.

What is the status on this? It's been open for 2 years and it's still unassigned.

(In reply to tyler.montney from comment #3)

...
What is the status on this? It's been open for 2 years and it's still unassigned.

That's an accurate assessment. There is no development planned in this area that I know of.

See Also: → 1342152, 1561542, 1648779

Right now, we can support only a select few providers with OAuth2 which we have specifically and manually signed up with. There is no standard procedure, we need to create an acccount with the provider, and find out how register our client. We often need to accept contracts from the provider, which is highly problematic in principle. It's similar in effect to software patents, in that the provider can dictate whatever terms they want on us.

We currently support tens of thousands of different IMAP servers (without OAuth2). That would be impossible with OAuth2 and manual client registration. So, this bug is highly important for the openness of email in general.

It would be interesting to implement this.

There are a few problems with the spec in itself. The spec makes open registration optional, allowing the client registration to be subject to authentication, which happens out of band, which then of course renders the whole point moot and the thing useless. Furthermore, the spec allows the server to require the client to constantly update the registration "during the lifetime of the client", which of course massively complicates matters even further. But all that depends on the concrete implementations.

That said, it's pointless to implement in the client, if the servers/email providers don't support it. Does anybody know the status of implementation for:

  • Microsoft
  • Google
  • Yahoo
  • Yandex
  • All the others who support OAuth2 on IMAP?

But the biggest issue with the spec that I see is that there is no discovery protocol. I didn't see anything that tells me how to find the "client registration endpoint". I need to know how to go from email domain (or imap server hostname) to the "client registration endpoint" URL. Nothing tells me what this is. So, I cannot call it, because I don't know its URL/hostname.

Ben, thanks for picking this up! A few comments to your post above, to make things look a little less grim.

The spec makes open registration optional, allowing the client registration to be subject to authentication, which happens out of band, which then of course renders the whole point moot and the thing useless. Furthermore, the spec allows the server to require the client to constantly update the registration "during the lifetime of the client", which of course massively complicates matters even further. But all that depends on the concrete implementations.

Yes, it's optional. So some IMAP providers will allow you to register freely, others don't. But that is not a problem: you first try to find and register openly based on the mail domain. If you succeed, good. If not, look into the manual list you already have (yes, you need to keep such a list; and that list happens to include all the major providers that don't like to play open). If the server doesn't support open registration AND is not in your list - give up.

Doing this will make /more/ Thunderbird users happy (those whose provider allows open registration), but not all. That's fine. You didn't provide service to those unlucky ones before, so no change for them.

The constant update is not much of an issue - you just do the same lookup you did initially every once in a while, maybe on every client startup, and take note of any changes. This is non-interactive and quick (going to a URL and parsing some JSON).

That said, it's pointless to implement in the client, if the servers/email providers don't support it.

Being pointless to implement is a bit of a non-argument as we are dealing with a chicken-and-egg problem. It is as pointless to make a server that could onboard dynamic clients if there is no client that supports it. Running in this circle doesn't help with a problem you describe as being highly important for the openness of email in general. Looking at the big incumbent mail providers and their non-implementation is not evidence of uselessness. They have no incentive towards openness because they are being catered for manually, as the current table of supported OAuth2 IMAP servers in Thunderbird shows.

But the biggest issue with the spec that I see is that there is no discovery protocol. I didn't see anything that tells me how to find the "client registration endpoint". I need to know how to go from email domain (or imap server hostname) to the "client registration endpoint" URL. Nothing tells me what this is. So, I cannot call it, because I don't know its URL/hostname.

This one is a tougher nut to crack indeed. You may want to take a look at RFC 8414 "OAuth 2.0 Authorization Server Metadata", section 3: "Obtaining Authorization Server Metadata" (https://www.rfc-editor.org/rfc/rfc8414.html#section-3) which is the closest I could find, but happy to be proven wrong with better specs :-) It states:

Authorization servers supporting metadata MUST make a JSON document
containing metadata as specified in Section 2 available at a path
formed by inserting a well-known URI string into the authorization
server's issuer identifier between the host component and the path
component, if any. By default, the well-known URI string used is
"/.well-known/oauth-authorization-server". This path MUST use the
"https" scheme.

So, a good bet would be to take the mail domain, prepend https:// then append /.well-known/oauth-authorization-server and see what you get. It's not a guarantee to be correct, but it's something autodiscovery can try and it doesn't hurt.

This may well not yield anything right now, given that IMAP server software architects look at the same spec and see the same degrees of freedom (mail domain or server hostname doesn't /necessarily/ also have a web server on the same host; and the .well-known location is just a default, not a guarantee) - but if there is a very popular client looking at that exact spot, it is an indication where to make it available as a server.

all the major providers that don't like to play open

Hi Stefan, so you're saying that Microsoft wrote the spec, but doesn't implement it?

Dovecot supports the spec. We should support it. Just haven't had time for it yet.

(In reply to Ben Bucksch (:BenB) from comment #7)

all the major providers that don't like to play open

Hi Stefan, so you're saying that Microsoft wrote the spec, but doesn't implement it?

You must be new to the standardization business :-)

Discovery is optional. You can claim full compliance with OAuth 2.0 by doing the mandatory parts, but not doing any single one of the optional elements. In fact, making a protocol feature optional to implement is a popular way of achieving compromise in a standardisation gremium (the feature is "in" after all) and at the same time making sure it won't take off in the real world - because noone can rely on this particular feature actually being there in the field. I'm not saying this is what happened here; I wasn't in the room when the spec work actually happened.

It is trivial for you to figure out if Microsoft has chosen to do this optional part of the OAuth 2 spec series. See what

curl https://hotmail.com/.well-known/oauth-authorization-server
curl https://hotmail.com/.well-known/openid-configuration

gives you. Spoiler: it's a generic HTML page with advertising on how great Hotmail is. It is NOT server authorization metadata.

Similarly, here's what GMail gives you:

curl https://gmail.com/.well-known/oauth-authorization-server or
curl https://gmail.com/.well-known/openid-configuration

<p><b>404.</b> <ins>That’s an error.</ins>
<p>The requested URL <code>/.well-known/oauth-authorization-server</code> was not found on this server. <ins>That’s all we know.</ins>

As for Dovecot support - I don't have an actual deployment handy. Does the Dovecot server have an HTTPS interface, and does it actually deliver the metadata on https://<maildomain>/.well-known/oauth-authorization-server ? If so, great! Now all Thunderbird needs to do is go look there.

Does the Dovecot server ... actually deliver the metadata on https://<maildomain>/.well-known/oauth-authorization-server ?

How could it? It's not serving https://mozilla.org (for @mozilla.org email addresses, as an example). That's also exactly why /.well-known/oauth-authorization-server is nice as a low-fi option, but will have problems to be deployed on a large scale, because you cannot automatically serve it as a mail server. Funny enough, a separate host like https://autoconfig.<domain> is actually easier to manage for many sites.

That's why I asked how it does it. I don't know; and the spec oddly enough is not clear on it (at least not to me; there's enough protocol-specific language in it to possibly hide more details that only comes out after a long and thorough read).

Maybe Dovecot simply serves this on its own hostname? In that case, the usual autodiscover options that lead you to the IMAP's DNS server name will also be suitable for discovering OAuth2 by using that same hostname. In your example, if autodiscover tells you that the IMAP server's name for mozilla.org is imap.mozilla.org, you would look at https://imap.mozilla.org/.well-known/oauth-authorization-server and https://imap.mozilla.org/.well-known/openid-configuration as a legacy fallback.

Thinking about it, that's much more easy to deploy server-side, logical, and compatible with the spec.

Dovecot, does just validate the OAUTH2 token, it does to the best of my knowledge not, give you any further information regarding the identity provider (https://doc.dovecot.org/configuration_manual/authentication/oauth2/)
the well-known ... normally comes from the IDP directly.

I would suggest, just adding the issuer url to the autoconfig xml

I would suggest, just adding the issuer url to the autoconfig xml

Yes, that's already specced for autoconfig as <oAuth2> and available in ISPDB for the biggest mail providers, e.g. https://autoconfig.thunderbird.net/v1.1/yahoo.com

Okay, so there is a good way to find the discovery endpoint. Problem solved?

This could obsolete the hard-coded oauth2 endpoints that are currently in source, right?

This could obsolete the hard-coded oauth2 endpoints that are currently in source, right?

That would be one goal of this bug, yes, but that works only if

  • the current hardcoded endpoints (Gmail, Yahoo, MS Office365, Yandex, etc.) support this protocol. Therefore my question in comment 5.
  • email admins do not use whitelists for allowing clients by ID. I know for a fact that this happens, because I've seen the error message after login. Some domain admins do that, particularly security conscious ones, which would lock us out, because the client ID would change constantly and we'd fail the whitelist.

I've got an interest in this for hosting my own mail domain. I know I'm few and far between, but I thought it worthwhile to ask about some of the challenges.

  • When would the client ID change? Would this for instance be different for every single install or would it be version changes?
  • What could be supported when the registration endpoint requires some form based authentication? Rather than looking strictly at the spec and asking what is in, what would be reasonable if you designing the authentication and authorization side?

My initial thoughts would be that I could provide the appropriate .well-known endpoints and have them redirect to the right places. When Thunderbird is attempting to register its client ID for the first time the user could be asked to enter their credentials (could be TOTP, U2F, Webauthn based) to register an arbitrary client ID. This would save me having to whitelist client IDs but not allow anonymous client IDs. I'm sure there's gaps in the security model but it is better than fully open.

I know this is a chicken-and-egg problem. I noticed that K-9Mail is about to come under the Thunderbird banner so that would mean a client across several platforms that supports OAuth2 login without being tied to a particular iDP. At least that's the dream.

Severity: normal → S3
Duplicate of this bug: 1342152

(In reply to Ben Bucksch (:BenB) from comment #13)

I would suggest, just adding the issuer url to the autoconfig xml

Yes, that's already specced for autoconfig as <oAuth2> and available in ISPDB for the biggest mail providers, e.g. https://autoconfig.thunderbird.net/v1.1/yahoo.com

Why we not allow to configure clientId and clientSecret also from autoconfig.xml?

From a security standpoint this is the same as to add them to the source code (as currently for all big isp).
But this would allow small (private) mail-servers to allow to switch to oauth2 for their servers.
I would deploy my own IDP for this (Keycloak) but currently without any ("customizable") MUA with oauth2 support this is useless.

What is the benefit of dynamically created clients?
Why each MUA(-Installation) needs an own Client for accessing imap/smtp services?

Is there any MUA already using the <oAuth2><issuer>... block already with dynamically created clients?

I agree 100% with Martin Schulz (comment 18).

(In reply to Markus Schulz from comment #18)

(In reply to Ben Bucksch (:BenB) from comment #13)

I would suggest, just adding the issuer url to the autoconfig xml

Yes, that's already specced for autoconfig as <oAuth2> and available in ISPDB for the biggest mail providers, e.g. https://autoconfig.thunderbird.net/v1.1/yahoo.com

Why we not allow to configure clientId and clientSecret also from autoconfig.xml?

From a security standpoint this is the same as to add them to the source code (as currently for all big isp).

What I don't understand is, if it is a security risk to have client_id's and secrets exposed, why does Thunderbird have it? If it isn't, why can't we add our own?

But this would allow small (private) mail-servers to allow to switch to oauth2 for their servers.

Yes, but not only that: Oauth2 is becoming the norm for online authentication. It helps with many things, including a nice way to handle 2FA for e-mail. Thunderbird could be the first e-mail client to actually allow OAuth authentication for everyone, not just for the chosen big providers.

I would deploy my own IDP for this (Keycloak) but currently without any ("customizable") MUA with oauth2 support this is useless.

Same here.

What is the benefit of dynamically created clients?
Why each MUA(-Installation) needs an own Client for accessing imap/smtp services?

Imagine the burden for the IdP to suddenly have millions of clients registered there. I also don't understand the point.

See Also: → 1866093

I just saw Ben's (BenB's) video from FOSDEM 2024 on this. I really appreciate that this hasn't gotten lost and forgotten about. It also helps to understand some of the technical challenges.

https://fosdem.org/2024/schedule/event/fosdem-2024-3026--security-email-autoconfiguration-and-2fa-for-email/

Francis Augusto Medeiros, The video covers client IDs quite nicely and a reasonable proposal on what could be done for mail.

Hey Ben, what about OAuth 2.0 device authorization for a potential way to solve the UI issues? No web browser embedded in the email client and no localhost callback URL.

I like the idea of using passkeys, but you'd still need to use some form of token associated with the client so that every new server connection did not require a new authentication with the passkey. If the token based authentication fails then you fall back to using the passkey. I'd expect the token to live for between 1-90 days depending on the security policy of the ISP. It'd be similar to how I need to re-authenticate with Outlook for mobile every 14 days when using my work account.

Happy to talk more over email or another bug on these bits since it's not related to the original thread anymore.

I found #1864917 "Passkey support in Thunderbird". That may be the better place to discuss such things.

But the biggest issue with the spec that I see is that there is no discovery protocol. I didn't see anything that tells me how to find the "client registration endpoint". I need to know how to go from email domain (or imap server hostname) to the "client registration endpoint" URL. Nothing tells me what this is. So, I cannot call it, because I don't know its URL/hostname.

Pretty old RFC 7628 describes this well. Section 3.2.2:

   For a failed authentication, the server returns an error result in
   JSON [RFC7159] format and fails the authentication.  The error result
   consists of the following values:

      status (REQUIRED):  The authorization error code.  Valid error
         codes are defined in the IANA "OAuth Extensions Error Registry"
         as specified in the OAuth 2.0 core specification.

      scope (OPTIONAL):  An OAuth scope that is valid to access the
         service.  This may be omitted, which implies that unscoped
         tokens are required.  If a scope is specified, then a single
         scope is preferred.  At the time this document was written,
         there are several implementations that do not properly support
         space-separated lists of scopes, so the use of a space-
         separated list of scopes is NOT RECOMMENDED.

      openid-configuration (OPTIONAL):  The URL for a document following
         the OpenID Provider Configuration Information schema as
         described in OIDCD [OpenID.Discovery], Section 3 that is
         appropriate for the user.  As specified in OIDCD, this will
         have the "https" URL scheme.  This document MUST have all
         OAuth-related data elements populated.  The server MAY return
         different URLs for users in different domains, and the client
         SHOULD NOT cache a single returned value and assume it applies
         for all users/domains that the server supports.  The returned
         discovery document SHOULD have all data elements required by
         the OpenID Connect Discovery specification populated.  In
         addition, the discovery document SHOULD contain the
         'registration_endpoint' element to identify the endpoint to be
         used with the Dynamic Client Registration protocol [RFC7591] to
         obtain the minimum number of parameters necessary for the OAuth
         protocol exchange to function.  Another comparable discovery or
         client registration mechanism MAY be used if available.

         The use of the 'offline_access' scope, as defined in
         [OpenID.Core], is RECOMMENDED to give clients the capability to
         explicitly request a refresh token.

So clients should send authentication request, get error filled with scopes and OpenID configuration discovery endpoint, fetch it, register client and try to authenticate again. Pretty straightforward in my opinion.

I just saw Ben's (BenB's) video from FOSDEM 2024 on this
https://fosdem.org/2024/schedule/event/fosdem-2024-3026--security-email-autoconfiguration-and-2fa-for-email/

Thank you for referencing this here.

In this talk, I (briefly) explain how inherently flawed Client IDs are, by design. As long as the entire concept of Client IDs exist, ISPs can simply not offer "Dynamic registration" and force email clients into a mandatory registration. That's inherently anti-competitive, because Google and Microsoft are direct competitors.

At least with Google and Microsoft, there's a hope that we can move them to passkeys or some other mechanism. Once we do allow OAuth2 for arbitary email providers, on countless third party ISPs, we're going to create a legacy protocol and we'll be forever stuck with the troubles that OAuth2 "framework" brings on us. Please note that OAuth2 is not a protocol, it's a "framework" for building more specific protocols. I am proposing to do just that.

Given that nobody else currently can do OAuth2, the much simpler way out of the problem is "mAuth": to simply require to allow the hardcoded-string "mail" as Client ID for email applications. Those providers will also have to adhere to a few other requirements, like returning specific error code and messages, realistic expiry times, and a few other pain points in OAuth2 for email. More details in my talk.

After seeing the video I understand your reluctance to add a web browser or a web server in an email client, but you already have it to connect to the few harcoded providers, so why don't allow either manual configuration of a custom provider or take the config from the autoconfig?
See also https://github.com/thunderbird/thunderbird-android/issues/7466 and https://github.com/thunderbird/thunderbird-android/issues/6152 (yes, not directly thunderbird related but I think it's relevant).
Maybe what you propose (passkeys* or a strict set of requirements for the providers) is better for the long term, but as Keynes said, in the long run we are all dead.

*I'm not a security expert so I cannot really say. Btw, while I can use the fingerprint reader in my mobile to authenticate with desktop chrome, I cannot do the same with desktop firefox.

Btw, while I can use the fingerprint reader in my mobile to authenticate with desktop chrome, I cannot do the same with desktop firefox.

Passkeys don't require any biometric sensor to work. Windows and macOS support platform passkeys which usually derived using TPM. In case of Windows you just need to enable PIN-based Windows Hello. Linux is somewhat troublesome, but you can still use password managers like Bitwarden or 1Password which allow you to create portable passkeys stored directly in the manager.

I can't really follow the argumentation in that talk. It's true that there is some variance in how you profile OAuth on the server side. Some of these deployment profiles will work well, others will work less well, and some may not work at all.

As a server operator, you will care, and will configure your server towards a good experience for your users. If you don't, your users will jump at you, the server operator. Maybe some of the deployment options need signalling between server and client which is not built-into the protocol: then express these options in autoconfig; then the client software knows how the server wants to play it.

If a server deployment works for some client, but not for another (which Ben brought as an example), then that is an indication that it really is the client software's fault: apparently other software can work better given the same server deployment configuration.

There is also the striking disparity that OAuth is implemented, but only for those big 3. In that situation, saying you don't want to be the guy who opened the floodgates for random other ISPs is the same as saying you are actively supporting this oligarchy. The "argument" brought forward about introducing future legacy is basically chasing ghosts that may or may not manifest at all.

The trade-off here is breaking an oligarchy of mail providers at the expense of writing code that may be frowned upon in a possible uncertain future; vs. supporting the oligarchy.

I have implemented parts of the WebAuthn spec and am involved in protocol development around FIDO CTAP2. Those last slides in the talk are pretty much handwaving of sorts "no idea about the details, but it sounds nice". A very basic sketch that doesn't fly without very much work and time to be put into.

In essence, FIDO CTAP2 and its deployment profile that is called Passkeys are protocols internal to a computer and do not deal with authentication on the internet at all (CTAP = Client to Authenticator Protocol).

To enable the use of these authenticators on the internet, one needs to write a higher-layer specification that explains how to leverage the cryptographic keys from the authenticator in that specific context. One such context is "the web browser" - the protocol is then called "WebAuthn". There are only few well-spec'ed use cases beyond that one. There is a way to leverage CTAP2 with SSH, generating SSH keys that bind to a specific authenticator (not standardised, OpenSSH wrote that spec on their own). There is also work in progress (mine et. al.) to leverage CTAP2 authenticators for authenticating with EAP in Wi-Fi, 802.1X and VPN contexts ( https://www.ietf.org/archive/id/draft-janfred-eap-fido-01.html ).

There is no protocol whatsoever to use Passkeys in the context of email authentication. (Well, you can use OAuth, present a browser windows and then authenticate in the browser window using your Passkey - for some reason, I don't think this is what Ben was hoping for.)

The usability in the context of email is also questionable (as it is in the Wi-Fi context): common authenticators require user interaction ** for every single authentication transaction **. I.e. every IMAP login includes entering a PIN, or using a biometric trait, or similar.

There is a CTAP2 operation mode called "silent authentication" which allows auth transactions to occur silently; we are trying to make use of that for those EAP/Wi-Fi use cases. But the reaction we hear from the platform authenticator people is that this is something that is non grata (even though defined perfectly fine in the CTAP2 spec).

So, you need to write specs, have server and client implementations, and rely on edge cases in dependent specs that are not typically available.

(Compare that to OAuth: there are specs (even if they may not be entirely perfect), there are server and client implementations, and you need to rely on common cases of the specs)

I cannot even guess how many years it would take to get all that worked upon and settled. We can easily talk a decade here. I cannot imagine Ben's suggestion is to ignore OAuth for that long time and stick with passwords until that magic moment has arrived. Because passwords are "simple"? That argumentation is a good way to make Thunderbird meaningless.

Why can't we have a compromise here? Maybe allowing configuring OAuth2 via some configuration file (like policies.js or something like that?

The thing with OAuth2 is not just about Thunderbird. Is that most major mail software already support it. Today corporate e-mail is forced to migrate to big providers because of MFA. If it takes 3-4 years to support something compatible with the current standards and that is open for everyone, it might be too late.

Since my company is directly affected by this, I'll just add my two cents.

I understand that you want to do it correctly and go to the best possible solution, without doing something half-way or temporary. That is something that makes sense in the long term.

However, thunderbird has obviously decided to do something else. That train already left the station.

Hard-coded support for google and co was clearly necessary to keep a certain userbase, as they require 2FA with oauth2. I don't blame you for supporting them, but it's offensive and extremely counter-productive to support webauthn over the browser and oauth2 with a few, specific, big providers and not for anyone else.

I prefer not creating a new standard or doing things in a non-standard way, but in this case I think it's best to just do it. As a system engineer responsible for email, this lack of oauth2/2FA support for thunderbird undermines my desire to move from gmail to our own dovecot-based solution. I had to test it all with mutt, which was a good learning experience for me, but practical for the non-technical people at my company.

The arguments against supporting oauth2 for general use (for self-hosted or other dovecot instances) are pathetic. If this doesn't get implemented soon (at the very least gives me the option of configuring it manually), then I think it's time to make a fork that more closely adheres to open-source values.

Hello,

I also don't understand the argumentation. TB market share is tiny (especially outside of Europe). If the strategy is to have TB in the enterprise/business environment (after all, there are TB policies through files/AD), then I think supporting custom Oauth2 configuration is a logical step (or something that is already supported by common on-premise/private cloud email servers which does allow to limit Oauth2 scopes/permissions).

BTW at my work, we moved away from TB last year and we noted the lack of Oauth and also the lack of motivation to implement this by reading up in the different forums. It was not the most important argument but it definitely didn't help.

I don't think there's a a lack of desire to implement this. It's just not very easy to do.

You need to log in before you can comment on or make changes to this bug.