Closed Bug 1656082 Opened 5 years ago Closed 4 years ago

Autoconfig detects STARTTLS, but verification of STARTTLS happens at the end while checking the credentials

Categories

(Thunderbird :: Account Manager, enhancement)

Product:
Thunderbird
Component:
Account Manager
Platform:
Unspecified
All
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: danielf, Unassigned)

References

Details

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0

Expected results:

Hello Thunderbird team,

While investigating the functionality of the autoconfig engine, we at Linuxmagic noticed a potential issue.

The autoconfig engine, when the auth mechanism is set to 'autodetect' mode for either an incoming or outgoing server, it will not perform a STARTTLS when probing STARTTLS configurations.

This is an issue because auth mechanisms may be advertised differently after TLS has been negotiated, for example, AUTH PLAIN should never be advertised over an insecure connection.

As per RFC the capabilities advertised before STARTTLS MUST be discarded after successfully negotiating TLS.

IMAP RFC: https://tools.ietf.org/html/rfc2595#section-3.1

  Once TLS has been started, the client MUST discard cached
  information about server capabilities and SHOULD re-issue the
  CAPABILITY command.  This is necessary to protect against
  man-in-the-middle attacks which alter the capabilities list prior
  to STARTTLS.  The server MAY advertise different capabilities
  after STARTTLS.

So if the account is configured to explicitly use STARTTLS then the capabilities advertised before TLS is negotiated must not be honoured.

Unfortunately in order to fix this it would require the STARTTLS command be performed mid-way through the autoconfiguration probe and I am at a loss for how this could be performed.

Any insight into how we might be able to perform STARTTLS within the autoconfig probe would be appreciated, I will cook up a patch if anybody could point me in the right direction.

Unfortunately in order to fix this it would require the STARTTLS command be performed mid-way through the autoconfiguration probe and I am at a loss for how this could be performed.

BenB, any thoughts?

Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(ben.bucksch)
OS: Unspecified → All

Hi Dan, can you please fill out a bug report with exact reproduction steps, expected result, and actual result? The dialog is complex and I don't know which exact step and situation you are talking about. I think you are talking about the manual configuration dialog, and what we do depends on the field values, so please describe the state of each config field. Please also include the error console debug output in an attachment.

Flags: needinfo?(ben.bucksch)

Hello Ben, apologies for not filling out the form correctly, first let me describe where in the UI these issues occur.

I am talking about the autoconfig popup window which is used when configuring a new email account.

If the user enters their Name + Email + Password and then chooses "Configure Manually" (half-manual config) then they are given the option to select "Autodetect" for certain fields, or simply set them to their desired value.

The user can then hit "Re-Test" to validate the fields they have manually configured, and automatically populate the rest of the fields.

Once the form is filled and nothing is left to autodetect it unlocks the "Done" button which can be used to perform a final probe that will actually test the authentication.

This problem is centralized around the "Re-test" button and is not an issue with the "Done" button.

I will try to describe a problematic situation below, but first let me try to explain the fundamental problem again.

Just some preliminary information:

Here is a snippet from the RFC listed above: https://tools.ietf.org/html/rfc2595#section-9

Any protocol interactions prior to the TLS handshake are performed in
the clear and can be modified by a man-in-the-middle attacker. For
this reason, clients MUST discard cached information about server
capabilities advertised prior to the start of the TLS handshake.

And additionally from the IMAP RFC itself: https://tools.ietf.org/html/rfc3501#section-6.2.1

 Once [TLS] has been started, the client MUST discard cached
 information about server capabilities and SHOULD re-issue the
 CAPABILITY command.  This is necessary to protect against man-in-
 the-middle attacks which alter the capabilities list prior to
 STARTTLS.  The server MAY advertise different capabilities after
 STARTTLS.

So there is 2x cases where it's described that a client "MUST" discard cached information from pre-TLS capabilities, which indicates we are at the very least non-conformant to the expected client behaviour.

And then there is the line: "The server MAY advertise different capabilities after STARTTLS."

Therefore it is perfectly legal to change the available capabilities after STARTTLS has been initialized, and clients are instructed to clear all cached capabilities from before STARTTLS was negotiated as a means of handling this.

This concludes the preliminary information.

So, a problematic example. This is not the only problem that can result from this but just one example of a possible problem, consider the following:

(This is just a hypothetical example based on the established fact that the capability MAY change after STARTTLS)

  1. An IMAP server which only advertises CRAM-MD5 AFTER TLS but advertises no auth mechanisms before TLS. The server does support CRAM-MD5 but it does not allow or advertise it over insecure channels.

  2. The user is semi-advanced and has chosen to manually configure their email account, they have chosen in the half-manual autoconfig GUI:

Protocol: IMAP
Server: imap.myserver.com
Port: Autodetect
SSL: STARTTLS
Authentication: Encrypted Password <<< this is the important selection

(for sake of example only using IMAP right now -- but the same problem may exist in SMTP due to the similar behaviour of STARTTLS)

Once the user filled in their options like above they would hit "Re-test" -- the important part is they chose "Encrypted Password" for the "Authentication" option; which pertains to CRAM-MD5. Keep in mind the server won't advertise CRAM-MD5 till after STARTTLS.

The scenario goes something like this:

  1. Thunderbird connects to this server and sees "LOGINDISABLED" and "STARTTLS" but sees no AUTH= mechanisms.

  2. Based on the logic in _advertisesAuthMethods() Thunderbird would determine that there is no supported auth methods

  3. Based on the logic in _hasTLS() Thunderbird would determine STARTTLS is supported

  4. Since there was no auth methods found and STARTTLS was found, it reverts the "Authentication" to "Normal Password", the the TLS method to "STARTTLS" (TLS method is effectively unchanged)

  5. The chosen "Authentication" which was explicitly set to "Encrypted Password" is reverted back to "Normal Password" because CRAM-MD5 was not seen in the server pre-TLS capabilities.

The problem described here is if they choose "Encrypted Password" and hit "Re-test" then it will wipe out their value and revert it back to "Normal Password" because CRAM-MD5 was not detected.

The expected behaviour would be the Thunderbird probe should perform STARTTLS and also scrape the capabilities that are advertised over a secure channel, and in turn allow the user to choose "Encrypted Password" without reverting it after a "Re-test".

The only time this doesn't happen is when the user has specifically chosen SSL/TLS instead of STARTTLS as their SSL mechanism, in which case the connection itself is launched under implicit SSL and there is no worry about capabilities being advertised differently.

Hopefully I have explained the issue better, unfortunately I am at a loss for how this might begin to be solved.

I believe the majority of the autoconfig probe would need to be enhanced to first perform STARTTLS and then scrape the capability and to ignore the capability before that for everything except whether STARTTLS is supported.

Thanks in advance, and please let me know if there is anything else I can do to help.

Flags: needinfo?(ben.bucksch)

Can you please give a real world example that I can test? I'd like to test this and observe it myself. Can you please give me actual values to fill out, to see this from beginning to end?

Which ISPs are affected?

if they choose "Encrypted Password" and hit "Re-test" then it will wipe out their value and revert it back to "Normal Password" because CRAM-MD5 was not detected.

I would like to observe that. We normally only set values that were on "Autodetect" and don't change explicitly entered user values. But auth method might be an exception.

The chosen "Authentication" which was explicitly set to "Encrypted Password" is reverted back to "Normal Password" [even though the server offers CRAM-MD5 after STARTTLS]

OK, speaking about this specific scenario:

Although this is unfortunate and wrong and a bug,

  1. it's not a security bug, because we still wouldn't send passwords in cleartext. They would still be protected by SSL. Most servers admins say that this is sufficient and there's no point in CRAM-MD5 with an SSL connection, because it would be double encrypted. You can observe that in our ISP database: Very few domains allow CRAM-MD5 within SSL. (We do list that, if we know that it's available.) So, the mentioned problem is of theoretical nature and not a security problem.
  2. More importantly, CRAM-MD5 has become unpopular, because it's incompatible with other password encryption methods like bcrypt. In fact, MD5 today is a red flag in itelf. So, the admin has the choice of using CRAM-MD5, and storing passwords with MD5, and being victim who steal password databases from disk and use rainbow tables, and decrypt all passwords. That's why you see CRAM-MD5 usage decreasing dramatically since many years.
    I'm just saying that I cannot see the real world security problem here.

I accept that this is a bug. If you have a patch, feel free to attach it.

Flags: needinfo?(ben.bucksch)

(In reply to Ben Bucksch (:BenB) from comment #4)

Can you please give a real world example that I can test? I'd like to test this and observe it myself. Can you please give me actual values to fill out, to see this from beginning to end?

My apologies, unfortunately I don't have a real world example to just hand you, but you should be able to demonstrate the "reverted value" behaviour with an unmodified IMAP server by simply selecting an auth mechanism that isn't supported and hitting "re-test".

Any auth mechanism you choose that isn't supported by the server in it's pre-TLS capabilities will be reverted, in fact this is probably an intended feature if not for the oversight in which capability list it honours (if it was reading the post-tls capabilities this revert would probably be a nice feature).

The inverse can be tested with a server that does support CRAM-MD5, if the pre-tls capability indicates CRAM-MD5 then your selection will not be reverted.

And of course I understand that you you have will have no way to test a "fix" because you need a server that only advertises CRAM-MD5 after TLS, but this shouldn't matter till later when we have a fix to test.

So as long as the pre-TLS capabilities don't include "CRAM-MD5" then your chosen option will be reverted.

And if the pre-TLS capabilities do include CRAM-MD5 then your chosen option will not be reverted.

If you need to confirm that the post-TLS capabilities have no bearing on the result of the above two tests then I can point out several places in the test which explicitly indicate that STARTTLS is never performed (the easiest being the CMDS[] arrays which simply don't have STARTTLS in them) and therefore the post-TLS capabilities cannot possibly play a role in reverting the value.

So if you setup the half-manual autoconfig to have the following IMAP info (it doesn't matter what you put into SMTP):

Protocol: IMAP
Server: imap.myserver.com
Port: Autodetect
SSL: STARTTLS
Authentication: Encrypted Password <<< this is the important selection

Then you hit "Re-test" to probe the server which doesn't advertise CRAM-MD5 then it will revert the selection despite not being set to "Autodetect".

Hopefully this is enough for you to confirm the behaviour.

Which ISPs are affected?

Well, unfortunately this isn't the result of a bug that was encountered, but rather the result of a code review determining that expected behaviour wasn't being honoured.

So I don't actually have any problematic servers or ISPs that I can provide to you.

However once we have a potential plan of action and I begin to carry out the changes I will inevitably need to be able to test the fixes and so I will look into setting up a test dovecot server once that's the case but until then I don't think I can provide a test server.

if they choose "Encrypted Password" and hit "Re-test" then it will wipe out their value and revert it back to "Normal Password" because CRAM-MD5 was not detected.

I would like to observe that. We normally only set values that were on "Autodetect" and don't change explicitly entered user values. But auth method might be an exception.

Indeed I didn't think Thunderbird updated the user values unless they were 'Autodetect' either, I was somewhat surprised when I saw it change.

The chosen "Authentication" which was explicitly set to "Encrypted Password" is reverted back to "Normal Password" [even though the server offers CRAM-MD5 after STARTTLS]

OK, speaking about this specific scenario:

Although this is unfortunate and wrong and a bug,

  1. it's not a security bug, because we still wouldn't send passwords in cleartext. They would still be protected by SSL. Most servers admins say that this is sufficient and there's no point in CRAM-MD5 with an SSL connection, because it would be double encrypted. You can observe that in our ISP database: Very few domains allow CRAM-MD5 within SSL. (We do list that, if we know that it's available.) So, the mentioned problem is of theoretical nature and not a security problem.

Yes I recognize that this isn't actually a security bug.

I believe at least one of the main reasons for the RFC mandating that pre-tls capabilities be discarded is the possibility of STARTTLS downgrade attack.

The attack would involve a Man-in-the-middle replacing STARTTLS and LOGINDISABLED with something else -- so as to cause the client to see:

  1. There is no LOGINDISABLED
  2. There is no STARTTLS

The result of this attack is the client is manipulated into sending credentials over the plaintext wire because AUTH PLAIN is the only available authentication mechanism and STARTTLS is not available.

HOWEVER! Thunderbird mitigates this attack by detecting if the final auth probe (the "Done" button) will effectively send credentials over a plaintext connection -- if that's the case then you receive the large red confirmation dialogue about the insecure connection.

So I would have to agree that the core purpose of discarding credentials is effectively mitigated through the unavoidable big red user prompt about insecure connections.

But still, the principle of trusting "data that could be altered by a MITM" is fundamentally bad.

For example, future additions to the protocol/capability may have security considerations that are predicated on the expectation that clients discard pre-tls capability. (after-all it's a "MUST" in the IMAP RFC to discard the capabilities so future protocol enhancements could predicate on this behaviour)

Therefore if Thunderbird is sidestepping the matter and not discarding the capabilities then the introduction of a new protocol/capability extension change could introduce security vulnerabilities similar to a STARTTLS downgrade attack.

  1. More importantly, CRAM-MD5 has become unpopular, because it's incompatible with other password encryption methods like bcrypt. In fact, MD5 today is a red flag in itelf. So, the admin has the choice of using CRAM-MD5, and storing passwords with MD5, and being victim who steal password databases from disk and use rainbow tables, and decrypt all passwords. That's why you see CRAM-MD5 usage decreasing dramatically since many years.
    I'm just saying that I cannot see the real world security problem here.

The use of CRAM-MD5 in this example has been purely hypothetical, it could be replaced with any authentication mechanism -- the problem will still arise if the given auth mechanism is only advertised after TLS.

You would be correct in that I don't think there is directly a security problem here because Thunderbird effectively sidesteps around the MITM issue by always vetting the final connection for potential plaintext leak and ensuring the user is always prompted beforehand.

So I would classify the problem as a matter of conformance to spec which could potentially lead to security vulnerabilities in the future.

I accept that this is a bug. If you have a patch, feel free to attach it.

Well, unfortunately, I was hoping the Thunderbird team could help out there.

I would be happy to prepare a patch but I'm not sure where to start or even how I would go about performing a TLS handshake from javascript.

Flags: needinfo?(ben.bucksch)

The attack would involve a Man-in-the-middle replacing STARTTLS and LOGINDISABLED with something else

This case would be true even after fixing this bug. If we're subject to a MITM during setup, we cannot prevent downgrade attacks.

That's why we have the other safeguards, like showing the server hostnames and explicitly warning about insecure setups, yes. That's exactly what they are for.

any authentication mechanism -- the problem will still arise if the given auth mechanism is only advertised after TLS.

I see the theoretical problem, yes.

I'm not sure where to start

UI: emailWizard.js half-manual setup
Network protocols: guessConfig.js

Flags: needinfo?(ben.bucksch)

UI: emailWizard.js half-manual setup
Network protocols: guessConfig.js

My apologies if I came across sounding helpless, I suppose I didn't explain my roadblocks quite so well.

I have done quite the investigation into all of the files in the accountcreation folder and I firmly understand how they work and I understand that's where the work would need to be done.

The problem is I cannot find anywhere that demonstrates how I would be able to perform a TLS handshake through the message pump itself.

All occurrences of TLS handshakes I see are handled by the C++ and simply exposed through a single function in javascript like transportService.createTransport() which can implicitly initialize a native SSL connection.

I can't seem to find anywhere that would demonstrate how I would initiate the TLS handshake in the message pump, or if that's even possible.

Thanks in advance and apologies for eating so much of your time with this.

Flags: needinfo?(ben.bucksch)

Hi Dan, I guess that's where I lost you with "initiate the TLS handshake in the message pump", because you don't do that from JS. From JS, you call functions like transportService.createTransport(). You are getting a TCP socket back, either with or without SSL, depending on how you started the socket.

If you need something else, you need to explain more why you need it and what exactly. In any case, I wouldn't be able to answer it. Maybe benc, who worked on bug 1590473, or kaie (but Kai is very busy).

Flags: needinfo?(ben.bucksch)

You are speaking specifically about STARTTLS, not the dedicated TLS ports.

I looked at the code, and the relevant code seems to be in https://searchfox.org/comm-central/source/mail/components/accountcreation/content/guessConfig.js#602 in function _processResult():

    // parse auth methods from server response and remember them
    thisTry.authMethods = this._advertisesAuthMethods(...)
    // if we're expecting STARTTLS (called "TLS" here), ensure that STARTTLS is in the capabilities response
    if (thisTry.ssl == TLS && !this._hasTLS(thisTry, wiredata)) {
      thisTry.status = kFailed; return;
    }
    ..."success"...

You can see that it simply takes the last server response line, treats that as CAPABILITIES line, and parses it. It takes the auth methods out of it, and it checks whether STARTTLS is supported in the capabilities, if we're testing for that.

Given that STARTTLS is inherently protocol-specific, and these sockets are generic TCP or TLS sockets, there is no STARTTLS implementation here. We're only checking what the server advertizes. Whether that actually works will be checked after the user approved the settings and we connect with the actual login. If STARTTLS was advertized, but doesn't work, then it will fail at this point and the user has to fix it.

So, to fix the bug you mention, you would need to create an actual STARTTLS connection, and wait for the response after that. However, given that it's protocol-specific, that's not trivial. But it should be possible. I don't know the SSL APIs that you would use for upgrading an existing TCP socket to TLS, but you could check the C++ implementation of IMAP, POP3 or SMTP (one of them), and see which SSL APIs they use. Probably, the same API would be available and accessible from JS as well, thanks to XPCOM, and then you would do the same to upgrade the TCP socket to TLS. After you've done that, you would re-start the same test. It would then run the normal EHLO and CAPABILITIES commands again, and use the same protocol-specific parsing functions, just this time for the responses after STARTTLS.

This is going to be quite some work.

Thank you very much for this feedback, it's really helped.

So, to fix the bug you mention, you would need to create an actual STARTTLS connection, and wait for the response after that.

That sounds exactly like what I was thinking, the only part that was unknown for me was how to get the STARTTLS to run through that message pump.

I don't know the SSL APIs that you would use for upgrading an existing TCP socket to TLS, but you could check the C++ implementation of IMAP, POP3 or SMTP (one of them), and see which SSL APIs they use.

For whatever reason it never occurred to me that there would already be APIs for simply upgrading the connection, I was looking entirely for handlers that would perform STARTTLS in it's entirety.

So I could theoretically couple a manual STARTTLS command with a call to upgrade the transport/connection and it would produce a STARTTLS handshake and TLS initiation -- it seems so obvious now I feel bad that I didn't think of this.

I'm going to investigate that avenue and I'll post back here when I've made some headway, hopefully there isn't too much in the way.

Thanks again

This is going to be quite some work.

Yeah, I definitely agree.

See Also: → 1149101

Unless somebody volunteers to implement this, I don't think it's worth spending time on this, for the reasons explained in the last comment. We are detecting STARTTLS (at the same time we detect auth methods), and we are checking STARTTLS (when we test the actual setup, before creating the final account), so I think it's OK as-is.

I've just tested this, and we are correctly detecting servers that offer STARTTLS and offer that.

Reproduction:

  • Advanced config editor, search for mailnews.auto_config_url and make the value empty (no URL).
  • In the dialog, enter foo@fastwebnet.it

Actual and expected result:

  • Config was guessed/probed
  • IMAP with STARTTLS found
  • SMTP with STARTTLS found

So, this works as expected. There's no actual bug here.

So, given that Dan has been inactive, and there's no real bug here, I'm going to mark this INVALID.
If Dan or anybody else wants to contribute the code to implement it, feel free to reopen this bug, but please try to implement it first, because it's not going to be easy. You need to re-implement parts of the IMAP, POP3 and SMTP protocols, at least to the point until you upgrade the connection, and you need to know the exact XPCOM SSL APIs to use.

Status: NEW → RESOLVED
Type: defect → enhancement
Closed: 4 years ago
Resolution: --- → INVALID
Summary: Autoconfig not performing STARTTLS when detecting auth mechanisms → Autoconfig detects STARTTLS, but verification of STARTTLS happens at the end while checking the credentials
You need to log in before you can comment on or make changes to this bug.