683783 - [config] Add support for hosted domains @fastmail.fm/fastmail.com and enable secure autoconfig

Status: RESOLVED FIXED

(Webtools :: ISPDB Database Entries, defect)

Description

[Casey]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0.1) Gecko/20100101 Firefox/6.0.1
Build ID: 20110830092941

Steps to reproduce:

Thunderbird does not recognize domains hosted with fastmail -- Business/Family and Enhanced on a personal account

you add a name server to your domain and have full dns with fastmail

ns1.messagingengine.com
ns2.messagingengine.com

or you add mx records to your domain for email only

in1.smtp.messagingengine.com, priority=10
in2.smtp.messagingengine.com, priority=20

http://www.fastmail.fm/help/domain_management_setup.html

I am using the name servers.

The domains with fastmail are the same settings for the standard auto-setup fastmail accounts using their domain addresses.

Comment 1

[Ludovic Hirlimann [:Usul]]

For this to work we would need to host a fastmail config. Could you create the file for fastmail as described at https://developer.mozilla.org/en/Thunderbird/Autoconfiguration/FileFormat/HowTo and attach it here. Once we host this file we should be able to recognized hosted domains.

Comment 2

[Casey]

thought it would use the same config file as the normal fastmail accounts with some slight adjustments for your domains.


STARTTLS IMAP works

IMAP and POP does not work with encrypted passwords

STARTTLS POP does not work -- thought it did work during testing but stopped working

STARTTLS SMTP 110 does not work / STARTTLS SMTP default 25 which does not work -- STARTTLS SMTP was working before and then stopped,  maybe my setup - An error occurred sending mail: The mail server sent an incorrect greeting:  +OK POP3 ready.

SMTP encrypted password does work but not IMAP,  and then encrypted passwords stopped working

I have taken out STARTTLS from POP and SMTP

username @ domain.com


<?xml version="1.0" encoding="UTF-8"?>

<clientConfig version="1.1">
  <emailProvider id="fastmail.fm">
    <domain>fastmail.fm</domain>
    <displayName>FastMail.FM</displayName>
    <displayShortName>FastMail.FM</displayShortName>
    <incomingServer type="imap">
      <hostname>mail.messagingengine.com</hostname>
      <port>993</port>
      <socketType>SSL</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <incomingServer type="imap">
      <hostname>mail.messagingengine.com</hostname>
      <port>143</port>
      <socketType>STARTTLS</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <incomingServer type="pop3">
      <hostname>mail.messagingengine.com</hostname>
      <port>995</port>
      <socketType>SSL</socketType>
      <authentication>password-cleartext</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <outgoingServer type="smtp">
      <hostname>mail.messagingengine.com</hostname>
      <port>465</port>
      <socketType>SSL</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </outgoingServer>
    <documentation url="https://www.fastmail.fm/help/remote_email_access_server_names_and_ports.html">
      <descr lang="en">Server Names and Ports</descr>
    </documentation>
  </emailProvider>
</clientConfig>

Comment 3

[Ludovic Hirlimann [:Usul]]

can you attach the configuration as an xml file to this bug ?

Comment 4

[Casey]

Attachment: FastMail.FM.xml -- Windows 7 Notepad ANSI Encoding

Comment 5

[Casey]

had it on the wrong ports for STARTTLS SMTP 110 25

587 does work,  does not work on encrypted passwords


xml attachment -- FastMail.FM.xml
Windows 7 Notepad ANSI encoding


<?xml version="1.0" encoding="UTF-8"?>

<clientConfig version="1.1">
  <emailProvider id="fastmail.fm">
    <domain>fastmail.fm</domain>
    <displayName>FastMail.FM</displayName>
    <displayShortName>FastMail.FM</displayShortName>
    <incomingServer type="imap">
      <hostname>mail.messagingengine.com</hostname>
      <port>993</port>
      <socketType>SSL</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <incomingServer type="imap">
      <hostname>mail.messagingengine.com</hostname>
      <port>143</port>
      <socketType>STARTTLS</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <incomingServer type="pop3">
      <hostname>mail.messagingengine.com</hostname>
      <port>995</port>
      <socketType>SSL</socketType>
      <authentication>password-cleartext</authentication>
      <username>%EMAILADDRESS%</username>
    </incomingServer>
    <outgoingServer type="smtp">
      <hostname>mail.messagingengine.com</hostname>
      <port>465</port>
      <socketType>SSL</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </outgoingServer>
    <outgoingServer type="smtp">
      <hostname>mail.messagingengine.com</hostname>
      <port>587</port>
      <socketType>STARTTLS</socketType>
      <authentication>password-encrypted</authentication>
      <username>%EMAILADDRESS%</username>
    </outgoingServer>
    <documentation url="https://www.fastmail.fm/help/remote_email_access_server_names_and_ports.html">
      <descr lang="en">Server Names and Ports</descr>
    </documentation>
  </emailProvider>
</clientConfig>

Comment 6

[Casey]

110 POP STARTTLS does not work

Comment 7

[Casey]

Thunderbird is now detecting my domain as fastmail.

with it not detecting before,  the name servers were up and was receiving mail,  but was still on the domain provider's page for www.domain.com and mail.domain.com was not updated to the login page.

the pages are now pointing over to fastmail.

Thunderbird may be on the new config detecting it,  or was still coming through detecting it.

Comment 8

[Ludovic Hirlimann [:Usul]]

Casey is the attached file the good version ?

Comment 9

[Casey]

the first one not attached I took out STARTTLS which I had the ports wrong and then put it in with the attached file with the code posted on screen.

Comment 10

[Ludovic Hirlimann [:Usul]]

(In reply to Casey from comment #9)
> the first one not attached I took out STARTTLS which I had the ports wrong
> and then put it in with the attached file with the code posted on screen.

/me is confused - I don't understand what you are telling me.

Comment 11

[Casey]

I think it was the first code I posted on screen which was not an attachment, I had the STARTTLS ports wrong.

And then I posted the attachment with it correct.

Thunderbird seems to be working correct detecting my domain as FastMail and setting the mail servers correct - mail.messagingengine.com

but it is not setting the Sent Folder correct under Copies and Folders,  it sets it as "Sent" -- the correct folder is "Sent Items"

new bug report

https://bugzilla.mozilla.org/show_bug.cgi?id=688963

Comment 12

[Ludovic Hirlimann [:Usul]]

Eric can you confirm that the currently deployed configuration works ?

Comment 13

[Andrew Sutherland [:asuth] (he/him)]

An update on this:
- fastmail self-hosts a number of autoconfig domains using http but not https, ex: autoconfig.fastmail.com, autoconfig.fastmail.com.au
- This self-hosting covers only the domains for which there is an autoconfig.* domain because Thunderbird won't perform self-hosting lookups after an autoconfig lookup.
- The lack of HTTPS support means that FxOS Gaia Mail can't use any of this unless we host on the ISPDB.
- If we add an entry to the ISPDB, custom domains that use MX (but have no autoconfig mapping) will work.

So I've landed the messagingengine.com autoconfig.  I've left port 992 used for the altnamespace option (covered at https://www.fastmail.com/help/technical/servernamesandports.html) and by the fastmail devs in bug 557718 comment 12 since 1) they chose it and 2) bug 557718 and bug 80858 suggest the UX implications of "INBOX." annoy people enough that it's best to stick with that.  (FxOS email doesn't really care.)

Landed:
Committed revision 144001.

Comment 14

[Rob Mueller]

This is really annoying, since we just changed all our server configs (new hostnames, *no* INBOX prefix, path separator changed from "." to "/", regular 993 port), so this is now *wrong*.

https://www.fastmail.com/help/account/securityupgrade.html

> - This self-hosting covers only the domains for which there is an autoconfig.* domain because Thunderbird won't perform self-hosting lookups after an autoconfig lookup.

By default, we have the correct setup for all our domains. e.g.

http://autoconfig.fastmail.com.au/mail/config-v1.1.xml?emailaddress=[e-mail]
http://autoconfig.fastmail.net/mail/config-v1.1.xml?emailaddress=[e-mail]
... etc ...

If you use us for DNS hosting, we automatically create autoconfig.* for all domains as well. e.g.

http://autoconfig.uberengineer.com/mail/config-v1.1.xml?emailaddress=[e-mail]

Looking at your docs:

https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration#Configuration_server_at_ISP

This says nothing about requiring things to be https:// hosted.

But anyway, let me get this right, you're quite happy to do a DNS lookup of the MX records for a domain (completely insecure), but require the site you fetch the data from to be https://. *sigh*

Requiring https just isn't scalable for ISP/domain hosters. Even MS realised this and allowed a http:// -> https:// redirect option in their autodiscover protocol. It's no worse than your automated MX lookup option.

https://technet.microsoft.com/en-au/library/cc511507(v=office.14).aspx

Anyway, since FxOS is dead, I'd prefer if you please just remove "messagingengine.com" from the mozilla ispdb. The fact you even added it without actually ever contacting us is a pretty **** policy as well.

Comment 15

[Andrew Sutherland [:asuth] (he/him)]

I conflated a few too many things in my comment which I think have led to some confusion.  My apologies on that and not contacting you; I'd landed the change as part of an attempt to clean out the backlog of entries in the previously un-maintained ISPDB bug component combined with a selfish desire to easily get autoconfig working for FxOS mail at the same time.  As a happy Fastmail customer, I know how responsive you are and responsible you are.  Most of the other entries in the backlog were for ISPs where self-signed certificates would be an improvement.

Clarifying Thunderbird's behavior:
- If a user enters "bob@DOMAIN" and there's an "autoconfig.DOMAIN", Thunderbird will use it.
- Thunderbird also does some weird domain guessing stuff that's not particularly secure.  Thunderbird can absolutely be 100% owned if a user attempts to run autoconfig on a network where the attacker can spoof DNS responses or perform MITM attacks.  I'm not exactly sure where the domain guessing stuff happens; it might happen after autoconfig lookup.
- If the self-hosted checks don't work out Thunderbird will ask the ISPDB to run a DNS MX lookup over HTTPS.  At original implementation time this was done because Gecko didn't make it possible to perform DNS MX queries and was not part of a threat model.  (And the platform still doesn't expose this.)  Firefox OS did depend on the Mozilla lookup being less vulnerable to attack than the networks FxOS devices were used on as part of its threat model.
- Once Thunderbird has done the MX lookup, it asks the ISPDB again (via HTTPS).  It doesn't look for "autoconfig.FOUND-VIA-MX" on its own.  It's not clear to me why this was done, but it's caused me to greenlight a number of entries into the ISPDB because no one is actively working on Thunderbird's autoconfig code, and usually adding things to the ISPDB does not result in weirdness.
- Thunderbird does require the ISPDB to be served over HTTPS, but that's it.  Firefox OS mail is/was the only one to require autoconfigs to be hosted via HTTPS.

I think this means that any fallout you are seeing is from domains where there is not an autoconfig.DOMAIN entry, but there is a DNS MX entry to messagingengine.com (the only fastmail domain for which the ISPDB reports anything).  In those cases, Thunderbird's autoconfig process would have failed since it had already tried the self-hosted autoconfig.  (Maybe domain guessing happens at this point?)

Unfortunately, as part of the train wreck that is the current ISPDB configuration because of:
- the legacy nature of the server from the days of Mozilla Messaging and
- the scripts that run (the implications of which weren't initially obvious due to a lack of server config documentation)
it's impossible to remove domains proper in their entirety.  I can put in a broken-ish ISPDB entry for messagingengine.com that should pass the script check but fail sanity-checking validation (basically just "<clientConfig><emailProvider><domain>"), or I can update what's there to what's currently served at autoconfig.messagingengine.com/mail/config-v1.1.xml or whatever you prefer.

To reiterate/summarize:
- Sorry!  It seems pretty clear my decision-in-haste here caused at least one long/frustrating day for you, possibly more.
- I can easily update messagingengine.com for you to be correct.  If you're okay with the implications of the "hard to remove" scenario, we can also include more domains.  This may be desirable since Thunderbird will currently only ask the ISPDB about the domain found via MX lookup.  (Unless Thunderbird's autoconfig logic gets enhanced.)
- If it's preferable to I can also try and put a broken entry in there.  I can also remove the entry from revision control afterwards, but I don't think that will entirely clean up the mess we serve.
- I can't remove it entirely, I'm sorry about this, but until Thunderbird moves off of the legacy Mozilla-hosted infrastructure to something else (which is the plan going forward), that's what we're stuck with.  And just to be clear, I am doing all of this in my capacity as a volunteer on the ISPDB.  I am a Mozilla employee, but have not been involved in an email capacity since the shutdown of Firefox OS many months ago.
- The ISPDB is barely maintained.  If you have any interest or know anyone who would have an interest in improving things, Thunderbird and the ISPDB could use all the help they can get.

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

Since it seems like the port 992 configuration we've been exposing via the ISPDB is particularly egregious and I haven't heard anything yet, I've speculatively updated the messagingengine.com entry the ISPDB is hosting to reflect what http://autoconfig.messagingengine.com/mail/config-v1.1.xml is hosting.  Changes should propagate within 30 minutes.

As noted, there are other things we can try and do, but ideally this makes things less bad.  (Noting that once Thunderbird creates an account, it will never do anything like check the ISPDB for updates, so this should not cause flapping, etc.)

$ svn commit
Sending        messagingengine.com
Transmitting file data .
Committed revision 150735.

I'm going to leave this open since it's clear we haven't addressed this to fastmail's satisfaction.

Comment 17

[Rob Mueller]

Hi. Sorry for my rather ranty comment, I believe I overreacted here. I *thought* (based on some limited testing) that the messagingengine.com ISPDB entry was overriding our http://autoconfig.*/... stuff, but I now think I was looking at a domain where we *don't* host DNS for the domain, only the MX servers for the domain. Obviously that means the autoconfig.* DNS wasn't set up for the domain, so of course it was falling back to ISPDB. Oops.

Based on what you've said though, things should now actually be in a good position.
1. All our domains (e.g. fastmail.com, eml.cc, etc) should work, because you check http://autoconfig.*/...
2. All user domains that host their DNS with us should work, because you check http://autoconfig.*/... and by default we server the correct DNS for those subdomain names
3. All users domains that just host their MX records with us should work, because you'll do an MX lookup, find the base hostname of "messagingengine.com" and look that up from the ISPDB. OT, would be nice if you checked http://autoconfig.$base-mx-domain, but it sounds like that's unlikely given the code hasn't been touched in quite some time

Anyway, thanks for updating the ISPDB config to the new values, that at least fixes the immediate problem with user domains only using us for MX hosting.

Comment 18

[Andrew Sutherland [:asuth] (he/him)]

No worries.

(In reply to Rob Mueller from comment #17)
> OT, would be nice if
> you checked http://autoconfig.$base-mx-domain, but it sounds like that's
> unlikely given the code hasn't been touched in quite some time

Yes, I wouldn't count on this happening anytime soon.  Although there has been some activity on bug 971347 to upstream the Tor Thunderbird fork changes which impact autoconfig, I think those changes are more concerned about making it possible to require HTTPS.

> Anyway, thanks for updating the ISPDB config to the new values, that at
> least fixes the immediate problem with user domains only using us for MX
> hosting.

Okay, I'm going to mark this fixed again, but feel free to comment on this bug/reopen in the future, or file a new bug in the same component if updates/changes are required.