551519 - Consider DNS MX in autoconfig

Status: RESOLVED FIXED

(Thunderbird :: Account Manager, enhancement)

Description

[Ben Bucksch (:BenB)]

When we try to find a configuration for an email address, we currently only look at the domain in the email address.
This won't work for domain hosters which host many customer domains and do not publish the config themselves (see bug 534722).
For those, we are likely to find a config, if we query based on the domain the DNS MX record (the server which receives mail for that domain). Due to SSL limitations, it typically has the hostname in the domain of the hoster, not the customer.

So,
If all lookups apart from the heuristic failed,
1. look up DNS MX <domain-in-emailaddress>, get e.g. mx1.dreamhost.com
2. take the SLD only, e.g. dreamhost.com
   (attention for .co.uk - Mozilla/Firefox has a list of those)
3. start the lookup again from the beginning, for that new domain,
   including ISP fetch, Mozilla DB.
4. I'm not sure whether we should start the heuristic on the original domain or the MX domain, or both. In favor of a faster probe and/or limiting the amount of connections, we should be careful before doing both.

Comment 1

[Ben Bucksch (:BenB)]

Re using MX as incoming/outgoing mail server:
Somebody suggested adding the MX host as one of the hosts to probe in the heuristic. While that would be very useful in small companies with strange mail server names (like gregory.example.com), it is deceptive: We would always "find" an SMTP server on port 25, but that server may not act as outgoing SMTP server, only incoming. We'd *appear* to have found a working SMTP server, but it wouldn't work when actually trying to send mail to email addresses outside the domain. And there's no way for us to find out before actually trying to send such a mail. Too dangerous.

Comment 3

[sibul]

Sorry for duplicate bug 552989.

As BenB suggested there:
*quote*
Note that mail.example.com should be searched before / preferred over
example.com by Thunderbird. You'd run into the problem only if mail.example.com
was not resolvable from where you are, or if the host is actually called
fred.example.com instead of mail.example.com.
*/quote*

This would work like 99.9% (IMHO) of the time too. In real world environments there is always a *.example.com record pointing to somewhere valid. People seperating their web and mail services know what they are doing there and if it's publicly available in wiki or somewhere, people would set pop/imap.example.com, smtp.example.com or simply mail.example.com additionally to their real dns names.

Would be even more simple than MX Record lookups. Roundrobin would work too.

I'd vote for this real simple solution.

Cut example.com from [e-mail] , add mail. infront.

(I would also not start searching automaticly, I'd wait for the user to push the autolookup button - there would be space enough to add the suggested config file lookup button which would be realy cool also!)

1. Try to reach mail.example.com for pop/imap and smtp
1.1 Can reach = Yes - Well done, do the rest of the checks
1.2 Can't Reach goto 2.

2. Check for smtp/pop/pop3/imap/imaps.example.com
2.1. Can reach = Yes - Well done, keep checking the rest
2.2 Can't reach goto 3

3. Check example.com for servers
3.1 Can reach = Yes - Well done, yatta..
3.2 Can't reach - Alert User

That's 3 lookups where the first one usualy will match anyways.

- Sibul

Comment 4

[Ben Bucksch (:BenB)]

sibul, we already do that since TB 3.0.

Comment 5

[sibul]

Why isn't this working for me than?

Comment 6

[Ben Bucksch (:BenB)]

I don't know, but it's not subject of this bug.

Comment 7

[Ben Bucksch (:BenB)]

See bug 545866 for ability to query DNS MX at all in Mozilla.

Comment 8

[Serge Gautherie (:sgautherie)]

This should depend on bug 545866, not block it, shouldn't it?

Comment 9

[Ben Bucksch (:BenB)]

Correct.

Comment 10

[Ben Bucksch (:BenB)]

Attachment: Fix using web service, v1

This is a short-term solution, using the web service from bug 491543 comment 5, until we have bug 545866 fixed in Mozilla. (When that happens, we will replace the implementation of function getMX() with one using that.)

This seems to work, but I don't have any test domain. bucksch.eu should work, finding Yahoo as mail provider, once the DNS caches expired (and until I change it back to normal DNS MX).

Comment 11

[Ben Bucksch (:BenB)]

Attachment: Fix using web service, v2

Fix typos

Comment 12

[Ben Bucksch (:BenB)]

If necessary, I can avoid the string changes, and just use "_db" strings.

Comment 13

[Philippe M. Chiasson (:gozer)]

Carefull here, I wouldn't want a released product to be talking to http://ectoplasm.org/bin/mx/ by default. It's my own little box, and shouldn't be relied on.

I suggest simply hosting that feature on live.mozillamessaging.com instead.

https://live.mozillamessaging.com/dnslookup/example.com/mx

Comment 14

[Ben Bucksch (:BenB)]

gozer, of course, I wouldn't have led that into the product. As said, I wanted to make a patch first and see who (and when!) the review goes, and the change of pref is then trivial once you have the service running on a momo server.

Comment 15

[Philippe M. Chiasson (:gozer)]

(In reply to comment #14)
> gozer, of course, I wouldn't have led that into the product. As said, I wanted
> to make a patch first and see who (and when!) the review goes, and the change
> of pref is then trivial once you have the service running on a momo server.

Ah, okay, good, and it's fine to use ectoplasm.org to give something that works for the reviewers indeed.

I've filed bug 560315 to track running an improved version on momo.com

Comment 16

[Ben Bucksch (:BenB)]

FYI, this solves Google Apps, too (if we host a config file for "google.com").

Comment 17

[Blake Winton (:bwinton) (:☕️)]

Comment on attachment 439908 [details] [diff] [review]
Fix using web service, v2

>+++ b/mailnews/base/prefs/content/accountcreation/emailWizard.js
>@@ -485,20 +485,33 @@ EmailConfigWizard.prototype =
>               function(e) // fetchConfigFromDB failed
>               {
>                 gEmailWizardLogger.info("fetchConfigFromDB failed: " + e);
>+                me.startSpinner("all", "looking_up_settings_mx");
>+                me._probeAbortable = fetchConfigForMX(domain,
>+                  function(config) // success
>+                  {
>+                    me.foundConfig(config);
>+                    me.stopSpinner("found_settings_mx");
>+                    me.showEditButton();
>+                    me._probeAbortable = null;
>+                  },
>+                  function(e) // fetchConfigForMX failed
>+                  {
>+                    gEmailWizardLogger.info("fetchConfigForMX failed: " + e);
>+                    var initialConfig = new AccountConfig();
>+                    me._prefillConfig(initialConfig);
>+                    me.startSpinner("all", "looking_up_settings_guess")
>+                    me._guessConfig(domain, initialConfig, "both");
>+                  });

Hmmm…  I think we want the mx-ified host to go through the whole Disk/ISP/DB cycle, to give the smaller meta-ISPs a chance to give the settings.  (It _should_ be a fairly easy change, since we are just doing the same things to the mx-ified host as we would to the base host, right?)

>+++ b/mailnews/base/prefs/content/accountcreation/fetchConfig.js
>@@ -137,8 +142,106 @@ function fetchConfigFromDB(domain, succe
>+kDebug = true;

I think this should probably be set to false.  ;)  Or better, deleted, since it looks like it's not used.

>+  var sucAbortable = new SuccessiveAbortable();
>+  var time = Date.now();
>+  sucAbortable.current = getMX(domain,
>+    function(mxHostname) // success
>+    {
>+      ddump("getmx took " + (Date.now() - time) + "ms");
>+      var tldServ = Cc["@mozilla.org/network/effective-tld-service;1"]
>+          .getService(Ci.nsIEffectiveTLDService);

Could you line this up with the [, please?

>+      var sld = tldServ.getBaseDomainFromHost(mxHostname);
>+      ddump("base domain " + sld + " for " + mxHostname);
>+      sucAbortable.current = fetchConfigFromDB(sld,
>+          successCallback, errorCallback);

And this seems like it can be split as:
>+      sucAbortable.current = fetchConfigFromDB(sld, successCallback,
>+                                               errorCallback);
which reads nicer to me.

>+++ b/mailnews/mailnews.js
>@@ -843,16 +843,18 @@ pref("mail.compose.max_recycled_windows"
>+// bug 551519

Should that be bug 545866?

>+pref("mailnews.mx_service_url", "http://ectoplasm.org/bin/mx/");

And that should definitely be https://live.mozillamessaging.com/dns/mx/

I'm going to give it an r=me with the nits fixed, but I would like to take a glance over the fix for the first change I requested before you commit it.

Also, please file a bug to add the google.com domain, so that we can get all those sweet Google Apps users.  :)

Thanks,
Blake.

Comment 18

[Blake Winton (:bwinton) (:☕️)]

Oh, two more things that I just remembered.

1) It would be kind of cool to see this in TB3.1, so you should chat about late-l10n with a driver, or post a patch which doesn't add strings.

2) If the mx is the same as the domain (i.e. latte.ca, or ascher.ca), then we should just skip the whole mx check.

Later,
Blake.

Comment 19

[Ben Bucksch (:BenB)]

> (It _should_ be a fairly easy change, since we are just doing the same
> things to the mx-ified host as we would to the base host, right?)

Not really, because I'd need to copy all those calls.
I probably should change the strings, too - I could do without, but then the user would see the same twice.

Last but not least, it increases the lookup time. The autoconfig server lookup is 2 network calls, which take 1-2s (on a fast connection), in addition to the ISPDB call. I'd like to keep it fast.

> I think we want the mx-ified host to go through the whole Disk/ISP/DB
> cycle, to give the smaller meta-ISPs a chance to give the settings.

Well, disk is not that important, as it's mostly for companies, and they can just add the 1-2 domains they use.

For autoconfig server fetch, they can serve the customer domains, too, all with one server and without any addition a configuration apart from an DNS entry at the customer domain.
What it would save ISPs is to add the A autoconfig.customer.com DNS entry at the customer DNS. But that's at the cost of us having to do more lookups and falling back to this MX lookup, which is less clean than a proper autoconfig, so I don't want to encourage this "shortcut" (which means longer and less certain way for us). Most ISPs control the domain anyways. Then again, there are domains which they don't control. So, I am uncertain about this, but for the moment, I think we shouldn't make the autoconfig server check and just check the ISPDB.

In other words, either the ISP supports us, then they have an autoconfig server and set up DNS at the customer domains, or they don't support us and then we use ISPDB. But not a combination of both. Which means people who run their own DNS, but not their own mail server, fall on the wayside - I hope they are not that many.

> please file a bug to add the google.com domain, so that we can get all
> those sweet Google Apps users.  :)

Bug 561918.

(How do you know that Google Apps users are sweet, Mr. Lecter?)

> 2) If the mx is the same as the domain (i.e. latte.ca, or ascher.ca), then we
> should just skip the whole mx check.

I already do that, but I guess I didn't attach that newer patch yet. Will include, and good catch anyways!

Comment 20

[Blake Winton (:bwinton) (:☕️)]

I think we could refactor the code to make it easier for you to copy, but I'm satisfied with your reasoning for not adding it yet.

Comment 21

[Ben Bucksch (:BenB)]

Attachment: Fix, v4 - webservice version, no strings - commited

- Fixed review nits
- (Had already) Added check to not make the same lookup twice
- Removed new strings (string freeze), using ISPDB strings for now.

Ready for commit

Comment 22

[Ben Bucksch (:BenB)]

Tested and works nicely for [e-mail] (a random Google Apps domain).
Just waiting for tree to open.

Comment 24

[Ben Bucksch (:BenB)]

Comment on attachment 442780 [details] [diff] [review]
Fix, v4 - webservice version, no strings - commited

Commited as <http://hg.mozilla.org/comm-central/rev/7e399e08ea8d>

Thanks for the review!

Leaving bug open for the proper fix with client-side DNS lookup, based on bug 545866.

Comment 25

[Ben Bucksch (:BenB)]

Requesting approval for 3.1

Comment 26

[Ben Bucksch (:BenB)]

Per request, filed followup bug 563958 about the client-side DNS lookup change.
Marking FIXED.

Comment 27

[Ben Bucksch (:BenB)]

Comment on attachment 442780 [details] [diff] [review]
Fix, v4 - webservice version, no strings - commited

Requesting approval for 3.1.

Rationale:
This is expected to considerably increase our detection rate of the correct config. Esp. for small companies with domains hosted at hosters like 1&1, Dreamhost, Fastmail, Google Apps and similar.  With 100000 new setups per day, and e.g. 10% hit increase, that means helping 3 million setup per month (!).

This is fairly simple feature, low risk, with a very high gain.

The patch does not include tests, because it's all about integration with ISPs, so creating a test would be very hard to create.

Comment 28

[David Ascher (:davida)]

Couldn't we use one of our own domains setup as an "ISP" and have tests work against accounts on it?

Comment 29

[Ben Bucksch (:BenB)]

I'd really like to make tests like that, but we're not allowed to use the network in those tests.
(I think we need such a testsuite, possibly also based on xpcshell-tests, which may use Internet services, so that we can do integration tests. I could test against autoconfig, guess config, test IMAP stack against a real Cyrus, a real ISP etc., and we could notice when gmail or web.de IMAP servers break for us - something we'd want to know. But we have no such test suite at the moment.)

Comment 30

[Dan Mosedale (:dmosedale, :dmose)]

Comment on attachment 442780 [details] [diff] [review]
Fix, v4 - webservice version, no strings - commited

a=dmose, conditional on adding a litmus test case first.

Comment 31

[Ben Bucksch (:BenB)]

I'm told I can't add them myself, but here's the description:
* menu File | New | Mail Account...
* Enter:
  Your name: "Litmus"
  Email address: "[e-mail]"
  Password: "litmus"
* Click "Continue"
* Click "Manual Config" (or Advanced Config)

You should see:
IMAP server: imap.googlemail.com
SMTP server: smtp.googlemail.com

I might later add a test for a domain at another web hoster.

Comment 32

[Ben Bucksch (:BenB)]

> You should see:
And you should *not* see:
"Trying common server names" (or similar)

Comment 33

[Ben Bucksch (:BenB)]

Thanks! whooo.
Commited to 3.1 branch as <http://hg.mozilla.org/releases/comm-1.9.2/rev/4a7e888fb90e>

Comment 34

[Ludovic Hirlimann [:Usul]]

https://litmus.mozilla.org/show_test.cgi?id=11815