1500105 - (owl) Exchange AutoDiscover and addon setup in account creation dialog

Status: RESOLVED FIXED

(Thunderbird :: Account Manager, enhancement, P1)

Description

[Ben Bucksch (:BenB)]

The Thunderbird account creation dialog that automatically finds the email server configuration, but we so far don't support Exchange.

Support the AutoDiscover method of Exchange servers. It allows to find the Exchange server hostname, which is typically a very odd local name like mex09.example.com that cannot be guessed.

The AutoDiscover returns an XML document that describes various protocols that are supported. IMAP may be one of them. If so, offer an IMAP config.

If the server only supports Exchange-only protocols, offer to install a Thunderbird addon that supports these protocols, and (if accepted by the user), then use this to set up the account.

Comment 1

[Ben Bucksch (:BenB)]

Specs:
https://technet.microsoft.com/en-us/library/bb124251(v=exchg.160).aspx#Autodiscover%20services%20in%20Outlook
https://docs.microsoft.com/en-us/previous-versions/office/developer/exchange-server-interoperability-guidance/hh352638(v%3Dexchg.140)
https://www.msxfaq.de/exchange/autodiscover/autodiscover_xml.htm

Comment 3

[Ben Bucksch (:BenB)]

Attachment: Automatically find configs for Exchange servers and sets up addon, in account creation dialog

* Exchange AutoDiscover protocol implementation
* Try to find IMAP servers in the result, or based on the hostname from there
* Offer Owl extension which supports the Exchange OWA protocol to get mails
* OWA is supported by most Exchange installations, while other protocols
  are often not enabled against the Internet.

Comment 4

[Ben Bucksch (:BenB)]

Attachment: Support Exchange AutoDiscover and parallelize network calls

Run all the ISP config lookup network calls in parallel. Then, after they return, choose the best config.

Class ParallelAbortable simply runs all calls in parallel. Class PriorityOrderAbortable also implements a policy that waits until one of of the better calls returns successfully, then takes that result and cancels all lesser desirable calls that are still pending.

That means e.g. the MX call call already start at the very beginning, but if the normal DB lookup returns (which is better), and the MX call hasn't finished yet, we'll abort the MX callm and use the result from the normal DB call.

Comment 5

[Ben Bucksch (:BenB)]

TODO:
* Server fetches trigger password prompt, they should not. Possibly related to redirects.
* Styling and sizing of addon [Install] button and text
* Smaller issues

Comment 6

[Ben Bucksch (:BenB)]

> * Server fetches trigger password prompt, they should not

Fixed

Comment 7

[Ben Bucksch (:BenB)]

This can be tested with an hotmail/live/outlook.com account, but to test with outlook.com or Rackspace, you need to set a few prefs in your Thunderbird test profile, because Outlook is in the ISP DB and we prefer that.

mailnews.auto_config.guess.enabled = false
mailnews.auto_config_url = ""
mailnews.mx_service_url = ""
security.pki.distrust_ca_policy = 0

Rackspace is a big hoster and therefore in our ISP DB. we prefer configs from ISB DB, so my new code doesn't even trigger. you need to disable the ispdb etc. in a real world scenario, the exchange server would be a small corporate server and not in the ISP DB. So, the prefs simulate this.
(Finally, the CA pref is for a Rackspace server with an old Symantec cert, which Firefox trunk already distrusts, but Firefox release not yet.)

Comment 9

[Ben Bucksch (:BenB)]

Comment on attachment 9018521 [details]
Support Exchange AutoDiscover and parallelize network calls

Neil r+
aceman r+

Comment 10

[Magnus Melin [:mkmelin]]

Since this contains two (or more) separate issues, could you make them separate patches? One for detecting exchange, and one to figure out the path from there.

As it stands the second part is not acceptable, and will need more work. Those iterations could be more easily done as a separate patch.

Comment 11

[Ben Bucksch (:BenB)]

All review requests are implemented.

Comment 12

[Magnus Melin [:mkmelin]]

Is there some public server somewhere that requires owl? I mean something so I can try it out in practice.

Comment 13

[Ben Bucksch (:BenB)]

@Magnus: see comment 7.

Comment 14

[Magnus Melin [:mkmelin]]

Ok, I thought I'd seen something about that, but couldn't find it earlier. 
But no known services to test with without setting prefs?

Comment 15

[Ben Bucksch (:BenB)]

No, because I tried very hard to have a config for all public ISPs in the ISPDB. Our coverage (of public ISPs) is too good :).

Comment 16

[Ben Bucksch (:BenB)]

Attachment: Support Exchange AutoDiscover and parallelize network calls - TB 60 backport

* Parallelize network calls
* Exchange AutoDiscover protocol implementation
* Try to find IMAP servers in the server response
* Offer to install an extension which supports the Exchange protocol to get mails

Runs all the ISP config lookup network calls in parallel.  Class PriorityOrderAbortable (subclass of ParallelAbortable) implements a policy that waits until one of the calls returns successfully, then takes that result and cancels all pending less desirable calls.

Implements the Exchange AutoDiscover protocol to detect Exchange servers. If the server gives an IMAP configuration, we offer that to the user. Alternatively, we offer a compatible verified extension that implements the specific Exchange protocol that the Exchange server returned. Exchange has at least 4-5 proprietary protocols, and we show extensions that support the protocols that the server listed and that are known to work well and actively maintained. The setup process then continues without interruption.

Comment 17

[Jorg K (CEST = GMT+2)]

Last try run before landing this to avoid disappointment:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=57e7423a88d7373264c82862dbcfb0a04d3bdfe0

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1403c0fa3d1a
Support Exchange AutoDiscover and parallelize network calls. r=aceman,mkmelin,Neil

Comment 19

[Jorg K (CEST = GMT+2)]

As we already know, the Germans can't speak German any more, let alone write it:

"Eule ist eine Erweiterung von einem Drittanbieter, die Ihnen erlaubt, Exchange server zu benutzen. Sie ist kostenpflichtig. Die Preise finden sie auf der Website."

Two mistakes in three sentences :-(

Compound nouns in German are written in either one word or with hyphens: Exchange-Server. The other mistake is obvious. It's only a comment, so not user facing.

Comment 20

[Ben Bucksch (:BenB)]

Fixed. (The production version already had "Exchange-Server".)

Comment 21

[Jorg K (CEST = GMT+2)]

Fixed where?

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/480eb7e81d39
Follow-up: Fix typos in German text in comment. rs=typo-fix-only DONTBUILD

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/2a6b32ef3fb6
[autoconfig] Make domain bold. r=aceman

Comment 24

[Jorg K (CEST = GMT+2)]

Grrr, that landed with the wrong bug number:

https://hg.mozilla.org/comm-central/rev/844f685dc9560a2b857c30a75d72df30f43e8930
Backed out changeset 2a6b32ef3fb6 for landing with the wrong bug number. a=backout

Comment 25

[Jorg K (CEST = GMT+2)]

Comment on attachment 9018521 [details]
Support Exchange AutoDiscover and parallelize network calls

Should get this into beta eventually.

Comment 26

[Jorg K (CEST = GMT+2)]

Beta (TB 65):
https://hg.mozilla.org/releases/comm-beta/rev/60a1cefdf2afe240abc3558542a8237276e7f000 (merged patch incl. typo fix)

Comment 27

[Jorg K (CEST = GMT+2)]

Beta backout:
https://hg.mozilla.org/releases/comm-beta/rev/195a2aef72e1ab507c07ca93e703600b4b09c3b3

Comment 28

[Jorg K (CEST = GMT+2)]

Comment on attachment 9031094 [details]
Support Exchange AutoDiscover and parallelize network calls - TB 60 backport

You should ask for approval and not leave the management to me. These flags are vital, without them, the uplift won't happen.

Comment 29

[Jorg K (CEST = GMT+2)]

TB 65 beta 2:
https://hg.mozilla.org/releases/comm-beta/rev/42dfab681c409905c0d84cd86d100452cd603e1a (merged patch incl. typo fix)

Comment 30

[Jorg K (CEST = GMT+2)]

You're making this a little painful :-(

Ben told me to get the patches from this try run:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=a141ab4f4e83bdaba447e4a0a330fc61311821b3

So I'm comparing D14408 here to https://hg.mozilla.org/try-comm-central/rev/df29a6d4527c977595099ce46e6dd902d2ca1599 and they are quite different :-( - Most strikingly, the try changeset has string changes:
mail/locales/en-US/chrome/messenger/accountCreation.dtd
mail/locales/en-US/chrome/messenger/accountCreation.properties

Please provide proper HG patches or somewhere to get them from. This try run can't be it.

I'll stop here now since if I can't land the base patch, there's nothing to do right now.

Comment 31

[Jorg K (CEST = GMT+2)]

And can you please provide all patches with 8 lines context so I can compare them. It's tedious to apply them, refresh them, then apply the other one, refresh it and then finally compare :-(

Comment 32

[Ben Bucksch (:BenB)]

Most strikingly, the try changeset has string changes

Yes, that's a mistake, D14408 does not, Neil must have picked the wrong patch. Sorry that I didn't catch that.

Comment 33

[[e-mail]]

I'm sorry for my mistake, I'll redo the Try run with the correct patch this time...

Comment 34

[[e-mail]]

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=31538f4ad208dce05f259a8ae38a74c9b5a4b118

Comment 35

[Ben Bucksch (:BenB)]

I'm testing it, and I've got an error. Investigating.
Please don't merge yet.

Comment 36

[Ben Bucksch (:BenB)]

Attachment: Fix, v38, for TB60

I found a small problem with an API change: Services.locale.getRequestedLocale() on ESR60 instead of .requestedLocale on trunk. Otherwise, it works.

I've tested it and it works as expected.

Comment 37

[Ben Bucksch (:BenB)]

Attachment: Fix, v38, for TB 60

(with HG header, manually created, I hope it works)

Comment 38

[Ben Bucksch (:BenB)]

Attachment: Fix, v38, TB60

(...and -U8 context 8 lines)

Comment 39

[Jorg K (CEST = GMT+2)]

(In reply to Ben Bucksch (:BenB) from comment #36)

I found a small problem with an API change: Services.locale.getRequestedLocale() on ESR60 instead of .requestedLocale on trunk. Otherwise, it works.

I knew that ;-) - How's the state of the other patches? I can see two try runs
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=31538f4ad208dce05f259a8ae38a74c9b5a4b118
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=eeb49d8b015520455369ddf697b2fd0e86a646d6

BTW, they do include the "bold" bug.

Comment 40

[Ben Bucksch (:BenB)]

they do include the "bold" bug.

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=eeb49d8b015520455369ddf697b2fd0e86a646d6
does not include the "domain bold" patch. I've verified that in:
https://hg.mozilla.org/try-comm-central/file/eeb49d8b015520455369ddf697b2fd0e86a646d6/mail/components/accountcreation/content/emailWizard.js#l965
It does include the patch for this bug, even though it does not show in the list, because the identical patch has in the first try build and hg then doesn't show it, but it's in this try build:
https://hg.mozilla.org/try-comm-central/file/eeb49d8b015520455369ddf697b2fd0e86a646d6/mail/components/accountcreation/content/exchangeAutoDiscover.js

The domain bold bug is optics only, has nothing to do with Exchange, and is optional, we can leave it out.

---

I've tested all the patches (aside from the domain bold patch) together in a build, ran my standard manual tests, and they all work fine, as far as I could see.

Comment 41

[Jorg K (CEST = GMT+2)]

My mistake, ...b118 had the bold patch, and for ....46d6 I incorrectly assumed I had, too. Sorry.

Comment 42

[Jorg K (CEST = GMT+2)]

OK, so for the record: We go with the ESR60 patch here, plus the five changesets from
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=eeb49d8b015520455369ddf697b2fd0e86a646d6
As discussed, they don't include the "bold" thing ;-)

Right?

Comment 43

[Ben Bucksch (:BenB)]

Right.

Once you merged it, we can test the builds.

Comment 44

[Jorg K (CEST = GMT+2)]

Comment on attachment 9037632 [details] [diff] [review]
Fix, v38, TB60

Thanks, I compared to Neil's try, only difference: The correction of the locale thing.

Comment 45

[Jorg K (CEST = GMT+2)]

TB 60.5.0 ESR:
https://hg.mozilla.org/releases/comm-esr60/rev/c728c1579f04d8c596c1224dc62ef1e6cd39cae6

Comment 46

[Ben Bucksch (:BenB)]

Thanks so much for merging this! This is a huge relieve for me!

I've downloaded the TB 60.5 build
treeherder: https://treeherder.mozilla.org/#/jobs?repo=comm-esr60&revision=acaf08e72ead14783d06d0cd8f14431b31ba3b97&selectedJob=223017085
Linux: https://queue.taskcluster.net/v1/task/DCetzb2GRWqQT5A_x3LK9Q/runs/0/artifacts/public/build/target.tar.bz2
and tested it, and it all works as expected.

Most accounts (even outlook.com etc.) have IMAP configs in our ISPDB, and they are still give IMAP configs as expected.
To emulate hosted on-premise Exchange servers for small domains, I disabled the ISPDB (see comment 7), and then we detect the AutoDiscover and everything works as expected. I tried 5 Exchange accounts on different servers, and they all were properly detected, automatically configured, and worked.

If you do happen to find something catastrophically wrong, please add it to bug 1514627 (alias exchange-meta). I will stand by in ready mode and will try to fix any severe and real problems that might be found right away.