autoconfig offers Exchange (and selects it by default!) even if standard protocols IMAP/SMTP setup is found and available (Office 365)
Categories
(Thunderbird :: Account Manager, defect)
Tracking
(thunderbird_esr68 fixed, thunderbird72 fixed, thunderbird73 fixed)
People
(Reporter: mkmelin, Assigned: BenB)
References
(Regression)
Details
(Keywords: regression)
Attachments
(2 files, 23 obsolete files)
|
181.90 KB,
video/mp4
|
Details | |
|
4.85 KB,
patch
|
mkmelin
:
review+
mkmelin
:
approval-comm-beta+
mkmelin
:
approval-comm-esr68+
|
Details | Diff | Splinter Review |
account creation.png
Bug 1571772 apparently changed how the autoconfig works wrt. offering Exchange and thus promoting the Owl add-on.
Now, even if IMAP and SMTP setups are found, the autoconfig will list Exchange as an option and select that by default. This is clearly not how this was supposed to be handled, and not how the flow was agreed to be handled when we discussed Owl integration points last year. We're not interested in listing a proprietary protocol that requires a commercial add-on to be installed, and certainly not interested in having it be the default. Exchange support should only be listed as a last resort, if no other means of connecting are available.
Besides obvious problems from setting unsuspecting users to use a proprietary protocol with unknown future support, it is also confusing to users that didn't realize they had no reason to buy an add-on to get their mails, and then Thunderbird still asked for a donation as the application download page. (A complaint in this regard is what lead me to discover this bug.) => Bad rep for Thunderbird.
STR: try to set up an @outlook.com address. You'll get set up to use Exchange as default.
The fact that this change was implemented + reviewed by the two people working the Owl add-on certainly doesn't make this any prettier! :(
|
||
Comment 1•5 years ago
|
||
I'm rather disappointed by this report. I see Neil and Ben work together and I always assumed they had enough peer status for that to be OK. I guess in the future everything will need to be reviewed by a "real" peer. So how do we fix this? Is there a quick fix we can ship in 68.2.1 soon or should we backout the function at least from the ESR?
|
|
||
Comment 2•5 years ago
|
||
Also, there was no attention paid to detail. This dialogue comes up and it is too small and the buttons are cut off :-(
|
Reporter | |
Comment 3•5 years ago
|
||
Backout patch.
|
|
Reporter | |
Comment 4•5 years ago
|
||
|
|
Reporter | |
Updated•5 years ago
|
|
|
||
Comment 5•5 years ago
|
||
Hmm, yes, backing out 2f10eda9d3a70a2d15a19ce80a066c7e7a0e3459 from bug 1592258 failed miserably. Just keep in mind that backing this out will only reach ~8% of TB users. The function is also in TB 60.9.
Wouldn't it be easier to just set the option to IMAP instead?
|
|
||
Comment 6•5 years ago
|
||
|
Assignee | |
Comment 7•5 years ago
|
||
|
|
Assignee | |
Comment 8•5 years ago
|
||
Exchange support should only be listed as a last resort, if no other means of connecting are available.
I certainly think that POP3 is an inferior protocol, nevertheless we still offer it, and people still choose it. Likewise, if we know that this is an Exchange server, and Owl is specifically written to support Exchange (and works etc.), then I see no reason why it should be hidden from the user. Exchange support should always be available. Why refuse an available option to the user that works really well?
I know that you, Magnus, are not personally affected, but if you had an Office365 email account in a large company, you probably would be. A large percentage of Office365 users cannot use IMAP anymore. Their companies are forcing MFA (multi-factor authentication) and are actively disabling "legacy protocols" (which is what Microsoft calls IMAP, POP3 and SMTP), because they are password based. Everybody is blogging and writing about how MFA helps security and it should really be used. More and more users are writing to our support that they are in this situation. It appears this has been rapidly increasing even in the last 2 months. Just in the last few weeks, I see more and more people reporting that in our support requests. A large percentage of Office365 are simply locked out of IMAP.
Microsoft is actively trying to lock out password-based protocols in favor of what they call "Modern Authentication" and MFA. You can expect this percentage of users to rapidly increase soon.
In the autoconfig dialog, we have no good way to know whether people are in this situation that they need MFA. Owl has specific support for it, but this is very Office365-specific and works only for Office365. Additionally, Thunderbird doesn't communicate the real reason why an IMAP login failed, and just says "Password wrong". This is very bad. Most end users will just try other passwords and give up frustrated, and will not know that it fails due to MFA. Selecting Exchange makes the configuration work by default, and still allows users to choose another protocol, if they like.
It is selected by default also for another reason: We can offer much tighter support with Office365 than standard protocols do. We integrate with the address book and the calendar (EWS now, OWA in the coming weeks). For many users, the calendar is really important, and they keep asking about it.
|
|
Assignee | |
Comment 9•5 years ago
|
||
r-, because this will break configurations that work now and will not work anymore after this backout.
Furthermore, this patch improved the code, as described in bug 1571772, where it landed. You can also see in that bug that there was a real review with a real discussion.
It even was mentioned in the release notes, and was covered by many news articles (!) in several countries as the primary improvement of that particular release, and the press mentioned it in a positive way as an improvement.
I understand now that you're against it for philosophical reasons, but if you take the facts of how the general userbase (with that I mean the average Thunderbird user) and the press and news articles received it, it was overwhelmingly positive. I've received many letters of thanks for Owl, too.
Ripping that feature out again now, or even removing it from being the default, would be a regression. That would be bad PR. It would make no sense, given that the feature was received well.
|
|
Assignee | |
Comment 10•5 years ago
|
||
The fact that @outlook.com addresses also default to Exchange is not intentional, but a consequence of the Microsoft server setup. They have the same MX server (which Thunderbird bases on) and look the same to Thunderbird. They also work in Owl, but you're right that these users could just as well use IMAP. I think for all outlook.com users, IMAP is enabled.
As a constructive proposal, what we could do is try to differentiate between outlook.com - which typically works with IMAP, and default that to IMAP -, and Office365 (the corporate cloud email addresses) - which often require MFA and will simply not work with IMAP, but will work with Owl, and default the latter to Owl. We could come up with a patch that does this.
|
|
Assignee | |
Comment 11•5 years ago
|
||
@Magnus: Is comment 10 a fair proposal? If you agree, we will take this bug and create a patch.
|
|
||
Comment 12•5 years ago
|
||
Sounds good to me. The backout is not a viable option and I'm quite convinced that not offering IMAP for Office365 also makes sense. Users who don't want to buy the add-on can click IMAP and see how they go.
Bonus points for improving the "Password wrong" message when IMAP fails due to MFA.
|
|
||
Comment 13•5 years ago
|
||
And more bonus points for making the dialogue big enough ;-)
|
|
Reporter | |
Comment 14•5 years ago
|
||
(In reply to Ben Bucksch (:BenB) from comment #8)
I certainly think that POP3 is an inferior protocol, nevertheless we still offer it, and people still choose it. Likewise, if we know that this is an Exchange server, and Owl is specifically written to support Exchange (and works etc.), then I see no reason why it should be hidden from the user. Exchange support should always be available. Why refuse an available option to the user that works really well?
POP3 has a different use case than IMAP. People can have good reasons to choose it (e.g. not wanting to leave anything on the server to be snooped, or due to storage limitations).
I'm not trying to "refuse an option that works really well". As you're very well aware, what we agreed upon back then is that Exchange would be shown when there were no standard compliant options available. I think you'd have severe difficulties convincing users that they have to pay Beonex 10€/year to access their email, when they could have been doing it all for free, with less hazzle, and is way more future proof.
I know that you, Magnus, are not personally affected, but if you had an Office365 email account in a large company, you probably would be. A large percentage of Office365 users cannot use IMAP anymore. Their companies are forcing MFA (multi-factor authentication) and are actively disabling "legacy protocols" (which is what Microsoft calls IMAP, POP3 and SMTP), because they are password based. Everybody is blogging and writing about how MFA helps security and it should really be used. More and more users are writing to our support that they are in this situation. It appears this has been rapidly increasing even in the last 2 months. Just in the last few weeks, I see more and more people reporting that in our support requests. A large percentage of Office365 are simply locked out of IMAP.
You're making a lot of assumptions about me and also about what what users want or not. Surely if they really need Exchange support they can install Owl and get that.
Microsoft is actively trying to lock out password-based protocols in favor of what they call "Modern Authentication" and MFA. You can expect this percentage of users to rapidly increase soon.
FYI for O365, the IMAP and POP MFA support is in the works.
In the autoconfig dialog, we have no good way to know whether people are in this situation that they need MFA. Owl has specific support for it, but this is very Office365-specific and works only for Office365. Additionally, Thunderbird doesn't communicate the real reason why an IMAP login failed, and just says "Password wrong". This is very bad. Most end users will just try other passwords and give up frustrated, and will not know that it fails due to MFA.
At that point, if Exchange was available, AND the normal IMAP/POP auth failed: that would be the appropriate place to advertise this alternative.
Selecting Exchange makes the configuration work by default, and still allows users to choose another protocol, if they like.
It's seems just as likely (or more so) that the standard IMAP set up would work by default. You're also dismissing that users have to pay to use you Exchange supporting add-ons, which is a rather big deal for the average user.
At the end of the day, I think you're assumptions are very biased. You want Owl to be advertised as widely as possible to make business as lucrative as possible. We get it. But its the unsuspecting Thunderbird users you are paying and who would like Thunderbird to do what's best for them.
|
|
Assignee | |
Comment 15•5 years ago
•
|
||
@Magnus: What about contacts and calendar? T hunderbird has 0 support for that right now. Owl supports some contacts and calendar out of the box with no configuration. (Calendar to be released as default in the coming weeks.)
pay Beonex 10€/year to access their email, when they could have been doing it all for free, with less hazzle, and is way more future proof.
That's not the what we're trying here. Like I said, that we show Owl for outlook.com was not intentional, and we'd fix that.
if Exchange was available, AND the normal IMAP/POP auth failed: that would be the appropriate place to advertise this alternative.
That's a reasonable proposal, but it will be difficult to implement, given the flow. What I don't want to do is offer an option to the user, have him confirm it, just to then tell him a second later that it's not available after all. That's bad UI.
What about my other proposal, to differentiate based on the domain, i.e. outlook.com/live/hotmail personal accounts vs. Office365 accounts? The latter are highly likely to want Owl, due to MFA, calendar, contacts etc..
|
|
Assignee | |
Comment 16•5 years ago
|
||
differentiate based on the domain
Andrew Sutherland has proposed the same in bug 1185366.
|
|
Assignee | |
Comment 17•5 years ago
•
|
||
This patch differentiates between Outlook.com/Hotmail personal accounts and Office365 business accounts.
This allows us to deliver different configs for both types of accounts. Given that they have different properties (e.g. outlook.com doesn't even support OAuth2), this makes sense to do even independent of the bug here. It will also help in other situations, see e.g. bug 1185366.
This is with the knowledge that all outlook.com accounts support IMAP with password-based authentication, and many Office365 domains do not even have IMAP enabled. For them, even IMAP with OAuth2 will not work. Furthermore, many Office365 users need the calendar, personal address book, and global company address book from the same server, for collaboration with their colleagues in the same domain.
|
|
Assignee | |
Comment 18•5 years ago
•
|
||
- If the domain is outlook.com, hotmail.fi or another known Hotmail domains, we will find that config directly in the ISPDB, and the MX code path won't be used at all. ISPDB will deliver a config that makes IMAP the default.
- For unknown Hotmail domains, we will use MX *.olc.protection.outlook.com, which also points to the Hotmail configuration
- If the domain is an Office365 domain (you can test with somebody@beonex.onmicrosoft.com), the MX will point to example.mail.protection.outlook.com. The direct ISPDB lookup for beonex.onmicrosoft.com will fail, and we will fall into MX lookup. We make 2 parallel lookups, for mail.protection.outlook.com and outlook.com, and give precedence to the mail.protection.outlook.com lookup (even if it arrives later). It will point to the Office365 configuration, which makes Exchange the default.
|
|
Assignee | |
Comment 19•5 years ago
|
||
|
|
Assignee | |
Comment 20•5 years ago
|
||
To state it clearly: That Owl was pre-selected for outlook.com/Hotmail was not intentional. It was a mistake, and I'm sorry about that. It was a side effect of the MX hostname setup of Microsoft. This patch fixes that.
|
|
Reporter | |
Comment 21•5 years ago
|
||
|
|
Reporter | |
Comment 22•5 years ago
|
||
Like discussed above, we should only show exchange when
A) it was the only choice we found
B) IMAP/POP3 was chosen but authentication failed (and we knew of an exchange option too)
|
|
||
Comment 23•5 years ago
|
||
|
|
Assignee | |
Comment 24•5 years ago
|
||
I tried mickeymouse@outlook.com and got offered Exchange as default.
Did you also apply the pref change and re-compile mailnews/ ? If not, that's why.
|
|
Assignee | |
Comment 25•5 years ago
|
||
|
|
||
Comment 26•5 years ago
|
||
(In reply to Ben Bucksch (:BenB) from comment #24)
I tried mickeymouse@outlook.com and got offered Exchange as default.
Did you also apply the pref change and re-compile mailnews/ ? If not, that's why.
No. I don't see how that's relevant. That pref change won't be shipped, so what we ships will show the behaviour we're trying to fix.
|
|
Assignee | |
Comment 27•5 years ago
|
||
@Jörg: The "Differentiate based on domain, v2" patch differentiates between outlook.com and Office365, and uses different configs for both, because they are different offers with very different properties. The current ISPDB doesn't have the new config and uses the same for both. The test ISPDB server already has the configs that we would add to ISPDB.
@Magnus: I'll make another patch.
|
|
||
Comment 28•5 years ago
|
||
Oops, the penny dropped. So why not push this to the official ISPDB first. Why do you need a new patch if the current one already works assuming it is fed the correct data? That was the only complaint from Magnus (quote): This doesn't really fix this bug.
|
|
Assignee | |
Comment 29•5 years ago
•
|
||
So why not push this to the official ISPDB first.
Because the patch was rejected :-) . Changing the ISPDB would only make sense, if this patch is accepted and shipped.
Why do you need a new patch if the current one already works assuming it is fed the correct data? That was the only complaint from Magnus (quote): This doesn't really fix this bug.
From how I understood Magnus, he was opposed to the idea of the approach, not the fact that he tried my patch and it failed. Magnus is dead-set on attempting IMAP first, even if this fails in most cases and also gives sub-par user experience (no calendar etc.pp.).
|
|
||
Comment 30•5 years ago
|
||
A misunderstanding? When I last spoke to Magnus he told me that the patch didn't work without the pref change to your server. At least I didn't realise what the background was. I think anything that fixes the bug and selects IMAP for outlook.com should be fine. Surely Magnus can speak for himself, but all the information I have is "doesn't work" (see comment #21). I see that you have attached your patch to bug 1185366. So do I see it correctly that if we fix that other bug, this one here becomes WFM? (Maybe I'm all wrong.)
|
|
Reporter | |
Comment 31•5 years ago
|
||
(In reply to Ben Bucksch (:BenB) from comment #25)
We shouldn't keep these flags everywhere. That leads to bugs.
I don't see how.
Anyway, we should fix it this way and not try to rely on ispdb changes. It's very possible there are cases where we just probe instead and find multiple alternatives.
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 32•5 years ago
|
||
|
|
Reporter | |
Comment 33•5 years ago
|
||
You don't like it because it doesn't advertise exchange as much as you'd like. That doesn't make the patch "not good" (to which you've given no qualifications). There's really no need for another patch. In comment 15 you even said it's reasonable, but now you back down just because there's a patch to achieve it?
|
|
Reporter | |
Updated•5 years ago
|
|
|
||
Comment 34•5 years ago
|
||
Shall we wait to see what Ben has to propose?
|
||
Comment 35•5 years ago
|
||
(In reply to Jorg K (GMT+2) from comment #34)
Shall we wait to see what Ben has to propose?
Absolutely not. It is a sheriff fail that this was not backed out immediately. I will have more to say later.
|
|
||
Comment 36•5 years ago
|
||
(Somewhat off-topic: Thanks for attributing so much power to the sheriff role, but after something has shipped in a beta and two(!!) ESR versions (60 and 68), the issue is more complex than a backout).
|
|
||
Comment 37•5 years ago
|
||
No way is that a credible excuse. A backout patch was even prepared. I suggest not attaching yourself to the wrong decision here.
|
|
Assignee | |
Comment 38•5 years ago
|
||
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 39•5 years ago
•
|
||
Requirements for this patch:
- We must not show a config to the user that will eventually fail. To the user, we must propose only configs that are known to work.
- The user approval before contacting an unknown IMAP server was a security measure, because we don't trust the network sources for configs. Together with the previous requirement, that makes the fix difficult. Making the fix specific to the Microsoft server gets us out of this problem.
- The code footprint should preferably be confined to one place, and be coherent with the rest.
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 40•5 years ago
|
||
This is another version of the "Try IMAP" patch which does not depend on ISPDB changes.
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Updated•5 years ago
|
|
|
Reporter | |
Comment 41•5 years ago
|
||
|
|
Assignee | |
Comment 42•5 years ago
•
|
||
noop
Not sure what you mean with that, but these checks were necessary, because this can indeed happen that there are no alternatives and there is only a single config. I did run into this in my testing scenarios.
password is not a required detail to fill in to get your configuration at least
If you don't fill out the password, a lot of the dialog and automatic configuration will be broken.
You wanted me to check whether IMAP works. (Remember I wanted to distinguish based on domain.) I don't know how you want me to test whether IMAP works, without a password. You cannot know without an actual login attempt. We know that the IMAP server outlook.office.com exists and works. These settings we check for here are per domain and often even per individual user account, so there's no way to know other than actually trying to log in.
If we skip this check here, and we also skip the password verification later at the end of the dialog, then there's a very high chance that the user ends up with a non-working account configuration. The whole point of the dialog is to find a working configuration.
I don't think it's ok to change what the user selected
This code is called from foundConfig(), which is called at the end of the automatic configuration. It will not be invoked after the user changes the radio buttons from IMAP to POP3.
There's a real edge case in which the code is invoked from the manual config, and we test the config again. We only run this new code here for the specific Office365 / outlook.com servers, and these are typically autodetected, so it's a highly unlikely case, but I have a simple idea how to avoid this edge case and I'll make a new patch.
|
|
Assignee | |
Comment 43•5 years ago
•
|
||
I don't think we want to have a per-hostname exception.
This dialog has as basic requirements, among others:
- We must not show a config to the user that will eventually fail. To the user, we must propose only the configs (as default) that are known to work.
- The user approval before logging in with password to an unknown IMAP server was a security measure, because we don't trust the network sources for configs.
Making the fix specific to the Microsoft server gets us out of this problem, and also adheres to Magnus' idea of checking whether the IMAP server works.
It also avoids that the patch has unintended side effects.
I see you went and made Exchange the first (==default) for the Microsoft domains in ISPDB
(I assume you mean outlook.com = Hotmail, what Microsoft calls "Personal accounts".)
Given the official documentation from Microsoft, that's just wrong. [Exchange is] not a supported option: https://support.office.com/en-us/article/pop-imap-and-smtp-settings-for-outlook-com-d088b986-291d-42b8-9564-9c414e2aa040
I don't know how you infer that from this document. The document only shows how to configure IMAP and POP3, which Microsoft itself in other documents calls "legacy protocols". It doesn't mean that other protocols are not supported.
Quite the opposite, Microsoft wants people to use their own protocols, not IMAP. Microsoft does advocacy to discourage IMAP.
And not just advocacy: Just while I was testing this patch, Microsoft blocked my @outlook.com account access, simply because I logged in from IMAP (they said "new device", but that's a false claim).
other clients are also using ISPDB and if they were not careful, they are now broken since exchange is listed as default.
The config file is defined so that we can add new types and new authentication mechanisms, and the client must ignore unknown types. If they are broken, they were broken from the start, and they've likely also been broken by earlier additions we made in the last few years.
|
|
||
Comment 44•5 years ago
|
||
I guess before we proceed any further, we need to define the desired agreed behaviour. No point submitting patches if that's not even clear.
I guess we have three cases:
- hotmail/outlook/live.com: They can all do IMAP and they can do Exchange as well, right? They are meant for personal use.
- office365.com: account works with IMAP (which needs to be tested somehow during the setup)
- office365.com: account won't work with IMAP (due to 2FA or other stuff).
There might be cases where the user hasn't provided a password, so some detection may not work and we have more cases.
Current behaviour is that in cases 1.-3. Exchange is always offered and selected. That's not desired.
With Magnus' patch in case 1, Exchange is not offered. Is this the agreed behaviour? I tend to agree with Magnus here that this is agreed.
With Ben's patch in case 1, Exchange is offered but not selected. Or is this the agreed behaviour?
I can't test cases 2 and 3. From the discussion I assume that for 2. and 3. Exchange is offered and selected with Ben's patch, I could be wrong. I don't understand what Magnus' patch tries to implement since " B) IMAP/POP3 was chosen but authentication failed" doesn't make sense to me since at first, the user only fills in their details and then the system goes checking and offers a choice. And it shouldn't offer IMAP if the authentication already failed.
So let's be clear on what we want to do and then do it in a constructive fashion and with enough detail to understand each other and any others marginally involved like Aceman or myself.
|
|
Assignee | |
Comment 45•5 years ago
•
|
||
Thanks, Jörg, for describing this clearly.
Case 1: hotmail/outlook/live.com
- Exchange should not be offered by default. This was a bug, and I'm trying to fix that.
- I think Exchange should be offered as non-default alternative, just like POP3. Exchange protocols work for these services, and the Owl implementation has certain advantages. E.g. people might want to use the calendar. However, I could compromise here.
Case 2: Office365 with IMAP available
- This is where Magnus and I differ.
- Magnus would like to default to IMAP. I can understand why, because IMAP is a free protocol.
- I think that Exchange should be the default, because Microsoft actively discourages the use of IMAP on Office365, and does so more and more aggressively. More importantly, these enterprise users also need some contacts access and the calendar for internal team organization, as central meeting planning tool, to book conference rooms etc.. Lots of enterprise users heavily depend on the calendar. We have out of the box calendar support, it works without additional configuration. Also, my users keep asking for shared mailboxes, e.g. for inbox@ and support@ addresses, and Thunderbird has no UI for it, and I'm not sure Exchange/Office365 even exposes it via IMAP. These features are critical for enterprise users, and e.g. the Auswärtige Amt in Germany has cited exactly the calendar and collaboration tools as reason why they moved away from Thunderbird to Outlook, so these features do matter. Many Office365 enterprise users will generally want and need these features, so together with the fact that Microsoft actively discourages and impairs the use of IMAP, I think Owl should be the default for Office365.
- Magnus and I agree that both IMAP and Exchange should be an option in this case, either way. The user should have the choice.
- My earlier patch "Differentate based on domain" patch offered Exchange by default in this case. (Magnus rejected it for this reason for this bug here. The patch is still useful for other reasons, and does not conflict with other fixes here.) My latest patch here implements what Magnus wants, and goes against my own interests.
Case 3: Office365 without IMAP working
- Exchange should offered, and the only option. I think Magnus and I agree on this.
- For usability, we must not offer options to the user and let him make choices which then do not work. So, my latest patch tests whether IMAP works before we let the user make a choice.
B) IMAP/POP3 was chosen but authentication failed" doesn't make sense to me since at first, the user only fills in their details and
then the system goes checking and offers a choice. And it shouldn't offer IMAP if the authentication already failed.
Exactly
|
|
||
Comment 46•5 years ago
|
||
Well, looks like we almost agree:
Case 1: hotmail/outlook/live.com: Select IMAP, offer POP, no Exchange.
Case 2: Office365 with IMAP available: Offer IMAP, POP, Exchange. Default yet to be determined. I'd be happy with Exchange. People are already on the Microsoft boat and whoever wants IMAP can just click it. Or we make IMAP the default and then we're 100% sure that people selecting Exchange made the deliberate choice.
Case 3: Office365 without IMAP working: Offer and select Exchange (offer POP?).
Case 4: No password given: Treat like case 2 since we can't test IMAP in that case, right?
So it should be possible to agree on the default in case 2 and then propose a mutually acceptable patch.
This issue has made a few waves and pre-selecting Exchange for unsuspecting hotmail/outlook/live.com users is quite ugly and we should fix it asap.
|
|
Assignee | |
Comment 47•5 years ago
|
||
(offer POP?)
If IMAP is disabled, then POP3 is as well, for the same reasons.
We should fix it ASAP
Agreed.
|
|
||
Comment 48•5 years ago
|
||
| important | ||
I had a long discussion with Magnus about bug 1185366, bug 1592258 and bug 1594366. Here's a summary for this bug:
Meanwhile changes to the ISPDB have landed to mitigate the issue:
https://autoconfig.thunderbird.net/v1.1/hotmail.com
https://autoconfig.thunderbird.net/v1.1/office365.com
https://autoconfig.thunderbird.net/v1.1/outlook.com
All "personal" use Hotmail/Live domains will now received IMAP preselected with Exchange also shown. Case 1. I think it would be good to remove the Exchange option from the ISPDB configuration right now.
Outlook.com and O365 can't be distinguished right now, see bug 1185366, and even when that bug lands, "old" clients before and including TB 60 won't be fixed. So let's assume a "new" client of TB 68 for the following. Since with such a client Outlook.com and O365 can be distinguished, outlook.com can receive the same treatment as its other "personal" use friends.
It remains to decide what to do for O365, three options exist in cases where IMAP works:
a) Only show IMAP
b) Select IMAP and offer Exchange as an option
c) Select Exchange and offer IMAP as an option
In cases where IMAP doesn't work, only Exchange should be shown (also no POP3 since that equally wouldn't work).
We'll discuss the "product placement" of option b) and c) further at the appropriate level.
|
|
Assignee | |
Comment 49•5 years ago
|
||
@Magnus: If you accept my comment 43, I can make the changes discussed in comment 42. This is waiting for your acceptance right now.
|
|
||
Comment 50•5 years ago
|
||
Can we do something here in time for TB 68.3 in early December? For this to happen it needs to ship in TB 71 beta 4 in about a week from now.
If it misses TB 68.3, the drama will be dragging on until 68.4 in January of 2020.
|
||
Updated•5 years ago
|
|
||
Comment 52•5 years ago
|
||
Disclaimer - I am a Thunderbird user - have been for years - and am also a Principal Engineer in a large finance firm.
As the person who reported 1597580, let me state my disappointment in the behaviour of some members of the Thunderbird development team.
Many open-source advocates constantly criticise Microsoft for embedding advertising and up-sells into their products and here we have Thunderbird, a product (originally?) from the Mozilla Foundation, up-selling paid-for commercial add-ons inside the codebase.
I mean - refer to questions such as: https://support.mozilla.org/bm/questions/1265452 from users trying to understand why they have plugins installed which are only a trial. How is this building trust with the community!?
Not only that, but I echo the surprise that two members of the development team were able to peer-approve this commit without oversight.
In my opinion, if you need Exchange support and your organisation has disabled IMAP, then you should use the web browser version until there's an open-source Exchange client inside Thunderbird.
I completely understand that people need to earn money, but sneaking in paid-for add-ons into open source codebases isn't the way to do it.
|
|
Assignee | |
Comment 53•5 years ago
|
||
In my opinion, if you need Exchange support and your organisation has disabled IMAP, then you should use the web browser version until there's an open-source Exchange client inside Thunderbird.
We have many many users which strongly disagree with you on that. They are very happy that this addon exists, and that they can continue to use Thunderbird. We received many emails with thank-you notes.
|
|
||
Comment 54•5 years ago
•
|
||
Please all follow the BMO guidelines and etiquette and keep to the technical aspect. The fact is that due some unfortunate circumstances Thunderbird was shipped offering the Exchange protocol via the Owl add-on to users that didn't need it. This has already partly been fixed, that is users of Hotmail/Live/Outlook accounts (with the exception of Outlook.com) will not be offered Exchange as default any more. We're working on further measures to fully rectify the situation, however, software has a release cycle and we can't just ship a new version for every issue we encounter.
|
|
Assignee | |
Comment 55•5 years ago
|
||
|
||
Comment 56•5 years ago
|
||
|
|
Assignee | |
Comment 57•5 years ago
|
||
I guess
onResultServerTypeChangedcan't use this function, can it?
Unfortunately not, because it selects a specific config and not a config type.
Also, I tried to keep the patch here small and to the problem at hand.
Nit: use
somerather thanfind.
Will do.
|
|
Assignee | |
Comment 58•5 years ago
|
||
Neil's NIT fixed.
This also fixes a number of Magnus' review comments.
|
|
||
Updated•5 years ago
|
|
|
Assignee | |
Comment 59•5 years ago
|
||
|
|
Assignee | |
Updated•5 years ago
|
|
|
Reporter | |
Comment 60•5 years ago
|
||
I thought we already made it clear cross-reviewing changes related to Owl isn't at all ok.
|
|
Assignee | |
Comment 61•5 years ago
|
||
@Magnus: Yes, that's why comment 59. The keyword was only to put it on Jörg's radar.
|
|
Assignee | |
Comment 62•5 years ago
•
|
||
What this patch does:
- Because IMAP might work on some accounts and not others, in the same Office365 customer domain:
- If there's both an IMAP config (default) and an Exchange config for Office365/Hotmail, check whether IMAP actually works for this account.
- If yes, show IMAP as default. If IMAP does not work, show Exchange as default.
|
|
Assignee | |
Comment 63•5 years ago
•
|
||
This is on top of v7. This is more aggressive and explicitly hides the Exchange config, if IMAP works.
Based on feedback from 2 council members, who suggested that this should be configurable via ISPDB, this is behavior is controlled by a new config flag fallbackOnly. If true for a given config, this behavior will be active.
The changed autoconfig URL has the new flag for Office365 (test with e.g. foo@beonex.onmicrosoft.com or any other Office365 account) and is for only for testing purposes for the reviewer. It's not supposed to land, just a convenience for the reviewer.
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Updated•5 years ago
|
|
|
||
Comment 64•5 years ago
|
||
|
|
Assignee | |
Comment 65•5 years ago
•
|
||
Both patches are ready to land, if a third developer approves
|
|
Assignee | |
Comment 66•5 years ago
•
|
||
Remove ISPDB hunk, use HG header. Ready to land.
r+ by Neil carried over from last patch.
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Updated•5 years ago
|
|
|
||
Comment 67•5 years ago
|
||
If there's both an IMAP config (default) and an Exchange config for Office365/Hotmail, check whether IMAP actually works for this account.
Good.
If yes, show IMAP as default. If IMAP does not work, show Exchange as default.
If IMAP doesn't work, why show it at all? You want to provide a non-working option? Or you want the user to confirm that it really doesn't work?
This is on top of v7. This is more aggressive and explicitly hides the Exchange config, if IMAP works. ... controlled by a new config flag fallbackOnly.
Good. Once shipped and adopted, we can drive the behaviour externally.
Seems strange to demonstrate it that way around, but what do I know?
What did Neil mean here? Did he mean to have a flag "alwaysShow".
|
|
Assignee | |
Comment 68•5 years ago
•
|
||
If IMAP doesn't work, why show it at all? You want to provide a non-working option? Or you want the user to confirm that it really doesn't work?
Yes, exactly, that was one of the thoughts I had.
Another was being "tactful".
Lastly, I'd need to remove POP3 as well, because I know that POP3 will not be enabled in Office365, if IMAP is disabled, because I've seen the policy setting available to Office365 admins that disable IMAP, and I know they also disable POP3. I don't want to actively test whether POP3 works with a second login attempt, because there's no point and it would actually be actively harmful to do so in some cases. But I don't know how this would be regarded by the reviewers, it might be a point of discussion, and I just wanted this to go smoothly, so I tried to leave this question out of the patch.
Seems strange to demonstrate it that way around, but what do I know?
What did Neil mean here?
I asked him offline and he was just referring to the test data. He was expecting me to showcase with outlook.com, and I showcased with Office365. So, based on that, my test autoconfig now also has such a fallback config for outlook.com. The comment was not referring to the patch itself.
@Jörg: Given that you're in here anyway: If you (or anybody else) could approve this, I'd be grateful. We can land this ourselves, if you prefer to not touch a commandline and hg and patches during your well-deserved vacation.
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Comment 69•5 years ago
|
||
The discussion we had in the Thunderbird council about this came to the conclusion that we want to get back to the state that was previously agreed on: only offer exchange if that truly was the only found option. We will not offer exchange to Office 365 users nor to other Microsoft mail services. (If there are users that want it, they should install Owl first through the add-ons manager and Owl can then make exchange available.)
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Comment 70•5 years ago
|
||
Please don't abuse the review flags: use the feedback flags when it's not a valid review we're talking about.
|
|
Reporter | |
Comment 71•5 years ago
|
||
https://github.com/thundernest/autoconfig/pull/13 is now up and will mitigate this bug across all branches.
What's left for this bug, is a patch to filter out exchange from found account types if other account types were found.
|
|
Reporter | |
Comment 72•5 years ago
|
||
Back to square 1 (without a backout).
|
|
Assignee | |
Comment 73•5 years ago
|
||
|
|
Reporter | |
Comment 74•5 years ago
|
||
Look, I'm implementing the council decision now. I understand you don't like it, but this is what we clearly decided. We almost unanimously passed the motion to:
** Remove exchange option for free accounts (Hotmail/Outlook). Remove exchange option for Office365 as well - “back to original terms” **
I don't know how this could be any clearer.
|
|
Reporter | |
Updated•5 years ago
|
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Updated•5 years ago
|
|
|
Reporter | |
Comment 75•5 years ago
|
||
This is ridiculous, can you stop playing with the flags.
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 76•5 years ago
•
|
||
|
|
||
Comment 77•5 years ago
|
||
|
|
||
Updated•5 years ago
|
|
|
||
Updated•5 years ago
|
|
|
Reporter | |
Comment 78•5 years ago
|
||
I'm not here to debate alternative solution options at this point. The decision was, explicitly, to remove exchange also for office 365.
There is no contradiction with comment 77. For Office 365, IMAP is available.
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Updated•5 years ago
|
|
|
||
Comment 79•5 years ago
|
||
(In reply to Magnus Melin [:mkmelin] from comment #78)
For Office 365, IMAP is available.
So why do we have bug bug 1528136 on file then?
|
|
||
Comment 80•5 years ago
|
||
And can you please explain why you need this code change at all? Remove the exchange options from
https://autoconfig.thunderbird.net/v1.1/outlook.com and
https://autoconfig.thunderbird.net/v1.1/office365.com and you're done. What am I missing?
|
|
Assignee | |
Comment 81•5 years ago
•
|
||
I did exactly what you asked for in comment 14: "if Exchange was available, AND the normal IMAP/POP auth failed: that would be the appropriate place to advertise this alternative". I don't understand why you now claim the exact opposite.
Remove exchange option for free accounts (Hotmail/Outlook). Remove exchange option for Office365 as well
For the record, I checked the text of the official TB Council decision as communicated to me (Philipp Kewisch on behalf of TB Council, 2019-11-20 00:07 CET), and this is sentence is not part of it. Rather, the sentence I and Jörg quoted: "Exchange should only be advertised to the user if the other supported protocols such as IMAP are not available". That's what I implemented.
For Office 365, IMAP is available.
It's disabled for many domains, or even for specific users within the domain.
I have tons of users in support here who have Office365 accounts and are saying the exact opposite of what you claim.
If IMAP is indeed working and enabled, then my patches do exactly what you want. We show only IMAP and not Exchange. My patches have an effect only for those Office365 users who cannot use IMAP.
|
|
||
Comment 82•5 years ago
|
||
|
|
||
Comment 83•5 years ago
|
||
I'd like to understand Magnus' patch better. Magnus, can you please answer the following question:
From comment #80: Q1: Can you please explain why you need this code change at all? Removing the exchange options from
https://autoconfig.thunderbird.net/v1.1/outlook.com and
https://autoconfig.thunderbird.net/v1.1/office365.com does the trick, no?
I see that you've written in comment #71:
https://github.com/thundernest/autoconfig/pull/13 is now up and will mitigate this bug across all branches.
What's left for this bug, is a patch to filter out exchange from found account types if other account types were found.
"Filter out exchange from found account types if other account types were found" - Q2: Why is that necessary if the exchange type is removed from the ISPDB?
From comment #78: For Office 365, IMAP is available.
Why do we have bug bug 1528136 on file? Q3: Is that not a reason why IMAP is sometimes not available?
Ben wrote on the TB Council list (non-public) (quote):
It's disabled for many domains, or even for specific users within the domain. I have tons of users in support here who have Office365 accounts and are saying the exact opposite of what you claim. In fact, that's the very reason why I started Owl: To help those Office365 (and Exchange) users who cannot use IMAP.
Q4: Do we dismiss this empirical data or do you think it is false?
And related to Q2: IMHO a solution that hardcodes the behaviour of the software is not desirable for the obvious reason that it is inflexible since needs to be rolled out in the next release and then we have to wait for user adoption. I suggested to implement something that can be controlled via the ISBDB across all client which include the change. Q5: How does your solution fit into this?
Kindly reply to my questions Q1 to Q5 and then we can assess your solution further. If you don't mind, I will do the review in order not burden Alex by pulling him into this discussion. Since I'm on PTO, I'll only occasionally check the bug for progress, so please drop a line on the Council list.
|
|
Reporter | |
Comment 84•5 years ago
|
||
(In reply to Jorg K (GMT+1) (PTO to 1st Dec 2019, not reading bugmail) from comment #83)
From comment #80: Q1: Can you please explain why you need this code change at all?
Yes, like I wrote in comment 71, not too much left to do in this bug. The patch ensures we do not show exchange for on premise exchange when IMAP and/or POP are available.
"Filter out exchange from found account types if other account types were found" - Q2: Why is that necessary if the exchange type is removed from the ISPDB?
See answer to Q1.
From comment #78: For Office 365, IMAP is available.
Why do we have bug bug 1528136 on file? Q3: Is that not a reason why IMAP is sometimes not available?
IMAP is always available. If it's working or not is a secondary issue - and verifying that is the next step in the wizard.
Q4: Do we dismiss this empirical data or do you think it is false?
I would be interested in numbers. That there is only 26 persons on cc of bug 1528136 does tell you something when the number of people setting up such accounts are in the 6 figure range monthly!
Either way, it's irrelevant for what the council decision was: to get back to square one.
And related to Q2: IMHO a solution that hardcodes the behaviour of the software is not desirable for the obvious reason that it is inflexible since needs to be rolled out in the next release and then we have to wait for user adoption. I suggested to implement something that can be controlled via the ISBDB across all client which include the change. Q5: How does your solution fit into this?
Eh, the very patch proposed by Ben hardcodes a outlook.office365.com into the source code. Basically unprecedented! Where is the flexibility in that? Given we release every 6 weeks at least, and going to start doing so even more frequently, the user wait period is usually not that long to do such changes in-client (when we release every 4 weeks, avg. would be 2 weeks).
Implementing client specific behaviour in ispdb is certainty against the philosophy behind that code: it's not meant to be Thunderbird only (though yes, catering for Thunderbird's needs), and in fact, sadly, Thunderbird only accounts for a fraction of the traffic.
|
|
||
Comment 85•5 years ago
•
|
||
(In reply to Magnus Melin [:mkmelin] from comment #84)
Yes, like I wrote in comment 71, not too much left to do in this bug. The patch ensures we do not show exchange for on premise exchange when IMAP and/or POP are available.
I think that detail is really not needed. We don't need to preach IMAP to people who are already on the M$ boat heading to their on-premise Exchange server. Why be even more hardlined than we've ever been?
IMAP is always available. If it's working or not is a secondary issue - and verifying that is the next step in the wizard.
OK, so what does the wizzard do if IMAP fails in the next step? Anything useful that would allow the user to create a working account setup? Funny that we're arguing about "available" vs. "working" now.
I would be interested in numbers.
I'm sure Ben can supply numbers.
Either way, it's irrelevant for what the council decision was: to get back to square one.
There is a new Council motion on the way. Even if we implemented the one you're referring to, removal from the ISPDB would be sufficient.
Eh, the very patch proposed by Ben hardcodes a outlook.office365.com into the source code. Basically unprecedented!
Really? How about:
https://searchfox.org/comm-central/rev/f40fe7e4be1d5dc5bd90c831a0bf87d354f3e74b/mail/extensions/openpgp/content/modules/wkdLookup.jsm#28
https://searchfox.org/comm-central/rev/f40fe7e4be1d5dc5bd90c831a0bf87d354f3e74b/mailnews/imap/src/nsImapProtocol.cpp#7340
https://searchfox.org/comm-central/rev/f40fe7e4be1d5dc5bd90c831a0bf87d354f3e74b/mailnews/mailnews.js#546
https://searchfox.org/comm-central/search?q=Gmail&case=true®exp=false&path=
Office365 is becoming a global mail provider like Gmail, so I don't have a problem with hardcoding the domain where relevant. That said: We might want to move the string into a pref, like we've done for the Openwave thing I've referenced above.
Implementing client specific behaviour in ispdb is certainty against the philosophy behind that code: it's not meant to be Thunderbird only (though yes, catering for Thunderbird's needs), and in fact, sadly, Thunderbird only accounts for a fraction of the traffic.
The ISPDB was invented for Thunderbird account setup and has been so successful that other clients are using it to, for example my Android app FairEmail. IMHO there's nothing wrong adding certain attribution to certain entries that a client may or may not process. Maybe we should consult the inventor of the ISPDB about its underlying philosophy.
|
|
Reporter | |
Comment 86•5 years ago
|
||
(In reply to Jorg K (GMT+1) (PTO to 1st Dec 2019, not reading bugmail) from comment #85)
I think that detail is really not needed. We don't need to preach IMAP to people who are already on the M$ boat heading to their on-premise Exchange server. Why be even more hardlined then we've ever been?
We're not any more hardlined. That was how it was before this bug: only show exchange if other protocols failed. That applied to on-premise exchange too.
As I already stated elsewhere: you're talking about different users. It's hardly the Thunderbird users who chose the O365 setup and then went to the trouble of disabling IMAP. That's a bad decision by their admin.
OK, so what does the wizzard do if IMAP fails in the next step? Anything useful that would allow the user to create a working account setup?
The usual: alerts the user about the situation, with potentially helpful information. They may easily have set the wrong password and would try again. Depending on situation and technical skill-set, can also make sure IMAP gets enabled.
There is a new Council motion on the way. Even if we implemented the one you're referring to, removal from the ISPDB would be sufficient.
It's not, as per my previous answer to Q1.
Really? How about:
There's not even any server dns names in your queries (except for in the still-wip-pgp code).
IMHO there's nothing wrong adding certain attribution to certain entries that a client may or may not process.
The proposition was to add a setting essentially saying if it should be usually shown or not. That's not an attribution a client may or may not posses.
|
|
||
Comment 87•5 years ago
|
||
|
|
Reporter | |
Comment 88•5 years ago
|
||
(In reply to Jorg K (GMT+1) (PTO to 1st Dec 2019, not reading bugmail) from comment #87)
Let me know what I'm missing.
You miss that this also gets called for results from guessConfig, i.e. the auto detection (guessing by name, probing and all that). I doubt there are many intra configuration files you'd find where someone would have listed exchange and others.
So, with no configuration file found - but guessed configuration found - you'd easily end up with an on-premise exchange that has IMAP+POP enabled. For this case, we should show IMAP/POP just like we used to.
Aha, "potentially" helpful information. Of which kind exactly? If they
mistyped the password, with Ben's patches, IMAP is still available, the user
can select it and enter the correct password.
[with the patch] After they figured out we "helpfully" auto selected exchange for them!?
a contract with Microsoft) and will under no circumstances enable IMAP.
Then there would be no problem, because they would not end up there and exchange is the only option. We're talking about O365 now where IMAP is enabled by default.
|
|
Assignee | |
Comment 89•5 years ago
•
|
||
Magnus wrote:
We're talking about O365 now
That's not what your patch does.
That was how it was before this bug: only show exchange if other protocols failed. That applied to on-premise exchange too.
That is not true, and it's clear to anyone who understands what your patch does. As the first words in comment 0 show, we're trying to fix a regression caused by bug 1571772, which was on Office365 and Hotmail. The effects on Hotmail were completely unintended and already fixed by bug 1185366. This leaves Office365 here, and the TB Council is currently discussing that.
Your patch removes options for on-premise Exchange that were there long before bug 1571772. And it's clear from your patch that it's deliberate. That's one reason why I called on the TB Council.
Jörg wrote:
Why do we have bug bug 1528136 on file? Q3: Is that not a reason why IMAP is sometimes not available?
FYI, that is a possible reason, but there are a number of other reasons why IMAP might be deactivated and not available. Microsoft offers so many different ways in so many different places and angles and reasons to disable IMAP. In effect, many users cannot use IMAP.
I know from first hand experience that whole organisations force users onto using the Outlook client (since the have a contract with Microsoft) and will under no circumstances enable IMAP.
Ditto. I was in the same situation, on Office365, in several different organizations. Many other people are in the same situation, I read that in support emails, from novices to highly technical users like software engineers, and it's getting increasingly worse instead of better.
Magnus wrote:
IMAP is always available. If it's working or not is a secondary issue
Now you're down to just playing with words. Something that doesn't work isn't available, no. A technical device that's sitting in front of you physically and is broken and not working is not "available". This is especially true when that device was deliberately deactivated by someone. It's then exactly not available.
Jörg wrote:
OK, so what does the wizzard do if IMAP fails in the next step?
That's exactly the problem. Without my patch, we would show "wrong password", which makes the user try other passwords. That's a complete disaster. We can't have that. I'm not going to show non-working configs to the user and then tell him "password wrong". That's a complete no-go.
That's why my patch is written the way it is.
|
|
||
Comment 90•5 years ago
|
||
(In reply to Magnus Melin [:mkmelin] from comment #88)
(In reply to Jorg K (GMT+1) (PTO to 1st Dec 2019, not reading bugmail) from comment #87)
Let me know what I'm missing.
You miss that this also gets called for results from guessConfig, i.e. the auto detection (guessing by name, probing and all that). I doubt there are many intra configuration files you'd find where someone would have listed exchange and others.
Hmm, "guess config" doesn't process config files, it guesses server names, like imap.domain.com, mail.domain.com, etc. and then does the probing.
As I said: Assuming we remove the exchange options form the ISPDB, your patch is not necessary. I'm also confused by the last paragraph of your comment #88:
We're talking about O365 now where IMAP is enabled by default.
I feel silly repeating it one more time: To address O365, no code change is necessary, this can be done via the ISPDB. How many times do we need to repeat that some setups via O365 have IMAP disabled? Or are you denying that as well? Apart from an unhelpful error message the user would be left stranded needing to get Owl or ExQuilla from ATN.
As Ben pointed out, your patch also affects on-premise servers, which is not desired. This bug was to rectify the Hotmail/Live/Outlook/O365 situation and I don't know why the configuration for on-premise servers has suddenly also become a target.
I can't apply another r-, but this comment here needs to be added to my review comments in comment #87.
|
|
Reporter | |
Comment 91•5 years ago
•
|
||
(In reply to Ben Bucksch (:BenB) from comment #89)
Your patch removes options for on-premise Exchange that were there long before bug 1571772.
If that's the case it was never agreed upon. Exactly when did that sneak in? Showing exchange ONLY when it's the only option is what we agreed upon. Anything else needs to be ripped out.
Magnus wrote:
IMAP is always available. If it's working or not is a secondary issue
Now you're down to just playing with words.
No, but you're trying to take the word out of its context. It's available as in: available as one found option.
|
|
Reporter | |
Comment 92•5 years ago
|
||
(In reply to Jorg K (GMT+1) (PTO to 1st Dec 2019, not reading bugmail) from comment #90)
As I said: Assuming we remove the exchange options from the ISPDB, your patch is not necessary.
It is, otherwise we still show exchange for cases POP/IMAP is available, as I already explained.
I feel silly repeating it one more time: To address O365, no code change is necessary, this can be done via the ISPDB.
Then merge the PR, as I've suggested.
How many times do we need to repeat that some setups via O365 have IMAP disabled? Or are you denying that as well? Apart from an unhelpful error message the user would be left stranded needing to get Owl or ExQuilla from ATN.
Some functionality require an add-on to be installed. Product placement of Owl was never agreed upon for any such case.
I don't think there official statistics on how many setups have IMAP disabled.
As Ben pointed out, your patch also affects on-premise servers, which is not desired.
Reason being?
This bug was to rectify the Hotmail/Live/Outlook/O365 situation and I don't know why the configuration for on-premise servers has suddenly also become a target.
Just because that might have gone unnoticed in comment 0, it doesn't mean we shouldn't address it since it was found while analysing the scenarios.
|
|
Assignee | |
Comment 93•5 years ago
|
||
|
|
Assignee | |
Comment 94•5 years ago
•
|
||
Both your ISPDB PR 13 and your patch here are going to prevent users from using Thunderbird, because both actively suppress the Exchange config, even for Office365 users that cannot use IMAP. That's breaking users.
My patch gives maximum exposure to IMAP: It shows IMAP users who can use it, and Exchange (plus IMAP) for the others. That's exactly what the Council demanded from me. I simply implemented the Council's wishes.
|
|
||
Comment 95•5 years ago
|
||
(In reply to Magnus Melin [:mkmelin] from comment #92)
It is, otherwise we still show exchange for cases POP/IMAP is available, as I already explained.
I think this is fairly hypothetical since this would only be the case for on-premise setups which are unlikely to offer both IMAP and Exchange. Does anyone have any data on that?
Then merge the PR, as I've suggested.
If the Council decides to kick it out altogether in the follow-up vote, I'm sure it will be merged. If we hadn't held up proceedings, the solution could now be on trunk and TB 71 beta 4 ready to get into 68.3 on the 2nd of Dec.
Just because that might have gone unnoticed in comment 0, it doesn't mean we shouldn't address it since it was found while analysing the scenarios.
So you're re-badging the bug as regression of bug 1500105 which you approved yourself? I thought we're here to fix the last remnants of the bug 1571772 mess-up.
|
|
||
Comment 96•5 years ago
•
|
||
Magnus did some experiments with his provider who uses Exchange of some description. I don't have an account there, but I tried anyway.
In TB 68.3 (pre-release) somehow the domain login password box popped up, that might have come from bug 1518155. However, in the end no Exchange/Owl is offered. Hmm. Seems to be independent of whether an invalid password is entered or no password is entered.
EDIT: Same behaviour on trunk, BTW.
|
|
Reporter | |
Comment 97•5 years ago
|
||
Re comment 93, filed as bug 1598861.
|
|
||
Comment 98•5 years ago
|
||
You mean comment #96.
|
|
||
Updated•5 years ago
|
|
|
||
Updated•5 years ago
|
|
|
||
Comment 99•5 years ago
|
||
|
|
||
Comment 100•5 years ago
|
||
With Magnus' patch: Step 1: Offering IMAP/POP3 which isn't working.
|
|
||
Comment 101•5 years ago
|
||
With Magnus' patch: Step 2: Showing semi-misleading message and offering Owl, but not selecting it.
IMHO, that's really bad UX.
|
|
||
Comment 102•5 years ago
|
||
With Ben's two patches: Selects Exchange when IMAP isn't working.
|
|
||
Comment 103•5 years ago
|
||
With Ben's two patches: Selects IMAP if working. Doesn't show exchange:
Disclaimer, picture provided by Ben. I can't get this to work on the open.click domain. Apparently the MX lookup that would send it to the https://autoconfig.thunderbird.net/v1.1/mail.protection.outlook.com file via this MX record openclick.mail.protection.outlook.com
https://mxtoolbox.com/SuperTool.aspx?action=mx%3aopen.click&run=toolpage#
doesn't work for me. I get totally non-working guess config where it doesn't even display a decent server name.
|
|
Reporter | |
Comment 104•5 years ago
|
||
In the normal case that you typoed the password, the messages are perfectly alright.
I've seen no data enforced MFA would be even a very common case, and it would be extraordinary if it was the majority. It's not the default configuration of your O365 account, and the comparative small number of people subscribed to bug 1528136 would say optimizing for that case is just wrong.
https://venturebeat.com/2018/08/22/global-survey-reveals-low-adoption-of-multi-factor-authentication-for-office-365/ cites 20 % of users using it. Even with steady adoption from there, you're still pretty far from half which would be an absolute minimum for the auto-selection to make any sense.
|
|
Assignee | |
Comment 105•5 years ago
|
||
|
|
Assignee | |
Comment 106•5 years ago
•
|
||
A lot of companies changed to MFA even in the last 3 months, even more so since 2018. Such old numbers are pointless for a hot topic like this.
But MFA is only one reason why IMAP might not work. You can also specifically disable IMAP in Exchange. Many companies do that, because they expect their users to use MS Outlook. I know that's not the case for your University account, but it's been the case for all orgs where I had Exchange accounts in the last years. They all had IMAP disabled.
I tried to get some numbers on this, and I ran a survey of Exchange servers myself (over 22000 servers contacted), and the vast majority have IMAP disabled, with ports completely closed or timeout.
Even with the IMAP port open, we can't know whether IMAP would work for a given user, because Exchange is extremely configurable and can disable stuff on multiple levels and reasons. For example, for the Office365 test accounts I created for you, IMAP port is open, but IMAP still is disabled for the user. And it's also blocked per auth policy rules, which is yet another setting. The accounts do not have MFA enabled, and IMAP port is open, but it still doesn't work.
In the normal case that you typoed the password
That is not the normal case. The normal case is that IMAP is disabled.
The situation before autoconfig was that I would enter data, and I would get "wrong password", even though some config setting is wrong. That made setting up the account so difficult. Autoconfig fixed all that.
The autoconfig dialog helps you find the right configuration that works for you. That's its purpose in life.
We cannot show a config by default that won't work. The job of the dialog is to figure that out.
|
|
Assignee | |
Comment 107•5 years ago
|
||
I've just re-tested the v7 patch with the 2 Office365 test domains, one has IMAP disabled on multiple levels, and the other has IMAP enabled, and they both work as expected. They have IMAP selected by default, if IMAP works for that user, otherwise Exchange.
|
|
||
Comment 108•5 years ago
|
||
|
|
Reporter | |
Comment 109•5 years ago
|
||
(In reply to Ben Bucksch (:BenB) from comment #106)
A lot of companies changed to MFA even in the last 3 months, even more so since 2018. Such old numbers are pointless for a hot topic like this.
So you have better data for O365? I can't find any news MFA adaption skyrocketed the last 12 months.
I tried to get some numbers on this, and I ran a survey of Exchange servers myself (over 22000 servers contacted), and the vast majority have IMAP disabled, with ports completely closed or timeout.
Irrelevant, since we're discussing O365 now.
In the normal case that you typoed the password
That is not the normal case. The normal case is that IMAP is disabled.
Eh, no. For O365, IMAP is on by default.
We cannot show a config by default that won't work.
If the password is wrong, it won't work for exchange either. Yet you want to select it automatically.
|
|
Assignee | |
Comment 110•5 years ago
•
|
||
The possibility of the user entering invalid data is not an excuse for us to show users who entered the right data a non-working config.
If the user indeed entered the wrong password, we check that later on, and still allow him to correct the mistake and then select IMAP manually.
If you think a wrong password is likely, we should re-run the config, after the user corrected the password. We already do that now when he corrected the email address. That solves the case you mentioned, because we'd then discover that IMAP works with the new password, and preselect that. It would also solve AutoDiscover and the username field, the latter should be made disappear as well. So, I think that's a good point.
|
|
Assignee | |
Comment 111•5 years ago
|
||
Filed as bug 1600954, because that is an independent issue. It should take care of this wrong password case you mentioned. It also helps in other cases.
|
|
Assignee | |
Comment 112•5 years ago
|
||
Filed as bug 1600954
It has a patch (which should land independently of this bug), which also takes care of the wrong password case here.
|
|
||
Comment 113•5 years ago
|
||
|
|
Reporter | |
Comment 114•5 years ago
|
||
Implement Alex's suggestion. That is, run the logon verification already before we show the config we found. If config failed, include exchange in the available account types. The first standard protocol (IMAP) is still selected, and it's up to the user to try another one. The user is shown an error message, which we will improve on trunk (improvement not included in this patch).
This patch also runs re-ordering of the account types of the config so we ensure not to end up with pre-selected exchange when other standard protocol options are available.
Yes, this patch also affects on-premise exchange.
|
|
Assignee | |
Comment 115•5 years ago
|
||
|
|
Reporter | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 116•5 years ago
•
|
||
Jörg wrote in comment 113:
- two-step process ... constitutes bad UI
- Please restrict the action .. to address: Account setup at O365.
- configurable via the ISPDB
Magnus' latest patch fixes point 1, but still fails points 2 and 3.
FWIW, my patch "Try IMAP, v7" fixes all of the above.
|
|
||
Comment 117•5 years ago
|
||
|
|
Assignee | |
Comment 118•5 years ago
•
|
||
|
|
||
Comment 119•5 years ago
|
||
(In reply to Jorg K (GMT+1) (PTO to 5th Jan 2020, sporadically reading bugmail) from comment #117)
Magnus, you know that Ben is the initial author and of the account setup
code and owner of this code (see Council discussion of Nov. 2019). He's also
the inventor of the ISPDB. I don't know why you keep degrading his role
here.
To put some facts behind this, please visit these links:
https://hg.mozilla.org/releases/comm-esr10/filelog/tip/mailnews/base/prefs/content/accountcreation/fetchConfig.js
https://hg.mozilla.org/releases/comm-esr10/filelog/tip/mailnews/base/prefs/content/accountcreation/guessConfig.js
(I'm sure there is more)
https://wiki.mozilla.org/Thunderbird:Autoconfiguration, author Ben B
https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration, author Ben B
|
|
||
Comment 120•5 years ago
|
||
|
|
||
Comment 121•5 years ago
|
||
| offensive | ||
(In reply to Jorg K (GMT+1) (PTO to 5th Jan 2020, sporadically reading bugmail) from comment #117)
Comment on attachment 9117227 [details] [diff] [review]
bug1592258_exchange-fix.patchReview of attachment 9117227 [details] [diff] [review]:
Magnus, you know that Ben is the initial author and of the account setup
code and owner of this code (see Council discussion of Nov. 2019). He's also
the inventor of the ISPDB. I don't know why you keep degrading his role
here. That said, we were aiming for a solution to fix the O365 issues and
not suppress Exchange for on-premise servers, especially after adding
IMAP/POP3 like for Aalto.fi. A configurable solution would also be desirable.
As a commercial interest who practices "self dealing for self enrichment", bucksch has no role here.
You have no authority to flip review flags, nor any authority to override the module owner. You have actually destroyed any semblance of governance on this project, making stuff up as you go along, seemingly taking 'sheriff' as a Wild West literal interpretation.
Study hard the lesson given by hsivonen and ms2ger in Bug 650776.
|
|
Assignee | |
Comment 122•5 years ago
|
||
TL;DR: Magnus' patch unconditionally recommends IMAP in all cases, even for an Office365 user that cannot use IMAP, and throws an "wrong config or password" error message to the user, and leaves it up to the user to figure out what to do. This is not what the TB Council asked. It is also patently unhelpful for end users, and I've rejected it on that basis.
My patch v7 specifically recognizes Office365, tests whether IMAP works, and if it does, defaults to IMAP, and otherwise defaults to Exchange. This patch was implemented as direct result to the request from the TB Council. It always defaults to a working configuration. My code is also a lot smaller and more confined.
(In all cases, it allows the user to make another choice. If you want to hide Exchange in case IMAP works, I've also created a patch for that.)
We've created a test build (based on Dec 12 trunk code) that demonstrates my patches:
|
|
Assignee | |
Comment 123•5 years ago
|
||
My negative review of Magnus' patch was based on simple reasons and principles that existed long before Owl even existed, e.g. as stated on Yahoo. The principle that it needs to work for all users and not just a part of the users is common sense.
My patch would suggest IMAP for those who can use it, and Exchange for those users who cannot use IMAP, so it's actively leading users to free protocols, while still helping people to use Thunderbird who cannot use IMAP.
|
|
Reporter | |
Comment 124•5 years ago
|
||
| private | ||
(In reply to Jorg K (GMT+1) (PTO to 5th Jan 2020, sporadically reading bugmail) from comment #120)
I while ago, I've asked Ben why we couldn't remove the
config.incoming.hostname == "outlook.office365.com"bit from his patch.
You can find his answer in comment #118, in the dot point starting with "We
cannot send passwords to random IMAP servers that AutoDiscover found".
That password is already sent to get the autodiscover.xml (now, even since exchange support landed). You're grasping at straws that don't exist.
There's no need for a policy in ispdb. What would that say? I can assure you we would not accept a policy that would not use IMAP when that's available. Are you saying you would? In the unlikely event a policy would be set up, that can be handled at that point.
Ben is not "the owner of this code". This is Thunderbird code, nobody's else. He's free to give feedback, but I have no intention of waiting for a r+ he won't give due to his commercial interests in not seeing that happen.
|
|
Reporter | |
Comment 125•5 years ago
|
||
(In reply to Ben Bucksch (:BenB) from comment #122)
TL;DR: Magnus' patch unconditionally recommends IMAP in all cases, even for an Office365 user that cannot use IMAP, and throws an "wrong config or password" error message to the user, and leaves it up to the user to figure out what to do. This is not what the TB Council asked.
The council wanted Alex to figure out a plan. That plan is now coded in the patch.
|
|
Reporter | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 126•5 years ago
•
|
||
Ben is not "the owner of this code".
For the record, that is factually untrue. I am the legal owner of the code in question. I own the copyright and merely allowed Thunderbird and others to use the code.
|
|
||
Comment 127•5 years ago
•
|
||
Ben is not "the owner of this code".
From the Council list:
Subject: Re: [tb-council] Code Contributions
On 12 Nov 2019 12:32, Magnus Melin wrote:
There is no copyright assignment of code at Mozilla. Code remains in the ownership (or is it co-ownership?) of their original authors, except for when done under contract with Mozilla. But of course, the original owners have to agree to license the code under the MPL.
So which one is it?
The council wanted Alex to figure out a plan. That plan is now coded in the patch.
The Council asked Alex to take a look with the aim to potentially mitigate the conflict. I have actually responded to Alex plan and haven't heard anything back. Alex hasn't been empowered to approve a patch that removes the Exchange option where the server offers it exclusively. As Ben said, even users what have Owl installed won't be able to set up an account now. Please don't burden Alex with a decision that even seven councillors couldn't resolve so far.
I'm not sure how Alex' review would help right now. Ben pointed out some technical issues as the original author of the code. I don't assume you're going to land the code in case Alex gave you an r+, or is that the plan?
|
||
Comment 128•5 years ago
•
|
||
Apologies for the lack of answers in the email thread, I got the flu and I've been trying to push forward with other bugs, but that's not an excuse for the lack of communication, so, apologies again.
I wrote in the email my suggestion from a UX point of view, which is:
- We should NOT rely on a 3rd party add-on to complete the account setup process. Thunderbird should be able to handle that all by itself.
- Asking the user to install a 3rd party add-on to complete a core process is not good, especially if the user ends up not needing the add-on at all.
The argument of "We recommend the Add-On because it's the only way for us to be sure that IMAP doesn't work and the password is correct" is not a good argument for me, and it shows an issue in the Thunderbird core that should be addressed, without relying on a 3rd party add-on.
Now, I'm the first to admit that my technical knowledge in this scenario is limited, but I trust Magnus' decision and expertise, so I will test this extensively the best I can, and I will leave my review.
|
|
Assignee | |
Comment 129•5 years ago
|
||
Alex, the problem with Magnus' patch is that it recommends (defaults to) IMAP even for those Office365 users who have IMAP disabled or cannot use it due to MFA. This is a large chuck of the Office365 users. For them, IMAP will not work, and Magnus' patch still defaults to IMAP and shows an error message 'wrong config, username or password'. Many users will see "wrong password" and try to fix this, although the password was indeed correct, so this message is actively misleading end users on the wrong path.
Notably, Magnus' patch is showing the wrong config and a misleading error message for users who entered the correct (!) data. This would happen for a significant part of the affected users.
This is directly counter to what the TB Council wrote, namely to default to Exchange for users who cannot use IMAP.
|
|
||
Comment 130•5 years ago
|
||
Yes, we all agree that needing an add-on to determine a bad password is not ideal. However, as I said in my e-mail reply, we should optimise the more likely case. Maybe in 5% of cases the password is entered incorrectly, even less these days with the eye icon. In mid-2018, 20% of IMAP was not functional due to MFA, and the trend has been on the rise. So there are 4-10 times more cases of non-working IMAP than wrong password user error cases. And to optimise your system for the user error case while badly serving a user group that is much larger is not good UX.
We can also display an info message when preselecting Exchange after a failed IMAP test. Details below.
For those who are included to read it, here's my last post on the internal thread:
Initial investigation has shown that the M$ IMAP server only replies "LOGIN Failed" and doesn't not detail the failure reason. So we can't distinguish a user error of entering wrong credentials from an inherent system error that makes IMAP completely unavailable.
We should optimise the UX for the general case and not for the "user error" case. The general case it that IMAP is not working due to MFA or being disabled. We have data that this percentage was at least 20% in mid-2018, with indications that it is far higher presently. We can only estimate the "user error" case, with perhaps 5%, less likely these days, since users can and will check using the eye icon.
IOW: Based on numbers, we're comparing a "user error" edge case with perhaps an occurrence of perhaps 5% of the cases with a case that is becoming more an more standard, that is IMAP not available on the server side. The likeliness of the latter is 4-10 times higher, so it's pretty clear to me, which case should be optimised.
If I understand Alex' suggestion correctly, he wants to jump directly to Magnus' step 2 where this message is displayed: "Unable to log in at server. Probably wrong configuration, username of password". Now, that's pretty confusing as a first screen particularly with a "Done" button underneath.
Hence my suggestion: In non-ESR68 versions we can add strings pointing out that the system failed to establish an IMAP authentication and suggest remedies.
So what we should do for trunk is this, IHMO: Run the IMAP test like Ben suggests, and if it failed pre-select Exchange. At the same time, display an info(!!) message that expresses the following:
"The system tried setting up the account using the standard IMAP protocol and failed. It therefore selected Exchange as a protocol that is known to work. It is possible that the IMAP setup failed due to wrong credentials, so check the credentials before proceeding with Exchange" ... or words to that extent.
For TB 68 we can't ship that message, so we'll just have to pre-select Exchange. That's not really all that bad, since after installing Owl, the user gets the chance to correct his/her password, and then IMAP might work in the second round (bug 1600954). So for TB 68 we'll have the slightly long-winded solution to detect a wrong password, and in TB 73 going forward we'll have an appropriate message. Or let's say it in different words: If Owl is the tool to detect an incorrect password reliably, let's use it!
We should add telemetry probes to see how many people set up accounts, how many end up with Exchange and how many restart with corrected credentials, be it e-mail address, domain login user or password.
|
|
||
Comment 131•5 years ago
|
||
| private | ||
(In reply to Magnus Melin [:mkmelin] from comment #124)
That password is already sent to get the autodiscover.xml (now, even since exchange support landed). You're grasping at straws that don't exist.
Let me just say that your patch opens more opens more vulnerabilities. Please think to which servers you may send the password in the future and how attackers could fake those servers. Now that I've said this, the bug needs to be security restricted.
(In reply to Alessandro Castellani (:aleca) from comment #128)
Now, I'm the first to admit that my technical knowledge in this scenario is limited, but I trust Magnus' decision and expertise, so I will test this extensively the best I can, and I will leave my review.
Well, this bug needs someone with technical knowledge, otherwise there is no need to review it. It has double r- from the author and copyright owner of the account setup code and a Thunderbird peer. Even in case of a positive review, this must not land.
Besides, if you read my comment #130 carefully, you will see what should land on trunk. That's Ben's patch or patches with the addition of an info message explaining why Exchange was selected and that could have been selected due to an incorrect password. Also note the second paragraph of comment #129.
In an ideal world, we'd transplant the part of Owl that does the exact checking into TB, but that's a different issue.
|
|
||
Comment 132•5 years ago
|
||
I gave my recommendations and explained my reasons, but Ben and Jorg don't agree with my suggestions, and that's OK.
I understood the problem, so writing "This is not good because...<reasons>" over and over will not bring us anywhere.
Reiterating the same concepts is not productive, so please stop.
I think we all have a good grasp of what's the problem, and the approach to solve it differs for a matter of opinions, not misunderstanding.
Opinion 1 - Don't offer Exchange preselected because:
- We can't be 100% sure that IMAP doesn't work. (I know that with OWL we can know this, so please don't write it again)
- We don't want to encourage the installation of a 3rd party add-on that might not be needed.
Opinion 2 - Offer Exchange preselected because:
- Even if IMAP works and it was only a credentials issue, we will know that for sure thanks to Owl.
- We won't offer an error message with a "Done" button to the user, which it can be confusing.
Both are suboptimal solutions which shouldn't be used, but we have to fix it somehow in 68.
Since my suggestion is not accepted by Ben and Jorg, I really don't know what to do here anymore. You asked for my opinion, I gave it to you, you don't like it, so we're stuck again.
Scratch this and actually solve the problem
In an ideal world, we'd transplant the part of Owl that does the exact checking into TB, but that's a different issue.
This seems to be the only logical solution to pursue. Why don't we do that?
If Owl has the ability to actually give us an accurate answer for login and config, why don't we integrate it in TB?
I think this is in the good interest of everyone:
- TB - We know for sure what to suggest to the user because the response is accurate.
- Ben - We won't have any reservation in suggesting Owl to the user since we know for sure that's needed.
- User - No dead ends.
I think we should focus on this, instead of keep battling between opinion 1 and 2, which is clear it won't bring us anywhere.
|
|
||
Comment 133•5 years ago
•
|
||
|
|
||
Comment 134•5 years ago
•
|
||
Reiterating the same concepts is not productive, so please stop.
Well, I'm happy to stop reiterating the same argument the minute that you acknowledge that you're comparing apples with pears. You're comparing a user error edge case which is rather unlikely with a very common case. If we can't have a perfect solution, we should have one that works well in the more likely use case. That's what you said during our call, IIRC. I even tried to mitigate any exchange selection confusion by clearly displaying an info message. Whilst TB cannot accurately tell whether we're dealing with a user error case, the user most likely can. Sadly in your deliberation you have ignored that suggestion, as you have ignored my e-mail to the people involved here.
So in absence of any acknowledgement, the only option I have is to repeat the point. So please adhere to efficient and productive communication before telling others to mend their ways :-(
The patch here has the five problems: 1) It opens an additional vulnerability 2) It affects on-premise Exchange servers 3) It has a UX issue in that a non-functioning option is presented 4) Existing Owl users can't set up a new account 5) it has implementation issues.
Yes, I repeated that now, but again, you didn't address these concerns directly related to the new patch.
I saw a new comment just came in in which you report on testing with the O365 test accounts. That's insufficient since the patch also affects on-premise Exchange accounts where after bug 1598861 the "authoritative" server reply will now be completely ignored. Also, please comment on comment #118, last dot point "Policy question".
|
|
Assignee | |
Comment 135•5 years ago
•
|
||
If Owl has the ability to actually give us an accurate answer for login and config, why don't we integrate it in TB?
We'd essentially put Owl into Thunderbird. It needs to parse web pages to get this specific error. The auth code is highly complex and we constantly improve it. I agree it would be nice, if this was possible, but it's 100x too complex for that. (That 100x is not hyperbole, but an actual value based both on code size and frequency of changes.)
|
|
Assignee | |
Comment 136•5 years ago
•
|
||
I think I can resolve at least one core opinion question you posed:
Alex wrote:
Opinion 1 - Don't offer Exchange preselected ...
Opinion 2 - Offer Exchange preselected ...
This question has been discussed by the TB Council, and the Council has ruled on it. We will offer Exchange preselected, where appropriate.
Note what Magnus wrote in comment 14:
if Exchange was available, AND the normal IMAP/POP auth failed: that would be the appropriate place to advertise this alternative.
That's exactly what I implemented.
I've also fixed bug 1600954 based on Magnus' feedback here. I've honestly tried.
|
|
Reporter | |
Comment 137•5 years ago
•
|
||
(In reply to Jorg K (GMT+1) (PTO to 5th Jan 2020, sporadically reading bugmail) from comment #131)
(In reply to Magnus Melin [:mkmelin] from comment #124)
That password is already sent to get the autodiscover.xml (now, even since exchange support landed). You're grasping at straws that don't exist.
Let me just say that your patch opens more opens more vulnerabilities. Please think to which servers you may send the password in the future and how attackers could fake those servers. Now that I've said this, the bug needs to be security restricted.
You realize that "fixing" that would require gutting out the exchange detection entirely, right? It can't detect a thing without sending credentials.
But it's not a security issue, you're just being purposely misled. We're sending credentials, over a secure connection, to the domain you want to set up an account for. If the server is compromised, you're out of luck, but there's no way to prevent that.
Re policy question: Owl (when already installed) can perfectly well enable showing the exchange option, don't you think?
|
|
||
Comment 138•5 years ago
|
||
Yes, I realise that some attack vectors exist due to the Exchange detection. Please don't remove the security flag from a bug that discussed vulnerabilities. That really doesn't serve anyone. I'll send you information one an additional attack vector your patch introduces in a PM.
|
|
Reporter | |
Comment 139•5 years ago
|
||
(In reply to Jorg K (GMT+1) (PTO to 5th Jan 2020, sporadically reading bugmail) from comment #134)
The patch here has the five problems:
- It opens an additional vulnerability
Nothing additional. And no vulnerability.
- It affects on-premise Exchange servers
It does, but for those it's not even a question of working IMAP or not.
Ben even told Owl could check imap and uninstall if working. In the case for on premise exchange, it will be working and with the patch we even verified the login. I don't see an issue here.
- It has a UX issue in that a non-functioning option is presented
Which might work after changing the password.
- Existing Owl users can't set up a new account
Owl can implement that part. Less than 10 lines of code?
- it has implementation issues.
I'll look at the additions, but those are minor.
|
|
||
Comment 140•5 years ago
|
||
(In reply to Magnus Melin [:mkmelin] from comment #139)
Nothing additional. And no vulnerability.
Sent as PM.
It does, but for those it's not even a question of working IMAP or not.
Ben even told Owl could check imap and uninstall if working. In the case for on premise exchange, it will be working and with the patch we even verified the login. I don't see an issue here.
The issue you keep ignoring is that after bug 1598861 the "authoritative" server reply will now be completely ignored. There is no mandate for this.
Which might work after changing the password.
Correct, it might work. But it certainly will not work in the majority of cases. Not even you deny that. Or do you claim that more than 20% of users mistype the their password? 20% was apparently uptake of MFA according to a mid-2018 study. I know, you convinced most people by now that the "wrong password" case is of paramount importance while ignoring my suggestion to show a message when Exchange is auto-selected alerting to user to checking the password before proceeding.
|
|
Reporter | |
Comment 141•5 years ago
|
||
(In reply to Jorg K (GMT+1) (PTO to 5th Jan 2020, sporadically reading bugmail) from comment #140)
The issue you keep ignoring is that after bug 1598861 the "authoritative" server reply will now be completely ignored. There is no mandate for this.
Funny you should mention this. Suggestion: look at the autodiscover.xml response from the O365 test account where IMAP is enabled.
Spoiler alert: no IMAP listed. Does IMAP work? Oh yes it does. So you're saying Microsoft misconfigured their main world wide server?
|
|
||
Comment 142•5 years ago
•
|
||
Hmm, I don't see any relation to the bug at hand. We already landed bug 1598861 to add "missing" IMAP for AutoDiscover responses. O365 accounts in TB are driven via the ISPDB via autoconfig XML files.
Just to answer what seems like a further attempt to distract: Yes, Micro$oft make Exchange, they sell O365 and Outlook, they classified IMAP as insecure "legacy" protocol, the internet is full of M$-sponsored articles urging people to use MFA and not use IMAP, so why on earth would O365 list IMAP in the AutoDiscover response?
EDIT: Just guessing: Not showing IMAP even if working might be a default config for Exchange servers, that's what might have happened at Aalto.fi.
|
|
Assignee | |
Comment 143•5 years ago
•
|
||
alexca wrote in comment 132:
Scratch this and actually solve the problem
This patch is in direct response to Aleca's suggestion in comment 132, and tries to implement Aleca's suggestion.
We've found a way to check whether the password is valid, for Office365, independent of IMAP and MFA enabled or disabled. That means if you enter a wrong password, we should detect that even before we try IMAP. This allows the user to correct it, then we try IMAP again, and if IMAP is enabled, it can be selected by default.
This considers Magnus concern about the "wrong password" case, solves it in the way Alecs suggested, and should resolve the bug here.
|
|
Assignee | |
Comment 144•5 years ago
•
|
||
|
|
||
Comment 145•5 years ago
|
||
|
|
Assignee | |
Comment 146•5 years ago
|
||
HI Alex,
thanks for looking into this.
- Setting up a regular IMAP account (non O365) doesn't seem to work anymore as the manual config is always returned. I tested it with my Fastmail account and correct credentials and it always goes to
Looking up configuration: trying common server names...and failing.- The successful message
Configuration Found...is showed prematurely on every occasion, leaving the user hanging without any feedback during the login attempt. That shouldn't be shown until the very end of every process.
Thanks. I will look into these.
- Testing with an O365 account with IMAP enabled and WRONG password, the selection defaults to Exchange. It should show the "wrong credentials" error message.
I tested that and that case worked for me and it did show the "username or password invalid" error message. I will check with you personally to find out why it didn't work for you.
|
|
Assignee | |
Comment 147•5 years ago
•
|
||
This should resolve the issues that Alex mentioned.
The case O365 account with IMAP enabled and WRONG password was probably failing due to caching (testing with working password before entering wrong password), so I disabled the cache now. That fixed it for me in my testing.
|
|
||
Comment 148•5 years ago
|
||
|
|
||
Comment 149•5 years ago
|
||
I fixed a few comments and removed some commented-out code.
Checked linting.
Removed prefs hunk. For testing set mailnews.auto_config_url to https://www.beonex.com/autoconfig/test/.
Fixed commit message (and patch format).
Tested:
- Setting up mail @jorgk.com
- Setting up mail @Fastmail
- O365 test accounts with and without working IMAP, both with correct and incorrect password.
I also found an issue :-( - I tested with magnus@openclick384.onmicrosoft.com, at first with the correct password. That defaulted to Exchange since for that account IMAP is disabled. Next I tried with a bad password. That gave the bad password message. Then I corrected the password and the panel doesn't actually clear the "invalid password" message. I thought changing the password should cause a complete "start over"? After correcting the password, I still got the same error. Maybe O365 had locked me out by then?
This patch has been around for a week and Magnus has been rather active during this time. So there was plenty of opportunity for a drive-by review.
This is the r+ from Alex, I'm happy to r+ it of you clear the message if someone changes the password. And maybe check what's happening with that account I mentioned.
|
|
||
Updated•5 years ago
|
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 150•5 years ago
|
||
I corrected the password and the panel doesn't actually clear the "invalid password" message. I thought changing the password should cause a complete "start over"?
Changing the password removes the existing config results, but does not immediately clear the error message. The message is cleared once you click "Continue" and we check the new config. This is what's intended, and also what I see. I'll attach a screencast of what I see, just in case you see something else.
|
|
Assignee | |
Comment 151•5 years ago
|
||
|
|
||
Comment 152•5 years ago
|
||
I corrected the password and the panel doesn't actually clear the "invalid password" message. I thought changing the password should cause a complete "start over"?
As Ben said, changing the password reverts back to the start panel in case the Protocol selection or the Manual config panels are visible.
The error message shouldn't be cleared until the user triggers the autoconfig again, as it's important to have a reference of what went wrong until the user is satisfied with the changes made to the typed info.
|
|
Reporter | |
Updated•5 years ago
|
|
|
||
Comment 153•5 years ago
|
||
|
|
Reporter | |
Comment 154•5 years ago
•
|
||
|
|
||
Comment 155•5 years ago
|
||
We didn't have one of those yet.
|
|
Assignee | |
Comment 156•5 years ago
•
|
||
@Magnus: If you test this, please make sure to apply my patch in attachment 9118083 [details] [diff] [review], not Jörg's.
Would you mind setting up a test account with MFA enabled?
Sure. I just set one up, and sent you the credentials by email.
|
|
Reporter | |
Comment 157•5 years ago
|
||
|
|
Reporter | |
Comment 158•5 years ago
|
||
Will land this shortly, then we can deal with the other issues in other bugs.
|
|
Reporter | |
Comment 159•5 years ago
|
||
|
|
||
Updated•5 years ago
|
|
|
Assignee | |
Comment 160•5 years ago
|
||
this should mention that why we do this is only due to MFA possibly being enabled
MFA is only one reason why IMAP might not work. There are several other reasons. IMAP can also be explicitly disabled.
Please move this if clause to the call site. We don't need yet anther layer of callbacks to make the code less readable.
Given this code is only for Office365, I put it into a separate function instead of putting it all in a common function which had another purpose.
Apart from that, thanks for landing this.
|
|
Reporter | |
Updated•5 years ago
|
|
||
Comment 161•5 years ago
|
||
|
|
||
Updated•5 years ago
|
|
|
Reporter | |
Comment 162•5 years ago
|
||
|
|
||
Comment 163•5 years ago
|
||
|
|
Assignee | |
Comment 164•5 years ago
|
||
The ISPDB change also needs to happen
Yes. The fix here needs to deploy first, though.
|
||
Updated•5 years ago
|
|
|
Assignee | |
Comment 165•5 years ago
|
||
Magnus is attempting to back out the patch here in bug 1528136, with attachment 9135402 [details] [diff] [review]. As mentioned repeatedly here and to the TB Council, the patch here was not only for MFA, but also for several other cases where IMAP is disabled or the user cannot use Thunderbird IMAP for other reasons. The most obvious is that admins can and do disable IMAP on Office365. There are various other options as well. Office365 is extremely configurable, and companies do use this in various ways to restrict access to only the "supported" configuration (usually Windows + MS Outlook plus Outlook Web Access plus ActiveSync), which means it's very hard to tell why a certain login does not work. This patch is still needed, even with MFA support for IMAP on Office365.
Owl supports 3 native Exchange protocols, so this is still helpful for people who cannot use Thunderbird with the native IMAP.
|
|
||
Comment 166•5 years ago
|
||
Just happened: https://hg.mozilla.org/comm-central/rev/62453d8cdbb7
|
|
Assignee | |
Comment 167•5 years ago
•
|
||
Magnus, this backout is in direct violation of TB Council decisions.
You keep claiming that only MFA users needed this, but this is factually wrong, as I've stated a number of times here in this very bug. There are a number of reasons why it might not work, e.g. IMAP being disabled by the domain admin, but also other custom configurations. I've even provided test Office365 accounts that show this exact behavior. You consistently ignore the facts, and you keep pretending that these cases do not exist.
This backout had an r- by me, who wrote most of this module and also still maintains this module, so it cannot land. In addition, your backout is also in direct violation of standing TB Council decisions. The TB Council decision had considered not only the MFA case, but also the disabled IMAP case and the custom configurations.
Most importantly, your backout breaks a certain set of users by offering a non-working config by default. We don't do that. This backout is simply destructive. The fix for this bug here is still needed.
|
|
||
Updated•4 years ago
|
Description
•