Closed Bug 1538409 Opened 5 years ago Closed 4 years ago

implement OAuth 2.0 authentication for POP accounts

Categories

(MailNews Core :: Networking: POP, enhancement, P2)

Product:
MailNews Core
Component:
Networking: POP
Type:
enhancement

Tracking

(thunderbird_esr68 fixed, thunderbird73 fixed)

Status:
RESOLVED FIXED
Milestone:
Thunderbird 74.0
Tracking Flags:
Tracking Status
thunderbird_esr68 --- fixed
thunderbird73 --- fixed

People

(Reporter: unicorn.consulting, Assigned: mkmelin)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 1 obsolete file)

bug1538409_pop3_oauth.patch
33.93 KB, patch
benc
: review+
mkmelin
: approval-comm-beta+
mkmelin
: approval-comm-esr68+
Details | Diff | Splinter Review

https://developers.google.com/gmail/imap/imap-smtp

The information at the link states POP is included in the oAuth2.0 authentication process at Google. Given the aggressive approach Google is taking to eliminating accounts with less secure apps enabled, I think we should update the ISPDB to use oAuth for POP as well as IMAP and SMTP.

+1, see also bug1565708

Tested with local config file and unfortunately there's no support in Thunderbird for Oauth2 when using POP3.
See comments in bug1174505

(In reply to Petr Hroudný from comment #2)

Tested with local config file and unfortunately there's no support in Thunderbird for Oauth2 when using POP3.
See comments in bug1174505

POP oAuth certainly did not work for Gmail but as there is new information it is time to revisit the setup. At least in my opinion. I do not like oAuth, but at least it does not keep getting disabled by Google in their so called security checks designed to fence the ignorant in.

My above comment meant it's not enough to just change ISPDB.

You need to open enhancement request against Thunderbird's POP3 code, since the support for Ouath is not yet there.
AFAIK, it was implemented only for IMAP and SMTP.

(Yes, there are entries in ISPDB listing Oauth for POP3 [aol, yahoo,..], but since the code is not yet there, they are ignored).

Component: ISPDB Database Entries → Account Manager
Product: Webtools → Thunderbird

I didn't check, but from the comments, I presume that OAuth2 is not implemented for POP3 yet, in Thunderbird. It shouldn't be terribly hard to add, because it should mirror the IMAP implementation, but the code needs to be there first, and POP3 is a completely different implementation, which also works differently internally (IMAP: threads, POP3: state machine).

Only once you can manually configure Google servers with POP3 and OAuth2, and it's tested and proven to work in reality, then you can file a bug to enable it in the ISPDB. (Even then, we would need to keep the password-based config, for older Thunderbirds, whereas "older" is any current TB including TB 70.)

Component: Account Manager → Networking: POP
Product: Thunderbird → MailNews Core

Lack of possibility to link Thunderbird + Gmail + POP3 + OAUTH is limiting security of accounts using POP on Gmail.

Can this bug get on some priority list, as Ben B appears to think it is fairly trivial? This functionality is now becoming fundamental to the program actually working for some folks.

Flags: needinfo?(mkmelin+mozilla)
Summary: GMail supports oAuth2.0 authentication for POP. → implement oAuth authentication for POP accounts

It probably isn't super hard to do yes. Some pointers:
https://searchfox.org/comm-central/source/mailnews/base/util/OAuth2Providers.jsm
https://searchfox.org/comm-central/search?q=MSGIOAUTH2MODULE_CONTRACTID&path=

Why is it becoming more important?

Flags: needinfo?(mkmelin+mozilla)

Its important if you use POP in gmail and want to use 2auth.
Its impossible to use 2auth with POP if there is no OAUTH possible.
So its important security feature. Please implement it, if somebody have the technical knowledge and is not super hard to do.

Flags: needinfo?(mkmelin+mozilla)
Flags: needinfo?(mkmelin+mozilla)
Priority: -- → P2

Is anybody working on that feature?
I mean does anybody know how to do "open enhancement request against Thunderbird's POP3 code"?

That would be this bug.
Nobody is actively working on it atm. If you want, you're free to take it on.

(In reply to Magnus Melin [:mkmelin] from comment #11)

That would be this bug.
Nobody is actively working on it atm. If you want, you're free to take it on.

Sorry for spaming than. Unfortunately I dont posses nessecary skill.

Google has [announced]{https://gsuiteupdates.googleblog.com/2019/12/less-secure-apps-oauth-google-username-password-incorrect.html) that they will be turning off simple authentication for some users in 2020, in favor of enforced OAuth authentication.

For Gmail users who use POP and don't want to change to IMAP, it has become crucial for Thunderbird to implement OAuth for Gmail over POP. Time is running out. Once Google starts to enforce OAuth later in 2020, POP users will have to find another client instead of Thunderbird if OAuth over POP for Gmail isn't implemented by then.

Will someone please consider prioritizing this request, in light of Google's recent announcement?

Thank you.

Please see this support topic. https://support.mozilla.org/en-US/questions/1275810

I am wondering how I am supposed to address this as obviously it will be getting very common over the next 6 months. HAve I gone the right approach?

Doing the sorts of things discussed in the support post with IMAP is a mess, and any "solution" that sees mail routinely moved/copied from IMAP to local folder with filters will eventually lead to data loss. Thunderbird's IMAP implementation just does not cope well with this, the more messages involved at one time the more likely failure.

Successful implementation of this bug well prior to Google stated implementation date of 15th June 2020 is critical to Thunderbird maintaining credibility with those users. I do not feel we can wait for 78, this is going to have to be implements and uplifted to ESR if we want to look in the least proactive.

Attached file [WIP] bug1538409_pop3_oauth.patch (obsolete) —

This is, almost, working. You get the oauth prompt and all that, but something's still wrong. Probably the next state things in nsPop3Protocol::OnSucces. So no mail.

Today's little snippet. This is the text that now appears on Yahoo account settings when you go to access the less secure apps setting.

Allow apps that use less secure sign in (This option will be going away on March 2nd, 2020)

Some non-Yahoo apps and devices use less secure sign-in technology, which could leave your account vulnerable. You can turn off access (which we recommend) or choose to use them despite the risks.

With a link to this useless information
https://help.yahoo.com/kb/account/SLN27791.html?impressions=true

So it would appear that POP access to Yahoo accounts will terminate on the 2nd of March unless something is done very quickly in this bug.

Blocks: 1310389

It is very near to a missed deadline on the Yahoo changes. They will enforce oAuth from March the 2nd, which does not leave very long to get this into a production release. I would assume it would have to be in 68.5 to make that march 2nd deadline to be in user's hands.

Alright, this seems to work for me. Tested with yahoo, aol, and gmail.
Sent to try now: https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=ac3bf953bcb9572a716734f60ecdae8e47608b99

Ben, please review ASAP so we could potentially get it into beta this week and get it released in time.

Assignee: nobody → mkmelin+mozilla
Attachment #9117916 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #9123519 - Flags: review?(benc)
Comment on attachment 9123519 [details] [diff] [review]
bug1538409_pop3_oauth.patch

Review of attachment 9123519 [details] [diff] [review]:
-----------------------------------------------------------------

::: mailnews/local/src/nsPop3Protocol.h
@@ +144,5 @@
>    POP3_FINISH_OBTAIN_PASSWORD_BEFORE_USERNAME,  // 48
>    POP3_OBTAIN_PASSWORD_BEFORE_PASSWORD,         // 49
> +  POP3_FINISH_OBTAIN_PASSWORD_BEFORE_PASSWORD,  // 50
> +
> +  POP3_SUSPENDED,             // 51

Oh, this one I didn't need. I'll remove it locally.
Comment on attachment 9123519 [details] [diff] [review]
bug1538409_pop3_oauth.patch

Review of attachment 9123519 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me, assuming you're happy with the NS_IMPL_ISUPPORTS_INHERITED usage.

::: mailnews/base/prefs/content/am-server.js
@@ +99,5 @@
>  
> +  // OAuth2 are only supported on IMAP and POP.
> +  document.getElementById("authMethod-oauth2").hidden =
> +    serverType != "imap" && serverType != "pop3";
> +  // TLS Cert (External) and OAuth2 only supported on IMAP.

// TLS Cert (External) only supported on IMAP.
maybe, given that OAuth2 now supported on POP3?

::: mailnews/local/src/nsPop3Protocol.cpp
@@ +379,5 @@
>  // nsPop3Protocol class implementation
>  
> +NS_IMPL_ISUPPORTS_INHERITED(nsPop3Protocol, nsMsgProtocol,
> +                            msgIOAuth2ModuleListener, nsIProtocolProxyCallback)
> +

Does this need to list all the inherited classed/interfaces?
nsIMsgAsyncPromptListener and nsIPop3Protocol aren't listed here.
(I still find the XPCOM macro magic a little bit... uh... magical)

@@ +1598,4 @@
>      return NS_ERROR_FAILURE;
>    }
>    MOZ_LOG(POP3LOGMODULE, LogLevel::Debug,
> +          (POP3LOG("Rrying auth method 0x%X"), m_currentAuthMethod));

typo
Attachment #9123519 - Flags: review?(benc) → review+

Thx, yes those macros are a bit magic. I just did what i had to to get it to compile ;)

Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/4a36f8aa2ddd
implement oAuth authentication for POP accounts. r=benc

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 74.0
Attachment #9123519 - Flags: approval-comm-esr68?
Attachment #9123519 - Flags: approval-comm-beta+
Comment on attachment 9123519 [details] [diff] [review]
bug1538409_pop3_oauth.patch

This is showing some test failures (probably). Hmm... will look into them.
Attachment #9123519 - Flags: approval-comm-beta+ → approval-comm-beta?
Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/499ff502287d
followup to pop oAuth2 - re-add nsIMsgAsyncPromptListener impl. rs=bustage-fix
Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/8da7b1c07139
followup 2 - back to original xpcom magic. rs=bustage-fix

Sorry for the noise. It should be all fine now.

Note to self: we need to add the OAuth2 support to pop for gmail. For yahoo it was already there.

Can we uplift this to ESR?

That's the intention. We just have it tried out a bit by beta users first so there's nothing obvious popping up.

Anyone want to verify it's working for them on beta/nightly?

Attachment #9123519 - Flags: approval-comm-esr68? → approval-comm-esr68+

What exactly do users have to do with existing Yahoo POP accounts in Thunderbird 68.5 to have the accounts working again?

Go to Account Settings | Server Settings. Change Authentication Method to OAuth2.

Summary: implement oAuth authentication for POP accounts → implement OAuth 2.0 authentication for POP accounts
Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/25e74f6bf106
back out wrong authentication method selection change. r=me

(In reply to Pulsebot from comment #40)

https://hg.mozilla.org/comm-central/rev/25e74f6bf106

The previous patches was causing test failures for 68. The change is wrong, but I can't remember exactly why I put it there.
It didn't cause test failures elsewhere since only on 68 is the "Advanced Config" disabled if Autoselect is selected as authentication method - this was changed in the redesign. It's not actually a problem when it's set to autoselect - but of course you need to set it to something good before you could connect.

Regressions: 1631437
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: