Closed Bug 1593611 Opened 2 years ago Closed 1 year ago

"Get New Messages for All Accounts" does not get new mails of imap OAuth2 authenticated accounts

Categories

(MailNews Core :: Backend, defect)

Unspecified
All
defect
Not set
normal

Tracking

(thunderbird_esr68 fixed, thunderbird74 fixed)

RESOLVED FIXED
Thunderbird 75.0
Tracking Status
thunderbird_esr68 --- fixed
thunderbird74 --- fixed

People

(Reporter: jmuto, Assigned: gds)

Details

Attachments

(1 file, 3 obsolete files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0

Steps to reproduce:

  1. Thunderbird has POP3 normal password authenticated account, IMAP4 normal password authenticated account and IMAP4 OAuth2 authenticated account.

  2. Send mails to each account.

  3. Menu (Burger button) - [File] - [Get New Messages for] - [All Accounts] .

Actual results:

  1. Get new messages from normal password authenticated accounts (POP3 normal password and IMAP4 normal password). But it does not get new messages from IMAP4 OAuth2 authenticated account.

Expected results:

  1. Get new messages from OAuth2 and normal password authenticated accounts (POP3 normal password, IMAP4 normal password and IMAP4 OAuth2).

My environment:
Thunderbird 68.2.1 (32-bit)
Windows 10 Pro x64 1903 (18362.418)

bug 1592407 should have fixed oauth in 68.2.1

Flags: needinfo?(jorgk)
Summary: "Get New Messages for All Accounts" does not get new mails of OAuth2 authenticated accounts → "Get New Messages for All Accounts" does not get new mails of imap OAuth2 authenticated accounts

(In reply to Wayne Mery (:wsmwk) from comment #2)

bug 1592407 should have fixed oauth in 68.2.1

This report is not authentication issue.

"Menu (Burger button) - [File] - [Get New Messages for] - [Current Account]" works fine with IMAP4 OAuth2 authenticated account.

It seems that "Get New Messages for All Accounts" ignores OAuth2 authenticated accounts.

I'm not sure how "Get New Messages for All Accounts" is meant to work exactly. I remember some work in this area in bug 281417. Another recent bug was bug 1590514.

Flags: needinfo?(jorgk) → needinfo?(acelists)

I verified the bug but only on 60.8. Only gmail (my only oauth) won't bring in new messages when "on all accounts" is selected. I did have to disable IDLE (immediate notification) on gmail to see the problem.
Edit: Don't remember how I tested this but immediate notification really has no effect. See edit for comment 7.

The 'Get New Message for all accounts function' does not work.
F5 does nothing
F9 does nothing
Shift-F5 does nothing
Shift-F9 does nothing
Only 'Get New Message for ' for a specific account works.

Am I the only one with this bug ?
I am using 68.2.2 (64-bit) on Linux Mate.

I'm on ubuntu linux with kde and it seems to work. I hit f5 and in the status area at the bottom it sweeps through the 3 accounts I have active. I didn't look in detail with a imap log or wireshark, but in comment 5 I verified this works except with oauth2 with immediate notification (imap idle) disabled. I assume you are only wanting new mail checked in Inbox of each account?

Edit: Actually F5 just checks the currently selected account. Have to do shift-f5 to sweep through them all. However, looking again, shift-f5 skips my on oauth2 account, gmail, regardless of whether immediately notification is checked or not.

Every 10 minutes all accounts get the messages into principal inbox and that is correct with me.

I checked with the safe mode and there is also no response to F5 and Shift-F5.
The only function keys working on TB
are F1 (Help) ; F10 (File) and F11 (Tasks).

My problem is that the function
File->Get new messages for -> All accounts
does not work.
Even in SAFE MODE .

Roger, Are your accounts using oauth like gmail and possible yahoo? How many accounts do you have and what type of security and authentication do they use?
I retested this on tb 60.9 with 3 tls/normal password accounts and 1 oauth2/normal password accounts and recorded an imap log. Before recording the log I set the server setting "check new msgs at startup", "check new messages every n Sec" and "immediate notification" all to off (unchecked) on ALL accounts and left tb on a non-inbox folder so when it starts up to record the log it only connects to that non-inbox mailbox/folder. Then I wait a few seconds and do shift-f5 and shutdown tb. Looking at the log, I see that all the tls based accounts are checked for new messages in Inbox. However, gmail is not checked or connected to at all with shift-f5. So this confirms the original subject of this bug. (See edit of comment 7 and comment 5.)

So if your accounts are not all oauth based, you might try recording an IMAP:5 log like I describe above. Be sure to include the "timestamp" item and wait a few seconds before hitting shift-f5 (or doing "get new messages for all accounts") so we can see in the log when the event occurs. See https://wiki.mozilla.org/MailNews:Logging for detail on record the imap log. You can attach it using the link above.

Flags: needinfo?(webmaistre.le.bilboquet)

Dear Gene Smith,
I know nothing about oauth.
3 of my accounts are gMail and use SSL/TLS.
8 off my accounts are general pop3 accounts and use STARTTLS.

How do I record an imap log ?
Thanks

Flags: needinfo?(webmaistre.le.bilboquet)

(In reply to Roger Gravel from comment #10)

How do I record an imap log ?

(In reply to gene smith from comment #9)

... recording an IMAP:5 log [and use the setup described above at " Before recording the log ...."] . Be sure to include the "timestamp" item and wait a few seconds before hitting shift-f5 (or doing "get new messages for all accounts") so we can see in the log when the event occurs. See https://wiki.mozilla.org/MailNews:Logging for detail on record the imap log. You can attach it using the "attach new file" link of the bug report page.

Flags: needinfo?(webmaistre.le.bilboquet)

(In reply to Roger Gravel from comment #10)

Dear Gene Smith,
I know nothing about oauth.
3 of my accounts are gMail and use SSL/TLS.

Shift-F5 doesn't check the gmail accounts, right? Also, I assume they use oauth2 for "Authentication Method" as set on the "Server Settings" configuration screen?
Are you expecting to see new gmail only in Inbox or also in other folders too?

8 of my accounts are general pop3 accounts and use STARTTLS.

Do these check for new with shift-f5?

How do I record an imap log ?

If my assumption are correct, you probably don't need to record an imap log since I can duplicate the problem with oauth2 authenticated accounts when shift-f5 is typed.

Flags: needinfo?(acelists)

This is going no where.
All I want is to be able to
GET ALL NEW MESSAGES

Flags: needinfo?(webmaistre.le.bilboquet)

(In reply to Roger Gravel from comment #13)

This is going no where.
All I want is to be able to
GET ALL NEW MESSAGES

Thanks for the ping. I'm looking at this again right now...

For some unknow reason, for gmail accounts that use oauth2, currentServer.passwordPromptRequired is getting set true. I can't find anywhere it is actually being set true or false so this may be an uninitialized variable. When true, this causes the server to not be scanned for new mail because a password would need to be prompted for. Both of my gmail accounts had this problem and I assume the reporter and other commenters have the same issue.

I've done a test fix by ignoring the passwordPromptRequired when the auth type is oauth2 which is the value 10. Now the gmail accounts are checked for new mail when ctrl-f5 is pressed.

So my question is why is the server attribute passwordPromptRequired set true for oauth2 accounts? Or is this actually an uninitialized variable that is randomly true or false?

Attachment #9126645 - Flags: feedback?(mkmelin+mozilla)
Attachment #9126645 - Flags: feedback?(acelists)
Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Unspecified → All
Attachment #9126645 - Attachment is patch: true
Comment on attachment 9126645 [details] [diff] [review]
Temporary workaround fix -- not a patch

Review of attachment 9126645 [details] [diff] [review]:
-----------------------------------------------------------------

I'm assuming it gets set here? https://searchfox.org/comm-central/rev/ed9f13d96ef6f1834449eeef9954b206679e6845/mailnews/base/util/nsMsgIncomingServer.cpp#1645
Attachment #9126645 - Flags: feedback?(mkmelin+mozilla)
Attachment #9126645 - Flags: feedback?(acelists)

Yes, that does seem to "set" it even though the function is a "get".

With oauth2 there seems to be no way to not store the credentials locally so that the username/password prompt is forced to occur on each restart. You can delete the oauth2 account line in password mgr but it comes back when you restart and do crtl-f5 or click on a gmail folder and re-enter your credentials with the special oauth2 entry screen. After that you are not prompted for credentials again unless you remove the line again in password mgr.

What I'm getting at is that currentServer.passwordPromptRequired for oauth2 gets set incorrectly to true.

Is it OK to just do what I did in the diff? It uses the hardcode value of 10 for oauth2 accounts. I didn't see a symbol to use. If it's OK, I'll go ahead and submit a formal patch.

Wayne, are you thinking this might be a regression? The only change regarding oauth2 I know of is bug 1176399. But maybe there are others.

I'll go ahead and submit a patch. I found a symbol for oauth2 that evaluates to 10 so I don't have to hardcode it.

Assignee: nobody → gds
Attachment #9126645 - Attachment is obsolete: true
Attachment #9127179 - Flags: review?(mkmelin+mozilla)
Comment on attachment 9127179 [details] [diff] [review]
check-oauth2-accts-on-shift-f5.patch

Review of attachment 9127179 [details] [diff] [review]:
-----------------------------------------------------------------

How about instead at https://searchfox.org/comm-central/rev/ed9f13d96ef6f1834449eeef9954b206679e6845/mailnews/base/util/nsMsgIncomingServer.cpp#1645 check AuthMethod and set it to false for authMethod == nsMsgAuthMethod.OAuth2

That would make oauth less special for the other passwordPromptRequired cases.
Attachment #9127179 - Flags: review?(mkmelin+mozilla)

Like this?

Attachment #9127179 - Attachment is obsolete: true
Attachment #9127420 - Flags: review?(mkmelin+mozilla)

Got rid of unneeded "if".

Attachment #9127420 - Attachment is obsolete: true
Attachment #9127420 - Flags: review?(mkmelin+mozilla)
Attachment #9127436 - Flags: review?(mkmelin+mozilla)
Comment on attachment 9127436 [details] [diff] [review]
check-oauth2-accts-on-shift-f5(try3).patch

Review of attachment 9127436 [details] [diff] [review]:
-----------------------------------------------------------------

Yes, looks good. I'll update the commit message a bit.
Attachment #9127436 - Flags: review?(mkmelin+mozilla) → review+
Status: NEW → ASSIGNED
Component: Mail Window Front End → Backend
Product: Thunderbird → MailNews Core
Target Milestone: --- → Thunderbird 75.0

Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/7a6415f8b8cd
For accounts with OAuth2 authentication, make passwordPromptRequired false since the normal password won't be used. r=mkmelin

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Attachment #9127436 - Flags: approval-comm-esr68?
Attachment #9127436 - Flags: approval-comm-beta?
Comment on attachment 9127436 [details] [diff] [review]
check-oauth2-accts-on-shift-f5(try3).patch

[Triage Comment] Pretty straightforward. Hard to believe we've been living with this.
Attachment #9127436 - Flags: approval-comm-beta? → approval-comm-beta+
Attachment #9127436 - Flags: approval-comm-esr68? → approval-comm-esr68+
You need to log in before you can comment on or make changes to this bug.