1912556 - sending SMTP email via Office365 times out

Status: RESOLVED FIXED

(MailNews Core :: Networking: SMTP, defect, P1)

Description

[Calum Mackay]

In Daily 0809, on MacOS 13.6.7, I am unable to send emails via office365.com:

The message could not be sent because the connection to Outgoing server (SMTP) smtp.office365.com timed out. Try again.

SMTP server settings:

Server Name: smtp.office365.com
Port: 587
Authentication method: OAuth2
Connection Security: STARTTLS

After the timeout, TB starts to behave oddly, not loading emails etc. Trying to Quit leaves it running, requiring a Force Quit.

Emails sent via other accounts are fine, including gmail.com

This problem is new in Daily 0809, where it occurs for every email sent via that O365 account.

Daily 0808 does not show the problem with the same account, which would seem to rule out external issues like firewalls, etc.

Comment 1

[Calum Mackay]

For the Daily 0809 failing case, the Error Console shows:

02:43:45.185 mailnews.oauth: Scope "https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.Send offline_access" was requested, but "https://outlook.office.com/EWS.AccessAsUser.All https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.Send" was granted OAuth2.sys.mjs:382:22
02:43:45.193 NS_ERROR_FAILURE: This login already exists. LoginHelper.sys.mjs:1847
02:43:45.196 NS_ERROR_FAILURE: This login already exists. LoginHelper.sys.mjs:1847

whereas in the successful 0808 case, nothing is shown.

Comment 2

[Magnus Melin [:mkmelin]]

Must be from bug 1909789.
Seems I can reproduce.

Comment 3

[Geoff Lankow (:darktrojan)]

We don't get granted the offline_access scope any more. I don't know when it happened, or if it ever worked. But now that we're actually taking some notice of the scopes that are granted, it's not there and that's a problem.

I think if we stop asking for that scope it all works as intended. But I could be wrong.

Comment 4

[Geoff Lankow (:darktrojan)]

Attachment: Bug 1912556 - Remove offline_access from the OAuth scopes we request from Microsoft. r=#thunderbird-reviewers

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1be42fa6836b
Remove offline_access from the OAuth scopes we request from Microsoft. r=tobyp

Comment 6

[Magnus Melin [:mkmelin]]

Just tested this.
Unfortunately, the offline token needs to be requested - or otherwise there's no usable refresh token.
So I can now send, but will go through the OAuth2 authentication process for each send :(

Comment 7

[Francesco]

Is "thunderbird130: unaffected" correct? If this is a regression from bug 1909789 and that bug got backported to 130 beta, then 130 is also affected. The same goes for the second regression of bug 1909789, which is bug 1912738.

Comment 8

[Francesco]

I tried this in Daily with an O365-based account. No prompt when sending a message via SMTP. However, bug 1912738 is still present.

Comment 9

[Wayne Mery (:wsmwk)]

(In reply to Francesco from comment #7)

Is "thunderbird130: unaffected" correct? If this is a regression from bug 1909789 and that bug got backported to 130 beta, then 130 is also affected. The same goes for the second regression of bug 1909789, which is bug 1912738.

It was unaffected at the time those fields were set. But yes, now, they are out of date.

Comment 10

[Magnus Melin [:mkmelin]]

Retesting I don't quite see my comment 6. Maybe or maybe not - it's only, at least, after restart reproducible. May be bug 1912738 or not.

Comment 11

[Francesco]

Reporter, please test this in a Daily and if it works for you (as it does for me), please close the bug as FIXED again. Maybe you can also check bug 1912738.

Comment 12

[Toby Pilling [:tobyp]]

Attachment: Bug 1912556 - Allow Scope requests to include parameters such as offline_access without failing. r=aleca,#thunderbird-reviewers

Comment 13

[Calum Mackay]

(In reply to Francesco from comment #11)

Reporter, please test this in a Daily and if it works for you (as it does for me), please close the bug as FIXED again. Maybe you can also check bug 1912738.

hi Francesco, since there still seems to be development activity here, from Toby, I'll leave it alone for now.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/64b27b028b68
Allow Scope requests to include parameters such as offline_access without failing. r=darktrojan

Comment 16

[Geoff Lankow (:darktrojan)]

Comment on attachment 9418887 [details]
Bug 1912556 - Remove offline_access from the OAuth scopes we request from Microsoft. r=#thunderbird-reviewers

[Approval Request Comment]
(This patch is somewhat obsoleted by the other patch in this bug, but it contains a bit that is still necessary.)

Comment 17

[Geoff Lankow (:darktrojan)]

Comment on attachment 9419101 [details]
Bug 1912556 - Allow Scope requests to include parameters such as offline_access without failing. r=aleca,#thunderbird-reviewers

[Approval Request Comment]
Regression caused by (bug #): bug 1909789
User impact if declined: users with Microsoft accounts get repeatedly prompted for authorisation
Testing completed (on c-c, etc.): this fix is only in the latest Daily, but we've tested a bunch of different providers manually
Risk to taking this patch (and alternatives if risky): low…ish.

Comment 18

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Hello,

Confirming this issue(and all related regressions) only as partially fixed:

• Closing and opening TB after authenticating with O365(ESW enabled) does not prompt again for credentials.
• Adding a new Gmail account is set up without any issues and emails are downloaded successfully.
• Subsequent restarts(or normal quit/re-open) do not prompt for authentication.
• Sending/Receiving emails from the Google IMAP account works successfully.
• Receiving email on the O365 account works successfully.
• Sending email from the O365 account times out(stuck) in the "Copying message to Sent folder " process with the following error console:
  mail.notification: onFolderIntPropertyChanged: [Exception... "Component returned failure code: 0x80004001 (NS_ERROR_NOT_IMPLEMENTED) [nsIMsgFolder.folderURL]" nsresult: "0x80004001 (NS_ERROR_NOT_IMPLEMENTED)" location: "JS frame :: resource:///modules/MailNotificationService.sys.mjs :: onFolderIntPropertyChanged :: line 158" data: no] MailNotificationService.sys.mjs:168:17

Tested this with the treeherder build 131.0a1(20240815220039) on Windows 11, macOS 14 and Ubuntu 24.

Comment 19

[Francesco]

FWIW, with a new profile and this build 131.0a1 (2024-08-16) (64-bit), 20240816123541, I get no repeated auth prompts on restart, so bug 1912738 appears to be fixed. Sending works fine, too. I was using IMAP on a O365-based account. I didn't try the EWS option.

Comment 20

[Corey Bryant]

Comment on attachment 9419101 [details]
Bug 1912556 - Allow Scope requests to include parameters such as offline_access without failing. r=aleca,#thunderbird-reviewers

[Triage Comment]
Approved for beta
(2 patches)

Comment 21

[Toby Pilling [:tobyp]]

(In reply to Francesco from comment #19)

FWIW, with a new profile and this build 131.0a1 (2024-08-16) (64-bit), 20240816123541, I get no repeated auth prompts on restart, so bug 1912738 appears to be fixed. Sending works fine, too. I was using IMAP on a O365-based account. I didn't try the EWS option.

It is worth a whole lot Francesco. Thanks very much for taking the time to verify this!

Comment 22

[Rob Lemley [:rjl]]

Comment on attachment 9418887 [details]
Bug 1912556 - Remove offline_access from the OAuth scopes we request from Microsoft. r=#thunderbird-reviewers

[Triage Comment]
Marking approved to fix bugzilla uplift querying later.

Comment 23

[Rob Lemley [:rjl]]

Thunderbird 130.0b2:
https://hg.mozilla.org/releases/comm-beta/rev/515b80359971
https://hg.mozilla.org/releases/comm-beta/rev/7cffda2900eb

Comment 24

[Corey Bryant]

Comment on attachment 9418887 [details]
Bug 1912556 - Remove offline_access from the OAuth scopes we request from Microsoft. r=#thunderbird-reviewers

[Triage Comment]
Approved for esr128
Approved for esr115

Comment 25

[Calum Mackay]

I've been ill, so hadn't previously had a chance to check on this.

Trying Daily 20240820 today, I find that SMTP sending still hangs, to O365, and now I can't read IMAP from that account either (which was working before, even with the SMTP issue). And TB hangs on app shutdown (force quit required) as before.

This could just be a coincidence (new bug, but same symptoms?), but seems a little odd.

Back to Daily 0808 and all is fine again, both IMAP & SMTP, suggesting that it's TB, rather than something external.

I'll try and collect more data tomorrow.

Comment 26

[Francesco]

Interesting, I retested, and it's still working for me on today's Daily. As I wrote in comment #19, I created a new profile. Can you try a new profile? I understand that the scopes are stored as prefs, if I look for oauth2.scope I get these scopes for both incoming and outgoing server:

https://outlook.office.com/EWS.AccessAsUser.All https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.Send offline_access

Comment 27

[Magnus Melin [:mkmelin]]

Calum: Like Francesco wrote, check the mail.server.server<NN>.oauth2.scope pref and fix it, or remove it (should get auto-recreated). You may also need to find the relevant old oauth login in the logins and delete it.

Comment 28

[Calum Mackay]

Thanks very much, Francesco & Magnus (and Toby).

All good now. I deleted the oauth2.scope for that server, and restarted.

Sending SMTP, and reading IMAP, now both work fine, with Daily 0822, and I'm not re-prompted for oauth on TB restart.

thanks very much again, for the bug fixes, and the support.

Comment 29

[Magnus Melin [:mkmelin]]

Vlad, can you test this and its sister bug 1909789 for the upgrade from 128 scenario? I think testing EWS is in general not needed yet - too much still wip. Comment 18 makes it unclear what type of account this was.

Have a fresh profile created using 128 (gmail, and o365 imap accounts both using OAuth2).
Upgrade to 130beta2.
Access mails.
Send mails.
There should be no reprompts

Comment 30

[Vlad Lucaci, Thunderbird QA (:vlucaci)]

Confirming this issue as verified fixed on 130.0b2(20240816160526) and 130.0b3(20240826151843) using macOS 14 , Windows 11 and Ubuntu 22 using the STR from comment 29.

Comment 31

[Geoff Lankow (:darktrojan)]

Comment on attachment 9419101 [details]
Bug 1912556 - Allow Scope requests to include parameters such as offline_access without failing. r=aleca,#thunderbird-reviewers

[Approval Request Comment]
See bug 1909789 comment 21.

Comment 32

[Wayne Mery (:wsmwk)]

correcting status flags

Comment 33

[Geoff Lankow (:darktrojan)]

Comment on attachment 9418887 [details]
Bug 1912556 - Remove offline_access from the OAuth scopes we request from Microsoft. r=#thunderbird-reviewers

This landed on esr128 in error and was backed out. Changing the flag back to ? to avoid confusion.

[Approval Request Comment]
See bug 1909789 comment 21. Both pieces of this patch are replaced by the other patch here, which won't land cleanly without this one.

Comment 34

[Corey Bryant]

Comment on attachment 9418887 [details]
Bug 1912556 - Remove offline_access from the OAuth scopes we request from Microsoft. r=#thunderbird-reviewers

[Triage Comment]
Approved for esr128.

Chatted with Geoff and while there is still risk, we'd like to get this in while the 128 user base is still somewhat low.

Comment 35

[Corey Bryant]

Comment on attachment 9419101 [details]
Bug 1912556 - Allow Scope requests to include parameters such as offline_access without failing. r=aleca,#thunderbird-reviewers

[Triage Comment]
Approved for esr128

Comment 36

[Daniel Darnell [:dandarnell]]

Thunderbird 128.2.2esr:
https://hg.mozilla.org/releases/comm-esr128/rev/1f8301f04627
https://hg.mozilla.org/releases/comm-esr128/rev/a61f88fd0c26

Comment 37

[Daniel Darnell [:dandarnell]]

Backed out in Thunderbird 128.2.3esr:
https://hg.mozilla.org/releases/comm-esr128/rev/9b84e9ae80b9
https://hg.mozilla.org/releases/comm-esr128/rev/4c20f8ccb108

Comment 38

[Geoff Lankow (:darktrojan)]

I'm setting this (and the two other bugs) back to fixed, as that's the normal state for bugs that have landed on central. When we manage to find and resolve the problems this caused on 128, I'll make one super-patch of all the changes as I'm tired of dragging 5 patches around.