1982007 - Sending through smtp-relay.google.com broken in 140.0.1esr (email domain is in Google Workspace)

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[C Haley]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36

Steps to reproduce:

My email domain is in Google Workspace. I send through smtp-relay.gmail.com. This works in 128.13.0esr and before. It stopped working in 140.0.1esr and later.

I'm using the following setup: Server name: smtp-relay.gmail, Port: 587, STARTTLS, Oauth2. In my google workspace, gmail relaying is turned on with "Require SMTP Authentication" and "Require TLS encryption".

Note: changing smtp-relay.gmail.com to smtp.gmail.com works to send mail but doesn't handle the multiple username setup we have.

Actual results:

When sending mail, the send times out after a minute or two. Looking in the error console I see only a POST to (IIRC) o.pki.goog then nothing else.

Expected results:

The email should be sent.

Comment 1

[Kai Engert [:KaiE:]]

How could we reproduce this bug?

Is there a mismatching TLS server certificate involved possibly?

Comment 2

[C Haley]

I'm sure the server certificate is fine.

I have a laptop I use when traveling that I'm willing to "break". What I see:

• Using 128.13.0esr, sending email works.
• Upgrade to 140.1.1esr. I didn't reboot.
• Now sending email I get the error "Sending of the message failed. The message could not be send because the connection to Outgoing server (SMTP) smtp-relay.gmail.com timed out".
• Rebooting/restarting Thunderbird doesn't fix it.

Here is the OAuth2 smtp setup that works in 128.13.0esr but doesn't in 140.1.1esr.

user_pref("mail.smtpserver.smtp2.authMethod", 10);
user_pref("mail.smtpserver.smtp2.clientid", "46527270-7a2a-4077-8485-c5f4cdb5f37d");
user_pref("mail.smtpserver.smtp2.description", "Google relay service");
user_pref("mail.smtpserver.smtp2.hostname", "smtp-relay.gmail.com");
user_pref("mail.smtpserver.smtp2.oauth2.issuer", "accounts.google.com");
user_pref("mail.smtpserver.smtp2.oauth2.scope", "https://mail.google.com/");
user_pref("mail.smtpserver.smtp2.port", 587);
user_pref("mail.smtpserver.smtp2.try_ssl", 2);
user_pref("mail.smtpserver.smtp2.type", "smtp");
user_pref("mail.smtpserver.smtp2.username", "xxxxxxx"); (name elided)

It still fails if I change the username to something that doesn't exist, for example "[e-mail]". It doesn't ask me to re-authenticate.

If I change from OAuth2 to "Normal Password" then I get prompted for my credentials. However, it doesn't work because smtp-relay.gmail.com requires OAuth2 to verify that the sending account is a Google account.

On a working 128.13.0esr, if I use smtp-relay.gmail.com for a "normal" gmail account ([e-mail]), it works. If I do the same thing with 140.1.1esr, it doesn't with the same result as above (timeout).

What all this tells me is that you should be able reproduce the problem using the above smtp settings with some gmail username/account.

Comment 3

[Magnus Melin [:mkmelin]]

Oh yeah we don't use the issuer/scope prefs from the settings anymore (since bug 1927998)
I guess smpt-relay needs to be added from https://searchfox.org/comm-central/rev/50f7fd4d2d7d5a42e8e6abe0b5f74e4beaab7ab6/mailnews/base/src/OAuth2Providers.sys.mjs#52

I'm curious how you got those into the prefs in the first place though? Did you add them manually in the past?
They used to get automatically saved but that would be only after we figured out we needed them in the first place.

Comment 4

[Geoff Lankow (:darktrojan) [PTO, back 2 Nov]]

Attachment: Bug 1982007 - In OAuth2 providers list, use wildcards for Gmail domains. r=mkmelin

This will match all gmail.com and googlemail.com domains, instead of just the specific ones we had listed.

Comment 5

[C Haley]

(In reply to Magnus Melin [:mkmelin] from comment #3)

I'm curious how you got those into the prefs in the first place though? Did you add them manually in the past?
They used to get automatically saved but that would be only after we figured out we needed them in the first place.

I did it years ago. I wasn't given OAuth2 as an authentication option setting up using smtp-relay.gmail.com (and am still not using 128.13.0esr) so I copied the OAuth2 settings from an account that used smtp.gmail.com and changed the server name.

Comment 6

[Geoff Lankow (:darktrojan) [PTO, back 2 Nov]]

To me, it seems like an odd thing to be doing, but whatever. We'll make this change and Thunderbird will stop preventing you from doing it.

Comment 7

[C Haley]

(In reply to Geoff Lankow (:darktrojan) from comment #6)

To me, it seems like an odd thing to be doing, but whatever. We'll make this change and Thunderbird will stop preventing you from doing it.

Thanks. To explain, we have users with multiple (as many as 20) different "From" addresses. smtp.gmail.com won't use one of these unless it is listed in "Send mail as" in gmail.com settings, instead changing the "From" to the primary email address. The addresses come and go fairly frequently so putting them as alternates isn't practical. The server smtp-relay doesn't enforce or change "from" addresses, thus solving our problem.

It is also mandatory when using a front-end (internal) smtp server as a relay because all the email is relayed using a single account. See https://support.google.com/a/answer/2956491 for more info.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/d1b3c14f810d
In OAuth2 providers list, use wildcards for Gmail domains. r=mkmelin

Comment 9

[Geoff Lankow (:darktrojan) [PTO, back 2 Nov]]

This fix will be in 143.0b1 which will be out this week. Please verify it works. You should be able to get it working in a new profile just by setting up the account and setting the SMTP hostname, no changes to preferences should be needed.

Comment 10

[C Haley]

It works in 143.0b1.

FWIW: the beta crashes when I give it my previous profile. I had to create a new one then add the necessary accounts to test. And yes, the only thing I changed in the new profile was the SMTP server.

Comment 11

[Geoff Lankow (:darktrojan) [PTO, back 2 Nov]]

Comment on attachment 9507076 [details]
Bug 1982007 - In OAuth2 providers list, use wildcards for Gmail domains. r=mkmelin

Uplift Approval Request

• Please state case for uplift consideration and ensure bug severity is set: Just a configuration change, no code changes.
• User impact if declined: Users can't send using smtp-relay.gmail.com.
• Is this code covered by automated tests?: No
• Has the fix been verified in Daily?: No
• Has the fix been verified in Beta?: Yes
• Needs manual test from QA?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This just expands the domains covered by the Google OAuth provider.
• Does the fix cause any migrations to be skipped?: No
• String changes made/needed:

Comment 12

[Corey Bryant]

Comment on attachment 9507076 [details]
Bug 1982007 - In OAuth2 providers list, use wildcards for Gmail domains. r=mkmelin

[Triage Comment]
Approved for esr140

Comment 13

[Stephen King (:sking)]

Thunderbird 140.2.1esr:
https://hg.mozilla.org/releases/comm-esr140/rev/5df036ddfe367ff4452a4aa6235bb652dca89fa4