1752397 - GMX say Thunderbird is using the POP protocol incorrectly

Status: RESOLVED FIXED

(MailNews Core :: Networking: POP, defect)

Description

[B.J. Herbison]

From https://chat.mozilla.org/#/room/#thunderbird:mozilla.org :
Matrix user @hannah-stern:mozilla.org :
Hi! I happen to be a developer at GMX and we're currently at the issue with adding POP accounts to Thunderbird.
I tested this issue and saw in the TB POP log that, when adding an account with manual setup, clicking "re-test" and then "ok", the client complains that it can't log in.
In the log I see that TB tries a POP connection and issues "AUTH" without any parameter as first command, which is syntactically wrong.

Our POP server got stricter with errors like this (rolled out this last Monday) and disconnects, which triggers the user visible problem.

We'll probably apply a workaround soon, but I'd like to ask if it could be possible to also fix the protocol misuse with Thunderbird.
As workaround for affected users asking here: Instead of clicking OK, click on "advanced settings" which saves the new account to Thunderbird and goes into account settings. After that, manually trigger fetching new messages (right click/get messages) twice (or once after Thunderbird restart). From then on, the periodic fetch will work while the fetch on Thunderbird start won't work.

Comment 1

[Ping Chen [:rnons] (away)]

I didn't find the exact description of AUTH without param in RFCs, but it's used to query supported auth methods. https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-pop3/733bacb3-de44-411f-bfcb-249f0031a7c4

On a side note, bug 1707548 is introducing a new POP3 implementation in JS. I don't know if there are still POP3 servers not supporting CAPA. AUTH without param is not useful if CAPA is supported. Anyway I copied the C++ code to send AUTH and make tests happy.

Comment 2

[Hannah Stern]

The RFC itself specifies AUTH only with one parameter, namely the authentication method. It's not specified that a POP server replies to AUTH with unknown method (including empty string) with any structured error message. Just any "-ERR" can result.

Perhaps there used to be some POP server out there which did reply as described in the Microsoft document that Ping Chen linked, but relying on that isn't portable.

Nowadays, with "CAPA", there's a portable mechanism of finding out if AUTH is supported and if so, which methods, with the "SASL" capability and its parameters.

So I'd suggest using CAPA. If CAPA is not supported, the POP client can still fall back to heuristics though I think for very old POP servers, USER/PASS might be the safer bet, but perhaps there are good reasons to heuristically try AUTH first.

Comment 3

[Hannah Stern]

Just some more information.

The current behavior was probably modeled after the Internet Draft https://datatracker.ietf.org/doc/html/draft-myers-sasl-pop3-05

This never became standardized, but as it sometimes happens, some clients (and I'd guess servers) were implemented along this draft.

The timeline is interesting:

• RFC 1734 (December 1994): Specifies AUTH with mandatory parameter, no method to find out supported mechanisms
• RFC 1939 (May 1996): POP3. No CAPA, no AUTH in this RFC, but refers to RFC 1734 for optional AUTH support. The RFC itself supports, optionally, USER/PASS or the now obsolete APOP.
• draft-mers-sasl-pop3 (01: November 1996, 05: August 1998): Specifies AUTH with optional parameter. Without parameters, supported mechanisms are returned. Note that the formats are different from the Microsoft extensions quoted above. This draft uses "AUTH" without parameter, while Microsoft specifies "AUTH " with space at the end (thus supplying an empty string as parameter).
• RFC 2449 (November 1998): POP3 extension mechanism, specifies CAPA and thus a standardized mechanism to check for feature support, including AUTH methods.

(I found out about this draft by seeing a parallel issue for the K9 Android mail client, by the way.)

Comment 4

[Ping Chen [:rnons] (away)]

I'm not against dropping the empty AUTH, but it takes time to ship it to the users. That's why I suggested the simple workaround to just return +OK.

Comment 5

[Ping Chen [:rnons] (away)]

Attachment: Bug 1752397 - Drop initial empty AUTH command in pop3-js. r=mkmelin

Comment 6

[Ping Chen [:rnons] (away)]

I just read the code, the logs in bug 1752842 and comment 0 again, turns out the response to the initial AUTH doesn't matter. The problem is as you described

Our POP server got stricter with errors like this (rolled out this last Monday) and disconnects

The CAPA after initial AUTH receives no data because you disconnected.

Comment 7

[Ping Chen [:rnons] (away)]

Attachment: Bug 1752397 - Drop initial empty AUTH command in nsPop3Protocol.cpp. r=mkmelin

Depends on D137597.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/cb7605c11e4d
Drop initial empty AUTH command in pop3-js. r=mkmelin
https://hg.mozilla.org/comm-central/rev/90ecd49f236c
Drop initial empty AUTH command in nsPop3Protocol.cpp. r=mkmelin

Comment 10

[Hannah Stern]

(Copied from bug#1752842)

Hi!

Even though this bug is closed now:

I'm glad this will be fixed on Thunderbird's side.

And I've clarified this, yes we can ensure that we'll not drop connections after AUTH-without-parameters for the foreseeable future. This will be at least up to the autumn Thunderbird ESR you named. We'll keep answering "-ERR" as we used to do before we introduced disconnecting, experience has shown that this is okay enough with existing clients. Answering "+OK" would mislead clients implementing draft-myers-sasl-pop3-05 into expecting additional data (authentication mechanisms, and a line with ".") before continuing.

Hope this helps!

Hannah.

Comment 11

[Felix B.]

Thunderbird is still not pulling Mails automatically from GMX with latest upadte.

Comment 12

[Hannah Stern]

(In reply to Felix B. from comment #11)

Thunderbird is still not pulling Mails automatically from GMX with latest upadte.

Did you double check your TB server settings for the account for automatic updates?

If so and automatic update still fails, can you provide logs, using a setup like this: close TB, on a shell (assuming Linux/Unix like syntax, for windows you might need to adapt this): export MOZ_LOG=POP3:5,timestamp; export MOZ_LOG_FILE=$HOME/pop.log; thunderbird

Comment 13

[Felix B.]

Attachment: Log

Hello Hannah,

Logfile seems to remain empty while simply starting TB (account settings should be correct for an immediate pull). 

A manual pull request shows this:

Comment 14

[Felix B.]

Hello Hannah,

Logfile seems to remain empty while simply starting TB (account settings should be correct for an immediate pull).

A manual pull request shows this (see logfile attachment).

Comment 15

[Felix B.]

Attachment: Logfile with timestamp

2nd try (also no logs while simply starting TB).

Comment 16

[Felix B.]

Attachment: Account_Settings.png

Comment 17

[Felix B.]

I have also attached a screenshot of my account settings.

What is surprising me: T-Online and Gmail aren't pulled either when starting TB!

Comment 18

[Ping Chen [:rnons] (away)]

Does bug 1735531 look familiar? Start with thunderbird --devtools to see if there are some errors in the console.

Comment 19

[Felix B.]

Attachment: Account_Settings.png

I have also attached a screenshot of my account settings.

What is surprising me: T-Online and Gmail aren't pulled either when starting TB!

Comment 20

[Felix B.]

Attachment: console.log

This is what I got from the devtools console by simply starting TB (without a manual pull request).

Comment 21

[Felix B.]

Attachment: Console LOG with timestamp by devtools

Console LOG with timestamp by devtools.

Comment 22

[Felix B.]

Hello Ping Chen,

bug 1735531 looks familiar indeed!

--

Going to bed now. Good night!

Comment 23

[Ping Chen [:rnons] (away)]

Yes, that's bug 1735531, the fix will go to the next 91 release in a few weeks. Thanks.

Comment 24

[Felix B.]

Thank you very much for analyzing this!

Comment 25

[Felix B.]

It seems as if TB 91.6.1 is working correctly at the moment!?

Comment 26

[Wayne Mery (:wsmwk)]

(In reply to Felix B. from comment #25)

It seems as if TB 91.6.1 is working correctly at the moment!?

Yes the fix is there per bug 1735531 comment 29