1800289 - AOL/Yahoo's UIDONLY IMAP extension

Status: NEW

(MailNews Core :: Networking: IMAP, enhancement, P4)

Description

[jkhsjdhjs]

Hey,

some of you may have already heard about limitations recently introduced by AOL and Yahoo mailservers, namely that they arbitrarily limit the number of messages one can access via standard IMAP on their main server to 10.000. This has been discussed in great length in the microsoft forums [1]. As Mark_302 found out, AOL/Yahoo are implementing an IMAP draft extension on their production systems, which was developed mainly by Yahoo engineers and is named UIDONLY [2,3]. If an IMAP client doesn't implement UIDONLY, they limit the number of accessible messages to 10.000. The client notifies the server about UIDONLY support by enabling the UIDONLY capability. I implemented a dirty hack in Thunderbird such that it enables UIDONLY when connecting to an IMAP server, but the extension seems to change too much about the protocol such that Thunderbird then isn't able to retrieve any email at all.

The official workaround proposed AOL [4] and Yahoo [5] is to switch the server to export.imap.yahoo.com (the default is imap.yahoo.com). This does indeed allow one to access all messages from the server, but (at least for me) Thunderbird constantly re-downloads all messages. This behavior has also been reported by some people in the microsoft forums [6]. To verify this behavior I backed my ~/.thunderbird directory up, started Thunderbird with the environment variables

MOZ_LOG=IMAP:5,timestamp
MOZ_LOG_FILE="$HOME/thunderbird_imap.log"

and added my Yahoo account again and had Thunderbird re-download all messages from scratch, which took about 5.5 hours. I didn't interact with Thunderbird during the download process and afterwards, just kept it running. One hour after Thunderbird had successfully downloaded all messages, it started re-downloading all messages again, see the log [7]. At least for me the log file doesn't seem to indicate why the messages are being re-downloaded. I still have the full log available, but it's 36G in size, so I only uploaded an excerpt. If additional bits may be helpful please tell me what I should look for and I'll provide it.

The question now is what can and what should be done about this issue. Although I don't use my Yahoo account that actively anymore, I would like to keep using my Yahoo mail account with Thunderbird, ideally with all emails available. And there are apparently others who would like to do that, too. I'm not sure if it would be reasonable to implement UIDONLY in Thunderbird, since it's just a draft and it's only used by AOL and Yahoo. So another solution would be to figure out what causes Thunderbird to re-download all messages.

But first of all, I would love to hear your thoughts on this topic. What do you think about the UIDONLY extension? Any ideas why Thunderbird re-downloads all messages when the export.imap.yahoo.com server is used?

Best Regards,
Leon

[1] https://answers.microsoft.com/en-us/outlook_com/forum/all/why-is-an-imap-inbox-only-displaying-10000-items/6d15de5f-9047-4b41-9b58-1d8345bbd002
[2] https://answers.microsoft.com/en-us/outlook_com/forum/all/why-is-an-imap-inbox-only-displaying-10000-items/6d15de5f-9047-4b41-9b58-1d8345bbd002?page=6 (scroll down to the reply by Mark_302, it doesn't seem to be possible to link individual replies)
[3] https://www.ietf.org/archive/id/draft-melnikov-imap-uidonly-00.html
[4] https://help.aol.com/articles/download-your-email-from-aol-mail-with-imap
[5] https://help.yahoo.com/kb/new-mail-for-desktop/download-email-yahoo-mail-third-party-sln28681.html
[6] https://answers.microsoft.com/en-us/outlook_com/forum/all/why-is-an-imap-inbox-only-displaying-10000-items/6d15de5f-9047-4b41-9b58-1d8345bbd002?page=27 (see the replies by Paul Dickler and BDSolomon)
[7] https://gist.github.com/jkhsjdhjs/8536abbf449f36a506f1aac53fe2bd36

Comment 1

[Ben Campbell]

(In reply to jkhsjdhjs from comment #0)

I implemented a dirty hack in Thunderbird such that it enables UIDONLY when connecting to an IMAP server, but the extension seems to change too much about the protocol such that Thunderbird then isn't able to retrieve any email at all.

From my quick reading of UIDONLY, it requires the client to handle UIDFETCH responses instead of standard FETCH responses. And TB doesn't seem to check for or handle UIDFETCH as far as I can see. Some other commands are similarly changed (eg EXPUNGE returns a different response...)
So yeah, there'd need to be some bigger changes in TB IMAP to get UIDONLY working.

But first of all, I would love to hear your thoughts on this topic. What do you think about the UIDONLY extension?

It's a draft RFC, so I'd say we just sit on it for now and look again if/when it gets standardised. From my superficial reading of it, I'm not entirely convinced it adds enough benefit over existing IMAP to justify the complexity required for clients to implement it...
(I mean, IMAP already handles folders with more than 10000 message in them...)

Any ideas why Thunderbird re-downloads all messages when the export.imap.yahoo.com server is used?

I haven't checked, but it wouldn't surprise me if export.imap.yahoo.com didn't support UIDs at all, and just used old-style sequence numbers. Those are invalidated when the session ends, so the only thing a client can do is to re-download the whole lot :-( Not a TB-specific issue - with sequence numbers changing every time there's just no real sensible way to keep messages synced.

Comment 2

[nvikram.iphone]

Like Ben mentioned, UIDFETCH & VANISHED responses are returned to clients after UIDONLY is enabled. Just fyi, IMAP UIDONLY has been standardized as an experimental RFC: https://www.rfc-editor.org/rfc/rfc9586.html.

Just fyi, IMAP is a legacy protocol and sequence numbers are quite inefficient for servers with large mailboxes. Clients have some complexities around identifying expunged messages but not as much on server. With IMAP rev4, all clients are storing messages by UIDs and it becomes efficient if both server and client exchange info using UIDs. It would be nice if we can have this implemented on Thunderbird.