610264 - UID FETCH: Too many messages for request when accessing yahoo mail (Yahoo! IMAP limits number of mails in FETCH command for HEADER. Max=5000)

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, enhancement)

Description

[jisakiel]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; es-ES; rv:1.9.2.12) Gecko/20101030 Iceweasel/3.6.12 (like Firefox/3.6.12)
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.9) Gecko/20100918 Icedove/3.1.4

Although yahoo imap mail support is not official it has worked for a while with SSL and imap.mail.yahoo.com server. However, since a couple of weeks ago, it stopped working on the inbox with this error message: 

"UID FETCH: Too many messages for request"

Evolution when configured with IMAPx instead of imap is able to fetch my inbox, as mentioned in https://bugs.edge.launchpad.net/ubuntu/+source/thunderbird/+bug/662400 . 

I have quite a large inbox, around 1 GB. However everything I have tested works so far when using s_client to check: 

* OK [CAPABILITY IMAP4rev1 ID NAMESPACE X-ID-ACLID UIDPLUS LITERAL+ AUTH=PLAIN AUTH=LOGIN AUTH=XYMCOOKIE AUTH=XYMECOOKIE AUTH=XYMCOOKIEB64 AUTH=XYMPKI] IMAP4rev1 imapgate-0.7.65_12.283249 imap401.mail.ukl.yahoo.com

* STATUS "Inbox" (MESSAGES 9339)
. OK STATUS completed
 Inbox
* 9339 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1281117869] UIDs valid
* OK [UIDNEXT 282711] Predicted next UID
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS ()] Permanent flags
. OK [READ-ONLY] EXAMINE completed; now in selected state


. fetch 1:500 rfc822.header

(a long stream of messages go by, without trouble). 


I believe osx's mail is also able to fetch inbox, it's just a thunderbird issue.

Reproducible: Always

Steps to Reproduce:
1. Configure yahoo account as imap with SSL and server imap.mail.yahoo.com
2. (optionally) uncheck "keep messages locally"
3. try to check mail
Actual Results:  
UID FETCH: Too many messages for request


Using icedove under debian squeeze, but I can replicate the problem on windows stock thunderbird.

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> "UID FETCH: Too many messages for request"

Probably response to "UID FETCH 1:* (FLAGS)" was recently changed by Yahoo.
As "* 9339 EXISTS", it turns out request for 9339 mails.

Can you attach IMAP log of Tb?
> https://wiki.mozilla.org/MailNews:Logging
> Win sample : SET NSPR_LOG_MODULES=timestamp,imap:5
Replace sensitive data in log before open to public, please.

Protcol of IMAP doesn't limit "number of messages per command". However, granuality of "bulk fetch", "bulk copy", "bulk move", "bulk delete" etc. may be needed for such server.

Comment 2

[jisakiel]

Attachment: requested imap logs

Server appears as imap.yahoo.es because that is the first name I tried ages ago when trying to make this work, before yahoo had opened it. Server name in the preferences is imap.mail.yahoo.com. 

Apparently it is already sanitized by thunderbird: "Logging suppressed for this command (it probably contained authentication information)". I removed folder names and UIDs to shorten it a bit; if required I can provide the precise UID fetch line, although I guess it might be a matter of tinkering with those ranges until it works?


If you could point me as to how to do it, I can provide the same logs by Evolution as well, to compare to the commands sent with IMAP+.

Comment 3

[jisakiel]

Although it doesn't appear in my trimmed attachment apparently the UID fetch is
not precise, as there are far more holes in my UIDs than they appear in the
command. 

For instance: 

* 7893 FETCH (FLAGS (\Seen) UID 98378)

* 7894 FETCH (FLAGS (\Seen) UID 98387)

But the fetch command makes a huge range as the last one: 

7 UID fetch  1:924,925(...)3398:282768 

which by itself is rejected by the mail server. 

I tinkered manually with the full message list and have discovered that yahoo
accepts a maximum of 5000 effective messages at the time, in total. For
instance, if I ask for . uid fetch 4368:282768 (4999 messages in total) it
works, but if I ask for 4367 (which exists) it fails. It fails as well with .
uid fetch 1:2,4368:282768 . 


Just in case it helps.

Comment 4

[David :Bienvenu]

imap servers are supposed to ignore missing UIDs - i.e., if there are are uid's 1, 5 and 10 in a mailbox,the client is supposed to ask for 1:10, not 1,5,10, so that should be fine. This is the only imap server I've heard of with a limit to the number of actual headers you can fetch. We limit ourselves as to the length of the command we send, but not the number of headers we fetch.

Comment 6

[WADA:World Anti-bad-Duping Agency]

I created a folder contains more than 5000 mails by;
(0) Put a mail in Folder-1
(1) At Folder-1, Select All, Copy to Folder-2
(2) At Folder-2, Select All, Copy to Folder-1
(3) Repeat step (1) & step (2) multiple times, till number of mails>5000.
    Fibonacci number. F18=2584, F19=4181. So 10 repeats is sufficient.
    http://en.wikipedia.org/wiki/Fibonacci_number
While step (3), no problem occurs, because number of new mails in folder from previous FETCH is always less than 5000 as F19=4181, and any mail is viewed with no problem.
(4) At Folder-2(number of mails>5000), Folder Properties, Repair Folders.
    Problem is reproduced, and nothing is shown at thread pane.
      UID fetch 1:* (FLAGS)
      => OK UID FETCH completed (no problem, if FETCH FLAGS only)
      UID fetch a:b,...y:z (... BODY.PEEK[HEADER.FIELDS ...
      => NO [UNAVAILABLE] UID FETCH Too many messages in request. 
    No limit if BODY.PEEK[HEADER]?

> 00000028 21.66012192	[1904] 2516[45c8780]: 46e9800:imap.mail.yahoo.com:S-Inbox-Test/Inbox-02:SendData: 12 UID fetch 1:* (FLAGS)
> 00000029 22.14470291	[1904] 2516[45c8780]: ReadNextLine [stream=76550c8 nb=35 needmore=0]
> 00000030 22.14483452	[1904] 2516[45c8780]: 46e9800:imap.mail.yahoo.com:S-Inbox-Test/Inbox-02:CreateNewLineFromSocket: * 1 FETCH (FLAGS (\Seen) UID 200)
>(snip, total mail number is more than 5000)
> 00011386 23.34727478	[1904] 2516[45c8780]: 46e9800:imap.mail.yahoo.com:S-Inbox-Test/Inbox-02:CreateNewLineFromSocket: * 5679 FETCH (FLAGS (\Seen) UID 9304)
> 00011387 23.34732437	[1904] 2516[45c8780]: ReadNextLine [stream=76550c8 nb=27 needmore=0]
> 00011388 23.34736824	[1904] 2516[45c8780]: 46e9800:imap.mail.yahoo.com:S-Inbox-Test/Inbox-02:CreateNewLineFromSocket: 12 OK UID FETCH completed
> 00011389 24.97305107	[1904] 2516[45c8780]: 46e9800:imap.mail.yahoo.com:S-Inbox-Test/Inbox-02:SendData: 13 UID fetch 200:282,366:500,719:1424,2349:3625,5827:9304 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)])
> 00011390 24.97310066	[1904] 2516[45c8780]: ReadNextLine [stream=76550c8 nb=60 needmore=0]
> 00011391 24.97318649	[1904] 2516[45c8780]: 46e9800:imap.mail.yahoo.com:S-Inbox-Test/Inbox-02:CreateNewLineFromSocket: 13 NO [UNAVAILABLE] UID FETCH Too many messages in request

I guess one of next.
(A) By bug of some mobile devices, many mails was requested by single FETCH command. "Max 5000 mails per single FETCH command" is protection from mobile device down due to too many header data from server at a time, which is caused by such bug.
(B) Yahoo! IMAP is for mobile user, and mobile phone doesn't request many mails by a single FETCH command even when first access after setup of mobile phone, because resource of mobile phone is limited. Recently, users started to use Yahoo! IMAP server from PC using free mailer(Tb, ...). Because mailer of PC like Tb requests header of all mails upon initial access of IMAP folder(like Repair Folder), server's performance issue happened.

Comment 7

[WADA:World Anti-bad-Duping Agency]

FYI.
Current IMAP server of Yahoo! Mail is  "piloting for mobile". See Bug 493064 comment #35, please.

Comment 8

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 9

[jisakiel]

I just rechecked on a recent Thunderbird 60 under Mojave, and it is now able to load over IMAP my (larger than then) inbox without any issue. Feel free to resolve.

Comment 10

[Wayne Mery (:wsmwk)]

Thanks for the test