IMAP Fetch-Query too long - long FETCH or STORE command may cause mail server disconnect (Too long argument)

NEW
Unassigned

Status

MailNews Core
Networking: IMAP
--
major
14 years ago
2 years ago

People

(Reporter: Boris Folgmann, Unassigned)

Tracking

({perf})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [bulkoperations])

Attachments

(1 attachment)

(Reporter)

Description

14 years ago
User-Agent:       Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.7.3) Gecko/20040922 MultiZilla/1.6.4.0b Mnenhy/0.6.0.104
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.7.3) Gecko/20040922 MultiZilla/1.6.4.0b Mnenhy/0.6.0.104

It's possible to easily construct FETCH or STORE queries with mozilla that crash
IMAP servers.

The same problem, reported by a different user can be found here:
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=23846

The reason here:
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=23849

Cite:
"Yes, it happens when you send a query that's to long. I.e. a very large
msgs_id string in combination with FETCH or STORE.
The imap-client you are using should split up those imap-queries and use
smaller message sets."


Reproducible: Always
Steps to Reproduce:
1. Select all Mails (Ctrl-A) in a large folder, e.g. "Sent".
2. Mark all Mails at once as "Not Junk" using the menu.

Actual Results:  
Mozilla displays error window, that the connection to the IMAP server was lost.

I could find the reason in my imapd.log:
Oct 11 10:06:24 gaia imapd[23161]: open: user boris opened Sent
Oct 11 10:06:53 gaia imapd[23161]: Fatal error: word too long
Oct 11 10:06:53 gaia master[1395]: process 23161 exited, status 75


Expected Results:  
According to Marc Groot Koerkamp (see links above) the query has to be splitted.
I don't know if that is an requirement of the IMAP RFC or a cyrus-specific issue.
Please fix it anyway.

I use cyrus-imapd 2.1.16 and mozilla 1.7.3.

Comment 1

14 years ago
It doesn't seem to be very clear in the RFC what the maximum length is, but for
compatibility reason, this should be fixed.

Can you log what Mozilla sends exactly, and make sure what is the maximum viable
length ? Have a look at 
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap to see how
to activate IMAP log in mozilla.
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Reporter)

Comment 2

14 years ago
Created attachment 161762 [details]
Gzipped IMAP log file from mozilla

I removed "private" parts from the start and end of the log file.
Search for "too long" to find the error message.
(Reporter)

Comment 3

14 years ago
Looking at the sources it seems to me that 8K is the right size to go. At least
cyrus and dovecot IMAPD use both 8192 bytes.

Comment 4

14 years ago
After more verifcation, I see we should probably close as a duplicate of bug
91269 (yes, it's an old problem ...).
Can you do it if you don't see a reason against that ?

It would be good to raise up the issue, but bugzilla is not the right place for
advocacy ...

Comment 5

14 years ago
Did you try a newer build? I fixed some instances of this problem in 1.8a builds
(in particular some of the Store commands, and some fetch commands).  bug
91269 is more about the COPY command, which I don't believe I've fixed, though
it's on my list of things to do. There is a method that knows how to deal with
the uid ranges to shorten the lines, but I haven't applied it to the COPY command.
Product: MailNews → Core

Comment 6

11 years ago
Currently using Thunderbird "version 2.0.0.0 (20070326)"

I am having the same problem where tbird sends a command that is too long and my mail server disconnects me.

1736[672f9b0]: 254a218:imap.[private].com:S-INBOX:SendData: 13 UID fetch 311261:311262,311268,311325,311336,311345,311351
,311356,311398,311417,311441,311457,311527,311538,311572,311591,311596,311762,311778,311785,311843,312062,312168,312172,
312307:312308,312793,312807,312812,312816,312819,313054,313414,313536,313645,313647,313723,313957,314106,314673,314922,3
15016,315441,315550,315819,315840,316062,316471,316484,316490,316509,316585,316597,316708,316734,31
1736[672f9b0]: ReadNextLine [stream=24a84d0 nb=79 needmore=0]
1736[672f9b0]: 254a218:imap.[private].com:S-INBOX:CreateNewLineFromSocket: * BYE [ALERT] Fatal error: max atom size too s
mall: No such file or directory
1736[672f9b0]: 254a218:imap.[private].com:NA:TellThreadToDie: close socket connection
1736[672f9b0]: ImapThreadMainLoop leaving [this=254a218]

As reported elsewhere, this is a message returned when the command being sent is too large for a buffer.  Is there a workaround presently?

Comment 7

11 years ago
I cannot presently build, and I'm still trying to figure out how to contribute properly.

Looking in nsImapProtocol.cpp, under nsImapProtocol::Store, I find this line:

PRUint32 msgsToHandle = msgCountLeft;

It also appears in ::Fetch and ::Copy.  All three methods take a list of messages, divide them into an array of message ids, then step through a loop until all the messages have been processed with the command.  The size of the batch of messages to process is msgsToHandle.

Unfortunately, this line makes it appear as though it always executes exactly one batch of messages, by setting it on the first run to be equivalent to the number remaining (all of them).

Perhaps what we meant was something like,
msgsToHandle = 50; //arbitrary batch size

if(msgsToHandle > msgCountLeft) msgsToHandle = msgCountLeft; //within the loop

There's some other strange interaction in here too: each of the methods takes an idsAreUid which seems to affect how the messageList string would be parsed.  But msgKeys (the array into which it is parsed) looks like it would never be filled, ever, if !idsAreUid.

Also, there's an nsCString messageIDList declared in each message that appears to be unused?

I'm going to try and get my own build working and see if I can fix this.  What confuses me most is that logs seem to indicate that there IS some splitting into batches for things, but that perhaps it's being done before the store/fetch methods are called?
(Reporter)

Comment 8

11 years ago
I tried my "Not Junk" test case on a folder with more than 9000 mails, and it worked perfectly. In the meanwhile I use Seamonkey 1.0.8 and cyrus-imapd-2.2.12-3.RHEL4.1.

Comment 9

10 years ago
Brian, any news?

Also, what is exact error message that displays in status/UI?
Keywords: perf
Summary: IMAP Fetch-Query too long → IMAP Fetch-Query too long - long FETCH or STORE command may cause mail server disconnect

Updated

10 years ago
OS: Linux → All
QA Contact: grylchan → networking.imap
Hardware: PC → All
(Assignee)

Updated

9 years ago
Product: Core → MailNews Core
(In reply to comment #7)

> I'm going to try and get my own build working and see if I can fix this.  What
> confuses me most is that logs seem to indicate that there IS some splitting
> into batches for things, but that perhaps it's being done before the
> store/fetch methods are called?

Any luck in building ?

Comment 11

9 years ago
ludo, Brian's address bounces. so we need someone to run with the draft patch

bienvenu, question from comment #7
> What confuses me most is that logs seem to indicate that there IS some splitting
> into batches for things, but that perhaps it's being done before the
> store/fetch methods are called?

(note, bug 91269 mentioned in comment 5 is fixed, per bienvenu)
Whiteboard: [patchlove][has draft patch][needs owner][batch operation]
(In reply to comment #11)
> ludo, Brian's address bounces. so we need someone to run with the draft patch

shall we try http://psinet.cjb.net/ ?

Comment 13

8 years ago
(In reply to comment #12)
> (In reply to comment #11)
> > ludo, Brian's address bounces. so we need someone to run with the draft patch
> 
> shall we try http://psinet.cjb.net/ ?

ludo, any feedback?
(In reply to comment #13)
> (In reply to comment #12)
> > (In reply to comment #11)
> > > ludo, Brian's address bounces. so we need someone to run with the draft patch
> > 
> > shall we try http://psinet.cjb.net/ ?
> 
> ludo, any feedback?

No :-(

Comment 15

7 years ago
Nikolay not taking?  :(
Whiteboard: [patchlove][has draft patch][needs owner][batch operation] → [bulkoperations]

Updated

6 years ago
Assignee: dbienvenu → nobody

Updated

4 years ago
Duplicate of this bug: 239951

Comment 17

4 years ago
I just duped bug 239951 and in bug 239951 comment 5 I attempt a history about command length and state that I found no support forum reports and find nothing in bug reports, except this bug 263821.  

Comment 6 above claims there was still a problem in version 2. 

Comment 7 is noteworthy - it's not clear the code around http://hg.mozilla.org/comm-central/annotate/12e6710c1ed6/mailnews/imap/src/nsImapProtocol.cpp#l5135 entirely does the job.  ref also http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/mailnews/imap/src/nsImapProtocol.cpp&rev=1.688&mark=4781#4781

I also wonder whether bienvenu's comment 5 has been fixed. (I didn't dig)
Severity: normal → major
Summary: IMAP Fetch-Query too long - long FETCH or STORE command may cause mail server disconnect → IMAP Fetch-Query too long - long FETCH or STORE command may cause mail server disconnect (Too long argument)
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.
You need to log in before you can comment on or make changes to this bug.