Open Bug 263821 Opened 20 years ago Updated 2 years ago

IMAP Fetch-Query too long - long FETCH or STORE command may cause mail server disconnect (Too long argument)

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Type:
defect

Tracking

(Not tracked)

People

(Reporter: boris, Unassigned)

References

(
URL
)

Details

(Keywords: perf, Whiteboard: [bulkoperations])

Attachments

(1 file)

Gzipped IMAP log file from mozilla
34.58 KB, application/octet-stream
Details 
User-Agent:       Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.7.3) Gecko/20040922 MultiZilla/1.6.4.0b Mnenhy/0.6.0.104
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.7.3) Gecko/20040922 MultiZilla/1.6.4.0b Mnenhy/0.6.0.104

It's possible to easily construct FETCH or STORE queries with mozilla that crash
IMAP servers.

The same problem, reported by a different user can be found here:
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=23846

The reason here:
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=23849

Cite:
"Yes, it happens when you send a query that's to long. I.e. a very large
msgs_id string in combination with FETCH or STORE.
The imap-client you are using should split up those imap-queries and use
smaller message sets."


Reproducible: Always
Steps to Reproduce:
1. Select all Mails (Ctrl-A) in a large folder, e.g. "Sent".
2. Mark all Mails at once as "Not Junk" using the menu.

Actual Results:  
Mozilla displays error window, that the connection to the IMAP server was lost.

I could find the reason in my imapd.log:
Oct 11 10:06:24 gaia imapd[23161]: open: user boris opened Sent
Oct 11 10:06:53 gaia imapd[23161]: Fatal error: word too long
Oct 11 10:06:53 gaia master[1395]: process 23161 exited, status 75


Expected Results:  
According to Marc Groot Koerkamp (see links above) the query has to be splitted.
I don't know if that is an requirement of the IMAP RFC or a cyrus-specific issue.
Please fix it anyway.

I use cyrus-imapd 2.1.16 and mozilla 1.7.3.
It doesn't seem to be very clear in the RFC what the maximum length is, but for
compatibility reason, this should be fixed.

Can you log what Mozilla sends exactly, and make sure what is the maximum viable
length ? Have a look at 
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap to see how
to activate IMAP log in mozilla.
Status: UNCONFIRMED → NEW
Ever confirmed: true
I removed "private" parts from the start and end of the log file.
Search for "too long" to find the error message.
Looking at the sources it seems to me that 8K is the right size to go. At least
cyrus and dovecot IMAPD use both 8192 bytes.


After more verifcation, I see we should probably close as a duplicate of bug
91269 (yes, it's an old problem ...).
Can you do it if you don't see a reason against that ?

It would be good to raise up the issue, but bugzilla is not the right place for
advocacy ...

Did you try a newer build? I fixed some instances of this problem in 1.8a builds
(in particular some of the Store commands, and some fetch commands).  bug
91269 is more about the COPY command, which I don't believe I've fixed, though
it's on my list of things to do. There is a method that knows how to deal with
the uid ranges to shorten the lines, but I haven't applied it to the COPY command.
Product: MailNews → Core
Currently using Thunderbird "version 2.0.0.0 (20070326)"

I am having the same problem where tbird sends a command that is too long and my mail server disconnects me.

1736[672f9b0]: 254a218:imap.[private].com:S-INBOX:SendData: 13 UID fetch 311261:311262,311268,311325,311336,311345,311351
,311356,311398,311417,311441,311457,311527,311538,311572,311591,311596,311762,311778,311785,311843,312062,312168,312172,
312307:312308,312793,312807,312812,312816,312819,313054,313414,313536,313645,313647,313723,313957,314106,314673,314922,3
15016,315441,315550,315819,315840,316062,316471,316484,316490,316509,316585,316597,316708,316734,31
1736[672f9b0]: ReadNextLine [stream=24a84d0 nb=79 needmore=0]
1736[672f9b0]: 254a218:imap.[private].com:S-INBOX:CreateNewLineFromSocket: * BYE [ALERT] Fatal error: max atom size too s
mall: No such file or directory
1736[672f9b0]: 254a218:imap.[private].com:NA:TellThreadToDie: close socket connection
1736[672f9b0]: ImapThreadMainLoop leaving [this=254a218]

As reported elsewhere, this is a message returned when the command being sent is too large for a buffer.  Is there a workaround presently?
I cannot presently build, and I'm still trying to figure out how to contribute properly.

Looking in nsImapProtocol.cpp, under nsImapProtocol::Store, I find this line:

PRUint32 msgsToHandle = msgCountLeft;

It also appears in ::Fetch and ::Copy.  All three methods take a list of messages, divide them into an array of message ids, then step through a loop until all the messages have been processed with the command.  The size of the batch of messages to process is msgsToHandle.

Unfortunately, this line makes it appear as though it always executes exactly one batch of messages, by setting it on the first run to be equivalent to the number remaining (all of them).

Perhaps what we meant was something like,
msgsToHandle = 50; //arbitrary batch size

if(msgsToHandle > msgCountLeft) msgsToHandle = msgCountLeft; //within the loop

There's some other strange interaction in here too: each of the methods takes an idsAreUid which seems to affect how the messageList string would be parsed.  But msgKeys (the array into which it is parsed) looks like it would never be filled, ever, if !idsAreUid.

Also, there's an nsCString messageIDList declared in each message that appears to be unused?

I'm going to try and get my own build working and see if I can fix this.  What confuses me most is that logs seem to indicate that there IS some splitting into batches for things, but that perhaps it's being done before the store/fetch methods are called?
I tried my "Not Junk" test case on a folder with more than 9000 mails, and it worked perfectly. In the meanwhile I use Seamonkey 1.0.8 and cyrus-imapd-2.2.12-3.RHEL4.1.

Brian, any news?

Also, what is exact error message that displays in status/UI?
Keywords: perf
Summary: IMAP Fetch-Query too long → IMAP Fetch-Query too long - long FETCH or STORE command may cause mail server disconnect
OS: Linux → All
QA Contact: grylchan → networking.imap
Hardware: PC → All
Product: Core → MailNews Core
(In reply to comment #7)

> I'm going to try and get my own build working and see if I can fix this.  What
> confuses me most is that logs seem to indicate that there IS some splitting
> into batches for things, but that perhaps it's being done before the
> store/fetch methods are called?

Any luck in building ?
ludo, Brian's address bounces. so we need someone to run with the draft patch

bienvenu, question from comment #7
> What confuses me most is that logs seem to indicate that there IS some splitting
> into batches for things, but that perhaps it's being done before the
> store/fetch methods are called?

(note, bug 91269 mentioned in comment 5 is fixed, per bienvenu)
Whiteboard: [patchlove][has draft patch][needs owner][batch operation]
(In reply to comment #11)
> ludo, Brian's address bounces. so we need someone to run with the draft patch

shall we try http://psinet.cjb.net/ ?
(In reply to comment #12)
> (In reply to comment #11)
> > ludo, Brian's address bounces. so we need someone to run with the draft patch
> 
> shall we try http://psinet.cjb.net/ ?

ludo, any feedback?
(In reply to comment #13)
> (In reply to comment #12)
> > (In reply to comment #11)
> > > ludo, Brian's address bounces. so we need someone to run with the draft patch
> > 
> > shall we try http://psinet.cjb.net/ ?
> 
> ludo, any feedback?

No :-(
Nikolay not taking?  :(
Whiteboard: [patchlove][has draft patch][needs owner][batch operation] → [bulkoperations]
Assignee: dbienvenu → nobody
I just duped bug 239951 and in bug 239951 comment 5 I attempt a history about command length and state that I found no support forum reports and find nothing in bug reports, except this bug 263821.  

Comment 6 above claims there was still a problem in version 2. 

Comment 7 is noteworthy - it's not clear the code around http://hg.mozilla.org/comm-central/annotate/12e6710c1ed6/mailnews/imap/src/nsImapProtocol.cpp#l5135 entirely does the job.  ref also http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/mailnews/imap/src/nsImapProtocol.cpp&rev=1.688&mark=4781#4781

I also wonder whether bienvenu's comment 5 has been fixed. (I didn't dig)
Severity: normal → major
Summary: IMAP Fetch-Query too long - long FETCH or STORE command may cause mail server disconnect → IMAP Fetch-Query too long - long FETCH or STORE command may cause mail server disconnect (Too long argument)
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

When setting flags on many messages in a single store command, I have observed tb using "range" syntax like "1:1000" when UIDs are consecutive, which they often are. So this problem should be rare and seems more like a server problem to me.

Severity: major → normal

Per RFC 7162 Section 4 "Long Command Lines"

   While [RFC3501](https://datatracker.ietf.org/doc/html/rfc3501) doesn't specify a specific line-length limit, several
   server implementations chose to implement the recommended line-length
   limit suggested in [Section 3.2.1.5 of [RFC2683]](https://datatracker.ietf.org/doc/html/rfc2683#section-3.2.1.5) in order to protect
   from Denial-of-Service attacks.  When the line-length limit is
   exceeded, such servers return a BAD response (as required by
   [RFC3501](https://datatracker.ietf.org/doc/html/rfc3501) in case of a syntactic error) and may even close the
   connection.  Clients that support CONDSTORE/QRESYNC extensions can
   trigger this limit by sending a long UID sequence (previously
   returned by the server) in an extended SELECT or FETCH command.

   This document updates recommended line-length limits specified in
   [Section 3.2.1.5 of [RFC2683]](https://datatracker.ietf.org/doc/html/rfc2683#section-3.2.1.5).  While the advice in the first
   paragraph of that section still applies (use compact message/UID set
   representations), the 1000-octet limit suggested in the second
   paragraph turns out to be quite problematic when the CONDSTORE and/or
   QRESYNC extension is used.

   The updated recommendation is as follows: a client should limit the
   length of the command lines it generates to approximately 8192 octets
   (including all quoted strings but not including literals).  If the
   client is unable to group things into ranges so that the command line
   is within that length, it should split the request into multiple
   commands.  The client should use literals instead of long quoted
   strings in order to keep the command length down.

Therefore, Thunderbird's transmission of unlimited length commands
is a violation of the IMAP RFCs and does negatively impact the end
user experience. This is especially true if the IMAP server or SNORT
responds to the "too long command" by closing the TCP connection.
Thunderbird's response to a failed connection is to retry the command
indefinitely.

I've recently experienced a problem when attempting to move a year's worth of messages from an Inbox to an Archive folder. The resulting OnlineMove command is nearly 21K octets which caused the IMAP server to terminate the TCP connection. Thunderbird sees this as a network failure and queues the operation to be retried the next time the IMAP server is successfully reached. This results in the failing command being retried forever.

The assumption that message ids will be consecutive is mistaken. Therefore, the predicted use of limited command message size is not realized. Here is a command that is the result of moving all messages belonging to one calendar year from an Inbox to an archive folder. The resulting message is nearly 21K octets which far exceeds the 8192-octet size guidance of the IMAP RFCs.

[Parent 15976: IMAP]: D/IMAP URL failed with code 0x80470002 (imap://user%40example%2Ecom@imap.example.com:143/onlinemove%3EUID%3EINBOX%3E19413,19422,19424,19427:19428,19441,19443,19446,19448,19450:19453,19457,19485:19487,19490:19491,19496,19498:19499,19508,19511,19517:19519,19521:19522,19532,19535:19537,19546:19547,19550,19553,19562:19566,19568,19570,19572,19576,19578:19579,19585,19587,19595,19599:19600,19606:19608,19611:19612,19616,19618:19619,19623,19629,19631:19632,19635:19637,19640,19648,19653:19654,19660,19665:19666,19675:19676,19678:19679,19683,19685,19688:19689,19691,19694:19695,19697,19700,19707,19709,19713,19723:19724,19728:19729,19736,19738:19739,19741,19743:19744,19749:19751,19753:19757,19759:19760,19762:19766,19771,19783,19785,19791:19792,19795:19797,19799,19804:19806,19808:19809,19816:19817,19822:19827,19830,19832:19833,19839,19842,19845,19848,19850:19851,19853,19862,19867,19873:19874,19884,19901,19909,19912,19914:19915,19917:19918,19921,19925:19929,19936,19939,19943,19965:19967,19969:19970,19973:19976,19987:19990,19994,19996:19997,19999,20001:20005,20007:20008,20013:20014,20016:20017,20023,20025:20026,20032,20037,20040,20044,20047,20051,20058,20062,20079,20081,20087,20091:20092,20094,20096,20098:20100,20102:20109,20111,20115,20123:20124,20126:20128,20130:20133,20137,20141,20147:20148,20151:20154,20157,20161,20165:20166,20168:20171,20184,20191,20194,20198:20199,20202:20204,20214:20216,20223:20225,20227,20232,20234,20236,20238,20241:20246,20249:20250,20252,20255,20260,20266:20267,20271,20273,20289,20295:20298,20300,20305,20308:20311,20316,20322:20324,20328,20339,20352,20354:20355,20359,20362:20363,20365,20368:20369,20381:20384,20386,20389:20390,20392,20394,20396:20397,20399,20403,20410,20412,20421,20424,20427,20429,20432,20437,20440,20442,20447:20448,20450,20455:20456,20465:20466,20472,20474:20478,20480,20482:20484,20488,20492,20497,20502,20504,20506,20510:20512,20514,20516,20519:20521,20523:20527,20530:20532,20535,20538,20542,20544,20548,20550,20552,20557:20559,20563,20566,20568,20577:20578,20580,20582:20583,20590:20594,20608,20610:20611,20613,20634,20636,20638:20639,20643,20649,20652,20655,20659,20662,20673,20677:20680,20685,20693:20695,20700,20704:20705,20717,20719,20722:20724,20726,20731:20733,20736,20739:20740,20752:20753,20760,20762:20765,20768,20773:20776,20778,20783,20786:20788,20790:20791,20805,20807:20808,20811:20812,20816:20818,20820,20822,20826,20835:20837,20839,20846:20847,20851,20855:20856,20860,20868,20873,20877,20883,20891:20893,20900,20904:20905,20912:20913,20916,20920:20921,20926,20929:20930,20933,20936,20938:20949,20951:20955,20961,20965,20968,20978,20994:21001,21007,21014,21018,21021:21022,21029,21038:21039,21041,21043:21044,21048,21051,21056:21059,21062:21063,21068,21070:21071,21073:21075,21078:21079,21084,21086,21093:21094,21098,21100,21103:21105,21110,21113:21116,21119:21120,21126,21128:21129,21131,21133:21138,21140,21146,21151,21155,21157,21159,21164,21169:21171,21174:21177,21179:21180,21184,21186,21189:21190,21194:21195,21197,21202:21203,21205:21206,21209,21211:21213,21215:21216,21220:21223,21225:21226,21233,21235:21236,21238:21239,21241,21244,21246,21252,21254,21256:21257,21260:21261,21264,21267,21269,21273,21276,21279:21284,21286:21287,21290,21292,21296,21299,21303:21304,21306:21307,21311:21313,21316:21318,21325:21326,21328,21330:21331,21335,21338,21341,21343,21347,21350,21356:21357,21362,21364,21371,21373,21377:21378,21381:21385,21389:21391,21395:21397,21400,21402,21404,21422,21426,21428,21432,21435,21443,21445:21446,21449,21458,21460:21461,21465:21468,21471,21476,21478,21480:21481,21488,21493,21497,21505,21507,21515,21518,21520:21521,21524:21525,21527,21531:21532,21534,21539:21541,21543,21545:21549,21556,21559:21560,21562,21565:21570,21572,21574,21577,21579,21581,21587,21594,21602:21605,21609:21611,21617:21618,21620:21621,21624,21626,21628,21630:21633,21636,21646,21649:21653,21655,21658,21663,21665,21668,21672,21686,21688,21694:21695,21698,21700:21701,21703,21705:21707,21710,21714,21719:21720,21735:21736,21739,21745:21748,21756,21760:21763,21765:21767,21769:21771,21776:21779,21784,21786:21787,21792,21795,21801,21803:21804,21809,21816,21821,21824,21827,21844,21847,21850:21851,21853,21857,21860,21872:21875,21877,21884,21888:21889,21891:21892,21894,21897:21898,21900,21905,21909,21913,21916,21920,21928,21935:21938,21947,21956:21961,21965,21973,21980,21988,21993:22000,22002,22009:22014,22017,22023,22026:22027,22029,22031,22038:22039,22045,22049:22057,22060,22063,22065:22066,22075:22076,22080:22082,22085,22089:22090,22093,22096:22097,22099,22103,22114,22118,22123,22129,22141,22144,22148:22149,22153,22163,22174:22176,22179:22182,22187,22189,22192,22195:22201,22205,22212,22219,22223:22228,22233:22235,22241,22245:22246,22249:22250,22256:22259,22261,22268,22271,22276,22278:22279,22283:22284,22288:22289,22291:22292,22295,22298:22299,22301,22303:22304,22308:22309,22311:22321,22324,22327,22330:22331,22337,22341,22348:22357,22363:22364,22366:22368,22375,22385,22387:22390,22392:22393,22396,22401:22406,22411:22412,22416,22422,22427,22431,22436:22438,22441,22450:22452,22455:22458,22460:22461,22466,22471,22476,22484,22487:22488,22492:22495,22497,22502:22505,22507,22510,22518:22519,22522,22525,22529:22530,22533:22534,22537,22540:22542,22544,22546,22548,22550,22552:22556,22558,22572,22575,22578,22582,22584,22587:22588,22596:22597,22605:22608,22611,22614:22616,22619,22621,22623:22624,22628:22630,22632:22634,22636,22638:22639,22643:22646,22648:22649,22653:22654,22656,22659,22661,22664,22668,22673:22674,22681,22688:22689,22693,22695,22697,22700:22701,22711,22715,22717,22719,22721,22723:22726,22728,22730:22733,22735:22736,22744,22746:22747,22754:22756,22760:22761,22766,22771,22775,22782:22783,22792,22800:22801,22804,22811,22816:22819,22821:22824,22826:22828,22834,22840:22841,22847,22851,22854:22855,22858,22860,22862:22863,22866,22868,22872,22874,22876,22880,22882,22885,22887,22894:22895,22897:22899,22901:22914,22918:22919,22921:22922,22924:22927,22940:22942,22944,22946,22951:22952,22958:22959,22965:22966,22973:22974,22976,22979,22996,22998,23001,23004,23008:23009,23011,23017,23022,23034:23035,23041,23044,23047:23051,23058,23064,23072:23073,23077:23082,23085:23088,23090:23097,23099,23101,23103,23106,23112:23113,23117:23118,23122:23124,23127,23131,23133,23135,23141,23146:23147,23149:23151,23155:23156,23158,23166,23173,23177,23179:23181,23184:23186,23188:23190,23203,23208,23212:23213,23219,23223,23228,23231,23233,23242,23251,23255,23261,23264,23266,23272,23284:23285,23288,23291:23292,23296:23297,23299,23306,23312,23314,23317,23320:23321,23324:23325,23328:23329,23331,23334,23337:23338,23346:23347,23356:23357,23363,23366,23372,23379,23382:23383,23391,23396,23399,23404:23406,23408,23422,23425:23428,23432:23433,23435,23437,23443,23447:23448,23452:23454,23459,23461,23463,23465,23470:23473,23477:23480,23486,23489,23491:23493,23495:23496,23499,23501:23503,23505,23507:23508,23511:23512,23514:23516,23521:23524,23526,23528,23537,23552:23554,23561,23564,23566,23569,23572:23573,23576,23578:23580,23582:23584,23598,23602:23603,23605,23608:23610,23617:23618,23625,23636,23638,23646,23648:23649,23677,23682,23685:23687,23691:23692,23701,23704:23706,23711:23712,23714,23719:23722,23724,23727,23729,23731,23733,23737:23738,23740,23745:23746,23750:23751,23759,23761:23762,23769:23770,23780,23789,23794,23801,23805:23806,23808:23810,23815,23820:23822,23826,23828:23829,23834,23838:23839,23841:23844,23846:23848,23851,23855,23857:23858,23864,23866,23874:23875,23877,23884,23890:23892,23895,23899:23901,23906:23908,23912:23913,23915,23917,23919:23920,23923:23927,23931,23944,23946,23949,23952:23953,23957,23960,23963:23966,23972:23973,23977,23980:23981,23983:23984,23986:23987,23993,23996,23998,24005,24009:24010,24013,24017,24025,24027:24029,24032,24034:24035,24039,24041:24043,24047:24048,24052,24056,24064,24070:24071,24073,24078:24081,24083:24084,24086,24093,24100:24101,24105:24107,24110:24112,24116,24119,24121:24122,24125,24127:24128,24132,24137:24142,24148,24150,24153:24157,24159:24160,24162:24163,24165:24167,24170:24173,24179:24180,24182,24186:24189,24194:24195,24197:24199,24203:24204,24208,24210,24214:24218,24221,24224:24225,24227,24233,24247,24249,24251:24253,24261,24263,24266:24267,24271,24273,24275,24277,24283,24286:24287,24294,24299:24300,24309:24310,24317,24323,24325,24333,24335:24336,24339,24342,24349,24351:24354,24356:24357,24363:24364,24368,24375,24378,24380,24384:24385,24389,24391,24396,24405,24410,24415:24419,24437,24443:24445,24447:24448,24452,24454,24459,24464,24468:24469,24478,24485:24486,24488,24496,24498,24503:24504,24506:24507,24521,24528:24529,24533:24535,24537:24538,24542,24545,24552,24557,24559,24564,24566,24568,24577,24580:24581,24585,24590,24592,24595,24598,24600,24604,24606,24608,24612,24619:24622,24624,24626:24627,24631,24634,24636,24638:24642,24646,24649:24650,24654:24655,24662,24664,24671:24675,24679,24684,24686:24687,24696,24705:24706,24709,24722,24724,24731:24737,24741,24744,24747,24752,24758:24759,24761,24764:24765,24767,24777,24779:24780,24787,24791,24793,24797:24800,24807,24821,24831:24832,24838:24839,24843,24850,24852,24855:24856,24858,24863,24867,24873,24884,24896,24899:24901,24903,24916,24927:24928,24934,24938:24939,24944,24950,24952,24956,24958:24959,24962:24963,24966:24967,24970,24972,24977,24979:24980,24986,24991,24994:24997,25002,25004,25007,25009:25010,25012,25022,25027:25028,25031,25040,25049,25051,25055:25056,25075,25078:25079,25083,25087,25093:25096,25098,25102,25104,25107,25112,25115,25121,25135,25137:25139,25147:25149,25155,25159:25161

Can this please be fixed?

References:

URL: https://support.mozilla.org/questions...
Flags: needinfo?(gds)

there is another bug report which I believe is relevant

Don't know what that would be either.

Can this please be fixed

I think it should be fixed. I'll take a look at it.

Were you able to work around this by copying the messages in smaller chunks?

Thunderbird's response to a failed connection is to retry the command indefinitely.

I would think TB would eventually report some type of error rather than just re-sending the command ad-infinitum?

I'm curious as to what type of imap server was involved that dropped the connection and didn't report a "BAD" imap response to the long command. Also, were you copying within the same server or to a different server?

The original client line length recommendation of 1000 doesn't mention disconnecting or denial of service, just that the server should report BAD if the clients sends more than 8000.

The updated recommendation is to help with CONDSTORE/QRESYNC and does mention denial or server and that the server might disconnect if command is too long or still send a BAD. It increases the client length to 8192 but doesn't recommend a larger server length (8000 per the original recommendation). Just an observation.

Anyhow, it seems TB never even implemented the original recommendation as it should have. I'll take a look at limiting to 8192 since I'm also working on adding QRESYNC as we speak. But the 8192 limit should be a patch independent of QRESYNC.

Flags: needinfo?(gds)

Hi Gene,

(In reply to gene smith from comment #22)

Can this please be fixed

I think it should be fixed. I'll take a look at it.

thank you.

Were you able to work around this by copying the messages in smaller chunks?

No. From the perspective of the user the messages were already locally moved to the new folder. Thunderbird is repeatedly trying to update the server state. Even if I switch to disconnected mode. Or restart Thunderbird which I did several times to enable and disable the IMAP protocol logging. The log rapidly grew in size to 800MB in a few minutes due to the length of the URLs and the number of retries.

Thunderbird's response to a failed connection is to retry the command indefinitely.

I would think TB would eventually report some type of error rather than just re-sending the command ad-infinitum?

Thunderbird was still attempting to update the server after two days. The problem was resolved when the server vendor provided a private build that accepted the largest of the URLs that Thunderbird was sending.

I'm curious as to what type of imap server was involved that dropped the connection and didn't report a "BAD" imap response to the long command. Also, were you copying within the same server or to a different server?

The IMAP server is MDaemon 2.5.1. The developers have already agreed to alter the behavior to both grow the acceptable message size above 8K and to return a BAD response when that increased limit is reached. However, I will point out that RFC7162 Section 4 paragraph one includes the following: "When the line-length limit is exceeded, such servers return a BAD response (as required by [RFC3501] in case of a syntactic error) and may even close the connection." In this case, the imap server only closed the connection.

I will point out that the a closed connection might also be the result of a SNORT rule for overly long IMAP requests. Obviously SNORT can only close the connection if TLS is not in use.

The move is between two folders on the same server. From "Inbox" to "Archive/2019/Inbox".

The original client line length recommendation of 1000 doesn't mention disconnecting or denial of service, just that the server should report BAD if the clients sends more than 8000.

The updated recommendation is to help with CONDSTORE/QRESYNC and does mention denial or server and that the server might disconnect if command is too long or still send a BAD. It increases the client length to 8192 but doesn't recommend a larger server length (8000 per the original recommendation). Just an observation.

RFC7162 Section 4 updates RFC 2683 Section 3.1.25 independent of the implementation of CONDSTORE/QRESYNC.

RFC2683 provides implementation guidance for the IMAP protocol but the actual protocol specification does not include any SHOULD, SHOULD NOT or MUST or MUST NOT language for command line length for either the client or the server. This is true even in the latest revision RFC9051. This leads me to believe that clients are permitted to send arbitrary length commands but can use BAD responses as an indication that it should retry with a smaller command line length. However, I have not found any servers that support larger command lines that 16K.

I believe the original recommendation of 8000 was intended to be sufficiently larger enough to permit a client to be a bit sloppy with its estimates. Hence Microsoft exchange defaulting to 10K and permitting (via configuration) messages up to 16K. MDaemon has an 8K limit at present but that will increased soon.

Anyhow, it seems TB never even implemented the original recommendation as it should have. I'll take a look at limiting to 8192 since I'm also working on adding QRESYNC as we speak. But the 8192 limit should be a patch independent of QRESYNC.

I agree that the 8192 limit or retrying with a smaller command line length should be independent of QRESYNC. I can imagine a command execution model that accepts an list of commands and if the command line is larger than 8000 octets when a BAD or unexpected disconnect occurs, the remaining commands in the list are split into smaller chunks. Then executed either serially or in parallel.

Thanks for your assistance. Please let me know if there is anything that I can do to help either with testing or reviewing code changes.

Jeffrey Altman

(In reply to Jeffrey Altman from comment #23)

The IMAP server is MDaemon 2.5.1. The developers have already agreed to alter the behavior to both grow the acceptable message size above 8K and to return a BAD response when that increased limit is reached.

MDaemon 2.5.2 has shipped with an input buffer that supports commands up to 20K. It also issues BAD responses when the command is too long.

After the issuance of the BAD response, Thunderbird stops sending data to the IMAP server until the connection times out.

Any progress on an implementation of command length restrictions in Thunderbird's IMAP client?

I did a relatively small move test and looked at the code again and it appears that there is something to limit the string of UIDs length to 950 bytes.. However, this limit may not occur if CONDSTORE is enabled since the flagState may be seen as "partial".
What you show in comment 20 are the UIDs as they appear in the "onlinemove" internal URL. When tb actually sends the command with imap it uses messages "flag state" to determine if the "gaps" in the list contain actual messages that have not yet be expunged (truly deleted). If they have been expunged, "range" strings (A:B) are used over a wider length resulting in a much shorter string.
So my question is have you enabled CONDSTORE? That seems like it would be the only thing that would prevent breaking the move url into 950 byte imap uid moves.
Also, do know the state of the messages in the "gaps" in comment 20? Were they still present in the mailbox and just marked as deleted or have they been expunged?
Do you know how many actual imap uid move commands occurred when you did the move shown in comment 20?

Gene,
Thank you for your reply.
Unfortunately too much time has passed for me to be able to answer your questions with any certainty. The server logs have rotated and the server has been updated with the version of the software that both significantly increased the size of the input buffer (at least 21K) and returns BAD if the new limit is exceeded.

mailserver.default.use_condstore is true.

The messages that were being moved were many years old. They were being moved from an Inbox which would have had few messages saved compared to the number of messages deleted on any given day. I would expect the gap messages to have been expunged.

I do not know how many actual imap uid move commands were issued. One very large one exceeding 20K was issued and re-issued repeatedly until the server's input buffer was increased to 21K. The IMAP server was restarted multiple times to increase the input buffer to 12K, 16K and eventually to 21K.

I'm sorry I cannot provide more clarity. However, mailserver.default.use_condstore is enabled so that might be sufficient.

The next question might be "why was use_condstore enabled?" Probably because I was trying to work around some failure but the details are lost in time. I've been running Thunderbird and migrating profiles to newer machines for two decades.

At one time, right after condstore was implemented (way before my time, over a decade ago) it was enabled by default for all new accounts. But this caused some problems so it was defaulted to off. So maybe yours somehow got left on after many updates. Not sure if updates would have turned it off on existing accounts. Anyhow, my proposed condstore/qresync upgrade (bug 1747311) should still allow the 950 byte limit to be respected since flag state would no longer be seen as partial.
All condstore may do for you is allow quicker initial access to folders with many thousands of messages and/or a very slow network. If that's not an issue for you I would recommend switching it off. Also, switch it off and restart tb if you are doing a bulk copy or move to keep the command length relatively small (950); you can switch it back on after the copy is done.

See Also: → 1747311

The MDaemon server as of 2.5.2 does not support CONDSTORE or QRESYNC although the developers are looking at implementing both. As that is the my primary server I have turned off mailserver.default.use_condstore. I am happy to test when Bug 1747311 is available in a form that permits it.

Thank you for your assistance.

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: