IMAP Fetch-Query too long - long FETCH or STORE command may cause mail server disconnect (Too long argument)
Categories
(MailNews Core :: Networking: IMAP, defect)
Tracking
(Not tracked)
People
(Reporter: boris, Unassigned)
References
()
Details
(Keywords: perf, Whiteboard: [bulkoperations])
Attachments
(1 file)
34.58 KB,
application/octet-stream
|
Details |
User-Agent: Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.7.3) Gecko/20040922 MultiZilla/1.6.4.0b Mnenhy/0.6.0.104 Build Identifier: Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.7.3) Gecko/20040922 MultiZilla/1.6.4.0b Mnenhy/0.6.0.104 It's possible to easily construct FETCH or STORE queries with mozilla that crash IMAP servers. The same problem, reported by a different user can be found here: http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=23846 The reason here: http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=23849 Cite: "Yes, it happens when you send a query that's to long. I.e. a very large msgs_id string in combination with FETCH or STORE. The imap-client you are using should split up those imap-queries and use smaller message sets." Reproducible: Always Steps to Reproduce: 1. Select all Mails (Ctrl-A) in a large folder, e.g. "Sent". 2. Mark all Mails at once as "Not Junk" using the menu. Actual Results: Mozilla displays error window, that the connection to the IMAP server was lost. I could find the reason in my imapd.log: Oct 11 10:06:24 gaia imapd[23161]: open: user boris opened Sent Oct 11 10:06:53 gaia imapd[23161]: Fatal error: word too long Oct 11 10:06:53 gaia master[1395]: process 23161 exited, status 75 Expected Results: According to Marc Groot Koerkamp (see links above) the query has to be splitted. I don't know if that is an requirement of the IMAP RFC or a cyrus-specific issue. Please fix it anyway. I use cyrus-imapd 2.1.16 and mozilla 1.7.3.
Comment 1•20 years ago
|
||
It doesn't seem to be very clear in the RFC what the maximum length is, but for compatibility reason, this should be fixed. Can you log what Mozilla sends exactly, and make sure what is the maximum viable length ? Have a look at http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap to see how to activate IMAP log in mozilla.
Reporter | ||
Comment 2•20 years ago
|
||
I removed "private" parts from the start and end of the log file. Search for "too long" to find the error message.
Reporter | ||
Comment 3•20 years ago
|
||
Looking at the sources it seems to me that 8K is the right size to go. At least cyrus and dovecot IMAPD use both 8192 bytes.
Comment 4•20 years ago
|
||
After more verifcation, I see we should probably close as a duplicate of bug 91269 (yes, it's an old problem ...). Can you do it if you don't see a reason against that ? It would be good to raise up the issue, but bugzilla is not the right place for advocacy ...
Comment 5•20 years ago
|
||
Did you try a newer build? I fixed some instances of this problem in 1.8a builds (in particular some of the Store commands, and some fetch commands). bug 91269 is more about the COPY command, which I don't believe I've fixed, though it's on my list of things to do. There is a method that knows how to deal with the uid ranges to shorten the lines, but I haven't applied it to the COPY command.
Updated•20 years ago
|
Comment 6•17 years ago
|
||
Currently using Thunderbird "version 2.0.0.0 (20070326)" I am having the same problem where tbird sends a command that is too long and my mail server disconnects me. 1736[672f9b0]: 254a218:imap.[private].com:S-INBOX:SendData: 13 UID fetch 311261:311262,311268,311325,311336,311345,311351 ,311356,311398,311417,311441,311457,311527,311538,311572,311591,311596,311762,311778,311785,311843,312062,312168,312172, 312307:312308,312793,312807,312812,312816,312819,313054,313414,313536,313645,313647,313723,313957,314106,314673,314922,3 15016,315441,315550,315819,315840,316062,316471,316484,316490,316509,316585,316597,316708,316734,31 1736[672f9b0]: ReadNextLine [stream=24a84d0 nb=79 needmore=0] 1736[672f9b0]: 254a218:imap.[private].com:S-INBOX:CreateNewLineFromSocket: * BYE [ALERT] Fatal error: max atom size too s mall: No such file or directory 1736[672f9b0]: 254a218:imap.[private].com:NA:TellThreadToDie: close socket connection 1736[672f9b0]: ImapThreadMainLoop leaving [this=254a218] As reported elsewhere, this is a message returned when the command being sent is too large for a buffer. Is there a workaround presently?
Comment 7•17 years ago
|
||
I cannot presently build, and I'm still trying to figure out how to contribute properly. Looking in nsImapProtocol.cpp, under nsImapProtocol::Store, I find this line: PRUint32 msgsToHandle = msgCountLeft; It also appears in ::Fetch and ::Copy. All three methods take a list of messages, divide them into an array of message ids, then step through a loop until all the messages have been processed with the command. The size of the batch of messages to process is msgsToHandle. Unfortunately, this line makes it appear as though it always executes exactly one batch of messages, by setting it on the first run to be equivalent to the number remaining (all of them). Perhaps what we meant was something like, msgsToHandle = 50; //arbitrary batch size if(msgsToHandle > msgCountLeft) msgsToHandle = msgCountLeft; //within the loop There's some other strange interaction in here too: each of the methods takes an idsAreUid which seems to affect how the messageList string would be parsed. But msgKeys (the array into which it is parsed) looks like it would never be filled, ever, if !idsAreUid. Also, there's an nsCString messageIDList declared in each message that appears to be unused? I'm going to try and get my own build working and see if I can fix this. What confuses me most is that logs seem to indicate that there IS some splitting into batches for things, but that perhaps it's being done before the store/fetch methods are called?
Reporter | ||
Comment 8•17 years ago
|
||
I tried my "Not Junk" test case on a folder with more than 9000 mails, and it worked perfectly. In the meanwhile I use Seamonkey 1.0.8 and cyrus-imapd-2.2.12-3.RHEL4.1.
Comment 9•16 years ago
|
||
Brian, any news? Also, what is exact error message that displays in status/UI?
Updated•16 years ago
|
Assignee | ||
Updated•15 years ago
|
Comment 10•15 years ago
|
||
(In reply to comment #7) > I'm going to try and get my own build working and see if I can fix this. What > confuses me most is that logs seem to indicate that there IS some splitting > into batches for things, but that perhaps it's being done before the > store/fetch methods are called? Any luck in building ?
Comment 11•15 years ago
|
||
ludo, Brian's address bounces. so we need someone to run with the draft patch bienvenu, question from comment #7 > What confuses me most is that logs seem to indicate that there IS some splitting > into batches for things, but that perhaps it's being done before the > store/fetch methods are called? (note, bug 91269 mentioned in comment 5 is fixed, per bienvenu)
Comment 12•15 years ago
|
||
(In reply to comment #11) > ludo, Brian's address bounces. so we need someone to run with the draft patch shall we try http://psinet.cjb.net/ ?
Comment 13•14 years ago
|
||
(In reply to comment #12) > (In reply to comment #11) > > ludo, Brian's address bounces. so we need someone to run with the draft patch > > shall we try http://psinet.cjb.net/ ? ludo, any feedback?
Comment 14•14 years ago
|
||
(In reply to comment #13) > (In reply to comment #12) > > (In reply to comment #11) > > > ludo, Brian's address bounces. so we need someone to run with the draft patch > > > > shall we try http://psinet.cjb.net/ ? > > ludo, any feedback? No :-(
Comment 15•13 years ago
|
||
Nikolay not taking? :(
Updated•12 years ago
|
Comment 17•10 years ago
|
||
I just duped bug 239951 and in bug 239951 comment 5 I attempt a history about command length and state that I found no support forum reports and find nothing in bug reports, except this bug 263821. Comment 6 above claims there was still a problem in version 2. Comment 7 is noteworthy - it's not clear the code around http://hg.mozilla.org/comm-central/annotate/12e6710c1ed6/mailnews/imap/src/nsImapProtocol.cpp#l5135 entirely does the job. ref also http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/mailnews/imap/src/nsImapProtocol.cpp&rev=1.688&mark=4781#4781 I also wonder whether bienvenu's comment 5 has been fixed. (I didn't dig)
Comment 18•9 years ago
|
||
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.
Comment 19•5 years ago
|
||
When setting flags on many messages in a single store command, I have observed tb using "range" syntax like "1:1000" when UIDs are consecutive, which they often are. So this problem should be rare and seems more like a server problem to me.
Comment 20•2 years ago
|
||
Per RFC 7162 Section 4 "Long Command Lines"
While [RFC3501](https://datatracker.ietf.org/doc/html/rfc3501) doesn't specify a specific line-length limit, several
server implementations chose to implement the recommended line-length
limit suggested in [Section 3.2.1.5 of [RFC2683]](https://datatracker.ietf.org/doc/html/rfc2683#section-3.2.1.5) in order to protect
from Denial-of-Service attacks. When the line-length limit is
exceeded, such servers return a BAD response (as required by
[RFC3501](https://datatracker.ietf.org/doc/html/rfc3501) in case of a syntactic error) and may even close the
connection. Clients that support CONDSTORE/QRESYNC extensions can
trigger this limit by sending a long UID sequence (previously
returned by the server) in an extended SELECT or FETCH command.
This document updates recommended line-length limits specified in
[Section 3.2.1.5 of [RFC2683]](https://datatracker.ietf.org/doc/html/rfc2683#section-3.2.1.5). While the advice in the first
paragraph of that section still applies (use compact message/UID set
representations), the 1000-octet limit suggested in the second
paragraph turns out to be quite problematic when the CONDSTORE and/or
QRESYNC extension is used.
The updated recommendation is as follows: a client should limit the
length of the command lines it generates to approximately 8192 octets
(including all quoted strings but not including literals). If the
client is unable to group things into ranges so that the command line
is within that length, it should split the request into multiple
commands. The client should use literals instead of long quoted
strings in order to keep the command length down.
Therefore, Thunderbird's transmission of unlimited length commands
is a violation of the IMAP RFCs and does negatively impact the end
user experience. This is especially true if the IMAP server or SNORT
responds to the "too long command" by closing the TCP connection.
Thunderbird's response to a failed connection is to retry the command
indefinitely.
I've recently experienced a problem when attempting to move a year's worth of messages from an Inbox to an Archive folder. The resulting OnlineMove command is nearly 21K octets which caused the IMAP server to terminate the TCP connection. Thunderbird sees this as a network failure and queues the operation to be retried the next time the IMAP server is successfully reached. This results in the failing command being retried forever.
The assumption that message ids will be consecutive is mistaken. Therefore, the predicted use of limited command message size is not realized. Here is a command that is the result of moving all messages belonging to one calendar year from an Inbox to an archive folder. The resulting message is nearly 21K octets which far exceeds the 8192-octet size guidance of the IMAP RFCs.
[Parent 15976: IMAP]: D/IMAP URL failed with code 0x80470002 (imap://user%40example%2Ecom@imap.example.com:143/onlinemove%3EUID%3EINBOX%3E19413,19422,19424,19427:19428,19441,19443,19446,19448,19450:19453,19457,19485:19487,19490:19491,19496,19498:19499,19508,19511,19517:19519,19521:19522,19532,19535:19537,19546:19547,19550,19553,19562:19566,19568,19570,19572,19576,19578:19579,19585,19587,19595,19599:19600,19606:19608,19611:19612,19616,19618:19619,19623,19629,19631:19632,19635:19637,19640,19648,19653:19654,19660,19665:19666,19675:19676,19678:19679,19683,19685,19688:19689,19691,19694:19695,19697,19700,19707,19709,19713,19723:19724,19728:19729,19736,19738:19739,19741,19743:19744,19749:19751,19753:19757,19759:19760,19762:19766,19771,19783,19785,19791:19792,19795:19797,19799,19804:19806,19808:19809,19816:19817,19822:19827,19830,19832:19833,19839,19842,19845,19848,19850:19851,19853,19862,19867,19873:19874,19884,19901,19909,19912,19914:19915,19917:19918,19921,19925:19929,19936,19939,19943,19965:19967,19969:19970,19973:19976,19987:19990,19994,19996:19997,19999,20001:20005,20007:20008,20013:20014,20016:20017,20023,20025:20026,20032,20037,20040,20044,20047,20051,20058,20062,20079,20081,20087,20091:20092,20094,20096,20098:20100,20102:20109,20111,20115,20123:20124,20126:20128,20130:20133,20137,20141,20147:20148,20151:20154,20157,20161,20165:20166,20168:20171,20184,20191,20194,20198:20199,20202:20204,20214:20216,20223:20225,20227,20232,20234,20236,20238,20241:20246,20249:20250,20252,20255,20260,20266:20267,20271,20273,20289,20295:20298,20300,20305,20308:20311,20316,20322:20324,20328,20339,20352,20354:20355,20359,20362:20363,20365,20368:20369,20381:20384,20386,20389:20390,20392,20394,20396:20397,20399,20403,20410,20412,20421,20424,20427,20429,20432,20437,20440,20442,20447:20448,20450,20455:20456,20465:20466,20472,20474:20478,20480,20482:20484,20488,20492,20497,20502,20504,20506,20510:20512,20514,20516,20519:20521,20523:20527,20530:20532,20535,20538,20542,20544,20548,20550,20552,20557:20559,20563,20566,20568,20577:20578,20580,20582:20583,20590:20594,20608,20610:20611,20613,20634,20636,20638:20639,20643,20649,20652,20655,20659,20662,20673,20677:20680,20685,20693:20695,20700,20704:20705,20717,20719,20722:20724,20726,20731:20733,20736,20739:20740,20752:20753,20760,20762:20765,20768,20773:20776,20778,20783,20786:20788,20790:20791,20805,20807:20808,20811:20812,20816:20818,20820,20822,20826,20835:20837,20839,20846:20847,20851,20855:20856,20860,20868,20873,20877,20883,20891:20893,20900,20904:20905,20912:20913,20916,20920:20921,20926,20929:20930,20933,20936,20938:20949,20951:20955,20961,20965,20968,20978,20994:21001,21007,21014,21018,21021:21022,21029,21038:21039,21041,21043:21044,21048,21051,21056:21059,21062:21063,21068,21070:21071,21073:21075,21078:21079,21084,21086,21093:21094,21098,21100,21103:21105,21110,21113:21116,21119:21120,21126,21128:21129,21131,21133:21138,21140,21146,21151,21155,21157,21159,21164,21169:21171,21174:21177,21179:21180,21184,21186,21189:21190,21194:21195,21197,21202:21203,21205:21206,21209,21211:21213,21215:21216,21220:21223,21225:21226,21233,21235:21236,21238:21239,21241,21244,21246,21252,21254,21256:21257,21260:21261,21264,21267,21269,21273,21276,21279:21284,21286:21287,21290,21292,21296,21299,21303:21304,21306:21307,21311:21313,21316:21318,21325:21326,21328,21330:21331,21335,21338,21341,21343,21347,21350,21356:21357,21362,21364,21371,21373,21377:21378,21381:21385,21389:21391,21395:21397,21400,21402,21404,21422,21426,21428,21432,21435,21443,21445:21446,21449,21458,21460:21461,21465:21468,21471,21476,21478,21480:21481,21488,21493,21497,21505,21507,21515,21518,21520:21521,21524:21525,21527,21531:21532,21534,21539:21541,21543,21545:21549,21556,21559:21560,21562,21565:21570,21572,21574,21577,21579,21581,21587,21594,21602:21605,21609:21611,21617:21618,21620:21621,21624,21626,21628,21630:21633,21636,21646,21649:21653,21655,21658,21663,21665,21668,21672,21686,21688,21694:21695,21698,21700:21701,21703,21705:21707,21710,21714,21719:21720,21735:21736,21739,21745:21748,21756,21760:21763,21765:21767,21769:21771,21776:21779,21784,21786:21787,21792,21795,21801,21803:21804,21809,21816,21821,21824,21827,21844,21847,21850:21851,21853,21857,21860,21872:21875,21877,21884,21888:21889,21891:21892,21894,21897:21898,21900,21905,21909,21913,21916,21920,21928,21935:21938,21947,21956:21961,21965,21973,21980,21988,21993:22000,22002,22009:22014,22017,22023,22026:22027,22029,22031,22038:22039,22045,22049:22057,22060,22063,22065:22066,22075:22076,22080:22082,22085,22089:22090,22093,22096:22097,22099,22103,22114,22118,22123,22129,22141,22144,22148:22149,22153,22163,22174:22176,22179:22182,22187,22189,22192,22195:22201,22205,22212,22219,22223:22228,22233:22235,22241,22245:22246,22249:22250,22256:22259,22261,22268,22271,22276,22278:22279,22283:22284,22288:22289,22291:22292,22295,22298:22299,22301,22303:22304,22308:22309,22311:22321,22324,22327,22330:22331,22337,22341,22348:22357,22363:22364,22366:22368,22375,22385,22387:22390,22392:22393,22396,22401:22406,22411:22412,22416,22422,22427,22431,22436:22438,22441,22450:22452,22455:22458,22460:22461,22466,22471,22476,22484,22487:22488,22492:22495,22497,22502:22505,22507,22510,22518:22519,22522,22525,22529:22530,22533:22534,22537,22540:22542,22544,22546,22548,22550,22552:22556,22558,22572,22575,22578,22582,22584,22587:22588,22596:22597,22605:22608,22611,22614:22616,22619,22621,22623:22624,22628:22630,22632:22634,22636,22638:22639,22643:22646,22648:22649,22653:22654,22656,22659,22661,22664,22668,22673:22674,22681,22688:22689,22693,22695,22697,22700:22701,22711,22715,22717,22719,22721,22723:22726,22728,22730:22733,22735:22736,22744,22746:22747,22754:22756,22760:22761,22766,22771,22775,22782:22783,22792,22800:22801,22804,22811,22816:22819,22821:22824,22826:22828,22834,22840:22841,22847,22851,22854:22855,22858,22860,22862:22863,22866,22868,22872,22874,22876,22880,22882,22885,22887,22894:22895,22897:22899,22901:22914,22918:22919,22921:22922,22924:22927,22940:22942,22944,22946,22951:22952,22958:22959,22965:22966,22973:22974,22976,22979,22996,22998,23001,23004,23008:23009,23011,23017,23022,23034:23035,23041,23044,23047:23051,23058,23064,23072:23073,23077:23082,23085:23088,23090:23097,23099,23101,23103,23106,23112:23113,23117:23118,23122:23124,23127,23131,23133,23135,23141,23146:23147,23149:23151,23155:23156,23158,23166,23173,23177,23179:23181,23184:23186,23188:23190,23203,23208,23212:23213,23219,23223,23228,23231,23233,23242,23251,23255,23261,23264,23266,23272,23284:23285,23288,23291:23292,23296:23297,23299,23306,23312,23314,23317,23320:23321,23324:23325,23328:23329,23331,23334,23337:23338,23346:23347,23356:23357,23363,23366,23372,23379,23382:23383,23391,23396,23399,23404:23406,23408,23422,23425:23428,23432:23433,23435,23437,23443,23447:23448,23452:23454,23459,23461,23463,23465,23470:23473,23477:23480,23486,23489,23491:23493,23495:23496,23499,23501:23503,23505,23507:23508,23511:23512,23514:23516,23521:23524,23526,23528,23537,23552:23554,23561,23564,23566,23569,23572:23573,23576,23578:23580,23582:23584,23598,23602:23603,23605,23608:23610,23617:23618,23625,23636,23638,23646,23648:23649,23677,23682,23685:23687,23691:23692,23701,23704:23706,23711:23712,23714,23719:23722,23724,23727,23729,23731,23733,23737:23738,23740,23745:23746,23750:23751,23759,23761:23762,23769:23770,23780,23789,23794,23801,23805:23806,23808:23810,23815,23820:23822,23826,23828:23829,23834,23838:23839,23841:23844,23846:23848,23851,23855,23857:23858,23864,23866,23874:23875,23877,23884,23890:23892,23895,23899:23901,23906:23908,23912:23913,23915,23917,23919:23920,23923:23927,23931,23944,23946,23949,23952:23953,23957,23960,23963:23966,23972:23973,23977,23980:23981,23983:23984,23986:23987,23993,23996,23998,24005,24009:24010,24013,24017,24025,24027:24029,24032,24034:24035,24039,24041:24043,24047:24048,24052,24056,24064,24070:24071,24073,24078:24081,24083:24084,24086,24093,24100:24101,24105:24107,24110:24112,24116,24119,24121:24122,24125,24127:24128,24132,24137:24142,24148,24150,24153:24157,24159:24160,24162:24163,24165:24167,24170:24173,24179:24180,24182,24186:24189,24194:24195,24197:24199,24203:24204,24208,24210,24214:24218,24221,24224:24225,24227,24233,24247,24249,24251:24253,24261,24263,24266:24267,24271,24273,24275,24277,24283,24286:24287,24294,24299:24300,24309:24310,24317,24323,24325,24333,24335:24336,24339,24342,24349,24351:24354,24356:24357,24363:24364,24368,24375,24378,24380,24384:24385,24389,24391,24396,24405,24410,24415:24419,24437,24443:24445,24447:24448,24452,24454,24459,24464,24468:24469,24478,24485:24486,24488,24496,24498,24503:24504,24506:24507,24521,24528:24529,24533:24535,24537:24538,24542,24545,24552,24557,24559,24564,24566,24568,24577,24580:24581,24585,24590,24592,24595,24598,24600,24604,24606,24608,24612,24619:24622,24624,24626:24627,24631,24634,24636,24638:24642,24646,24649:24650,24654:24655,24662,24664,24671:24675,24679,24684,24686:24687,24696,24705:24706,24709,24722,24724,24731:24737,24741,24744,24747,24752,24758:24759,24761,24764:24765,24767,24777,24779:24780,24787,24791,24793,24797:24800,24807,24821,24831:24832,24838:24839,24843,24850,24852,24855:24856,24858,24863,24867,24873,24884,24896,24899:24901,24903,24916,24927:24928,24934,24938:24939,24944,24950,24952,24956,24958:24959,24962:24963,24966:24967,24970,24972,24977,24979:24980,24986,24991,24994:24997,25002,25004,25007,25009:25010,25012,25022,25027:25028,25031,25040,25049,25051,25055:25056,25075,25078:25079,25083,25087,25093:25096,25098,25102,25104,25107,25112,25115,25121,25135,25137:25139,25147:25149,25155,25159:25161
Can this please be fixed?
Comment 21•2 years ago
|
||
References:
- https://support.mozilla.org/en-US/questions/1363379#answer-1471913
- https://datatracker.ietf.org/doc/html/rfc7162#section-4
- https://datatracker.ietf.org/doc/html/rfc3501
- "bulkoperations" bugs - most of which may not be relevant however ... there is another bug report which I believe is relevant which I had recently updated but which (sadly) I can't recall just now, where I thought WADA had commented, but perhaps it was you Gene.
Comment 22•2 years ago
|
||
there is another bug report which I believe is relevant
Don't know what that would be either.
Can this please be fixed
I think it should be fixed. I'll take a look at it.
Were you able to work around this by copying the messages in smaller chunks?
Thunderbird's response to a failed connection is to retry the command indefinitely.
I would think TB would eventually report some type of error rather than just re-sending the command ad-infinitum?
I'm curious as to what type of imap server was involved that dropped the connection and didn't report a "BAD" imap response to the long command. Also, were you copying within the same server or to a different server?
The original client line length recommendation of 1000 doesn't mention disconnecting or denial of service, just that the server should report BAD if the clients sends more than 8000.
The updated recommendation is to help with CONDSTORE/QRESYNC and does mention denial or server and that the server might disconnect if command is too long or still send a BAD. It increases the client length to 8192 but doesn't recommend a larger server length (8000 per the original recommendation). Just an observation.
Anyhow, it seems TB never even implemented the original recommendation as it should have. I'll take a look at limiting to 8192 since I'm also working on adding QRESYNC as we speak. But the 8192 limit should be a patch independent of QRESYNC.
Comment 23•2 years ago
|
||
Hi Gene,
(In reply to gene smith from comment #22)
Can this please be fixed
I think it should be fixed. I'll take a look at it.
thank you.
Were you able to work around this by copying the messages in smaller chunks?
No. From the perspective of the user the messages were already locally moved to the new folder. Thunderbird is repeatedly trying to update the server state. Even if I switch to disconnected mode. Or restart Thunderbird which I did several times to enable and disable the IMAP protocol logging. The log rapidly grew in size to 800MB in a few minutes due to the length of the URLs and the number of retries.
Thunderbird's response to a failed connection is to retry the command indefinitely.
I would think TB would eventually report some type of error rather than just re-sending the command ad-infinitum?
Thunderbird was still attempting to update the server after two days. The problem was resolved when the server vendor provided a private build that accepted the largest of the URLs that Thunderbird was sending.
I'm curious as to what type of imap server was involved that dropped the connection and didn't report a "BAD" imap response to the long command. Also, were you copying within the same server or to a different server?
The IMAP server is MDaemon 2.5.1. The developers have already agreed to alter the behavior to both grow the acceptable message size above 8K and to return a BAD response when that increased limit is reached. However, I will point out that RFC7162 Section 4 paragraph one includes the following: "When the line-length limit is exceeded, such servers return a BAD response (as required by [RFC3501] in case of a syntactic error) and may even close the connection." In this case, the imap server only closed the connection.
I will point out that the a closed connection might also be the result of a SNORT rule for overly long IMAP requests. Obviously SNORT can only close the connection if TLS is not in use.
The move is between two folders on the same server. From "Inbox" to "Archive/2019/Inbox".
The original client line length recommendation of 1000 doesn't mention disconnecting or denial of service, just that the server should report BAD if the clients sends more than 8000.
The updated recommendation is to help with CONDSTORE/QRESYNC and does mention denial or server and that the server might disconnect if command is too long or still send a BAD. It increases the client length to 8192 but doesn't recommend a larger server length (8000 per the original recommendation). Just an observation.
RFC7162 Section 4 updates RFC 2683 Section 3.1.25 independent of the implementation of CONDSTORE/QRESYNC.
RFC2683 provides implementation guidance for the IMAP protocol but the actual protocol specification does not include any SHOULD, SHOULD NOT or MUST or MUST NOT language for command line length for either the client or the server. This is true even in the latest revision RFC9051. This leads me to believe that clients are permitted to send arbitrary length commands but can use BAD responses as an indication that it should retry with a smaller command line length. However, I have not found any servers that support larger command lines that 16K.
I believe the original recommendation of 8000 was intended to be sufficiently larger enough to permit a client to be a bit sloppy with its estimates. Hence Microsoft exchange defaulting to 10K and permitting (via configuration) messages up to 16K. MDaemon has an 8K limit at present but that will increased soon.
Anyhow, it seems TB never even implemented the original recommendation as it should have. I'll take a look at limiting to 8192 since I'm also working on adding QRESYNC as we speak. But the 8192 limit should be a patch independent of QRESYNC.
I agree that the 8192 limit or retrying with a smaller command line length should be independent of QRESYNC. I can imagine a command execution model that accepts an list of commands and if the command line is larger than 8000 octets when a BAD or unexpected disconnect occurs, the remaining commands in the list are split into smaller chunks. Then executed either serially or in parallel.
Thanks for your assistance. Please let me know if there is anything that I can do to help either with testing or reviewing code changes.
Jeffrey Altman
Comment 24•2 years ago
|
||
(In reply to Jeffrey Altman from comment #23)
The IMAP server is MDaemon 2.5.1. The developers have already agreed to alter the behavior to both grow the acceptable message size above 8K and to return a BAD response when that increased limit is reached.
MDaemon 2.5.2 has shipped with an input buffer that supports commands up to 20K. It also issues BAD responses when the command is too long.
After the issuance of the BAD response, Thunderbird stops sending data to the IMAP server until the connection times out.
Any progress on an implementation of command length restrictions in Thunderbird's IMAP client?
Comment 25•2 years ago
|
||
I did a relatively small move test and looked at the code again and it appears that there is something to limit the string of UIDs length to 950 bytes.. However, this limit may not occur if CONDSTORE is enabled since the flagState may be seen as "partial".
What you show in comment 20 are the UIDs as they appear in the "onlinemove" internal URL. When tb actually sends the command with imap it uses messages "flag state" to determine if the "gaps" in the list contain actual messages that have not yet be expunged (truly deleted). If they have been expunged, "range" strings (A:B) are used over a wider length resulting in a much shorter string.
So my question is have you enabled CONDSTORE? That seems like it would be the only thing that would prevent breaking the move url into 950 byte imap uid moves.
Also, do know the state of the messages in the "gaps" in comment 20? Were they still present in the mailbox and just marked as deleted or have they been expunged?
Do you know how many actual imap uid move commands occurred when you did the move shown in comment 20?
Comment 26•2 years ago
|
||
Gene,
Thank you for your reply.
Unfortunately too much time has passed for me to be able to answer your questions with any certainty. The server logs have rotated and the server has been updated with the version of the software that both significantly increased the size of the input buffer (at least 21K) and returns BAD if the new limit is exceeded.
mailserver.default.use_condstore is true.
The messages that were being moved were many years old. They were being moved from an Inbox which would have had few messages saved compared to the number of messages deleted on any given day. I would expect the gap messages to have been expunged.
I do not know how many actual imap uid move commands were issued. One very large one exceeding 20K was issued and re-issued repeatedly until the server's input buffer was increased to 21K. The IMAP server was restarted multiple times to increase the input buffer to 12K, 16K and eventually to 21K.
I'm sorry I cannot provide more clarity. However, mailserver.default.use_condstore is enabled so that might be sufficient.
Comment 27•2 years ago
|
||
The next question might be "why was use_condstore enabled?" Probably because I was trying to work around some failure but the details are lost in time. I've been running Thunderbird and migrating profiles to newer machines for two decades.
Comment 28•2 years ago
|
||
At one time, right after condstore was implemented (way before my time, over a decade ago) it was enabled by default for all new accounts. But this caused some problems so it was defaulted to off. So maybe yours somehow got left on after many updates. Not sure if updates would have turned it off on existing accounts. Anyhow, my proposed condstore/qresync upgrade (bug 1747311) should still allow the 950 byte limit to be respected since flag state would no longer be seen as partial.
All condstore may do for you is allow quicker initial access to folders with many thousands of messages and/or a very slow network. If that's not an issue for you I would recommend switching it off. Also, switch it off and restart tb if you are doing a bulk copy or move to keep the command length relatively small (950); you can switch it back on after the copy is done.
Comment 29•2 years ago
|
||
The MDaemon server as of 2.5.2 does not support CONDSTORE or QRESYNC although the developers are looking at implementing both. As that is the my primary server I have turned off mailserver.default.use_condstore. I am happy to test when Bug 1747311 is available in a form that permits it.
Thank you for your assistance.
Updated•2 years ago
|
Description
•