307615 - Significantly reduce memory allocations for IMAP mail download by improving |nsIMAPGenericParser::AdvanceToNextLine()|

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Hans-Andreas Engel]

Note that mail messages are read in line-by-line via calls to
|AdvanceToNextLine()|, see
http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/mailnews/imap/src/nsImapServerResponseParser.cpp&rev=1.123&root=/cvsroot#2706
Note that in |AdvanceToNextLine()|, a |strdup| of every line is made in order to
be able to tokenize the line later.  However, this copy is never needed, since
no tokenization takes place while downloading the mail message.

This means that every incoming mail line triggers an unnecessary memory
allocation (!), e.g. about 20,000 extra allocations for downloading a 1 MB
attachment.  This should be avoided.

Comment 1

[Hans-Andreas Engel]

Attachment: Delay initialization of tokenizer (strdup)

Delay initialization of tokenizer (strdup) until either |AdvanceToNextToken| or
|AdvanceTokenizerStartingPoint| is invoked.
(Patch for Bug 307605 should be applied first.)

Comment 2

[Hans-Andreas Engel]

Note that the suggested patch does not alter the behavior of
|nsIMAPGenericParser|.  Particular care is taken when empty lines are parsed
(the current behavior is somewhat non-intuitive and should probably be changed
in the future; however, this should be examined in a different bug report). 

Comment 3

[David :Bienvenu]

fix checked in, thanks!

Comment 4

[Hans-Andreas Engel]

On a one-year-old computer, while downloading a large attachment of an IMAP
message with Thunderbird (the 30MB mozilla source code is very useful for such
tests), the CPU usage is roughly 5-10% lower on todays build than it is on
1.5b2; probably due to this patch.

Comment 5

[David :Bienvenu]

Comment on attachment 195364 [details] [diff] [review]
Delay initialization of tokenizer (strdup)

marking reviews.