670871 - imap folders grows because a message constantly redownloads - need to remember at the protocol parsing level when the server sends us an invalid message

Status: UNCONFIRMED

(MailNews Core :: Networking: IMAP, defect)

Description

[mail]

Attachment: Message file for message causing described behaviour.

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:6.0) Gecko/20100101 Firefox/6.0
Build ID: 20110705195857

Steps to reproduce:

I set my IMAP mail account to synchronize all messages locally.


Actual results:

I ran out of disk space and discovered that the file "C:\Documents and Settings\username\Application Data\Thunderbird\Profiles\zlsnxtyp.default\ImapMail\imap.1and1.co.uk\Sent-1" was over 15GB.

After discussion on the mozillazine support forum (http://forums.mozillazine.org/viewtopic.php?f=39&t=2248757&e=0) I discovered that when any folder in my IMAP mail account contains a particular message the OS file representing the local synchronization of that folder grows intermittently by the size of the message (554K), even when the folder only contains that message and when there is no user activity on the folder.


Expected results:

The size of the file should have remained at the size of the message.

Comment 1

[mail]

To reproduce - create a new folder in an IMAP mail account. Ensure the folder is set to synchronise locally. Drop the attached file on the folder. At apparently random intervals the OS file containing the local copy of the folder will grow in increments of 554K.

Comment 2

[David :Bienvenu]

Does the message actually come back with the first two lines:

-090206040500040500090307--
@duncanpearson.net>


that's not a valid rfc822 message, which makes us think there was an error downloading it.

Comment 3

[mail]

Those are the first two lines of the .EML file created when I saved the message.

A similar message from the same source began with the line:

Message-ID: <[e-mail]>

This message was in my "Sent" folder as a result of me auto-forwarding to someone else.

Comment 4

[David :Bienvenu]

(In reply to comment #3)

> 
> A similar message from the same source began with the line:
> 
> Message-ID: <[e-mail]>
> 
> This message was in my "Sent" folder as a result of me auto-forwarding to
> someone else.

That's actually OK - it begins with an RFC822 header, so it's a valid message.

Comment 5

[mail]

What I meant was that the corrupt message had been forwarded by me and was in my "Sent" folder as a result. Sorry for the misunderstanding.

Comment 6

[David :Bienvenu]

Oh, so the copy in your sent folder is correct, but corrupt in the IMAP inbox. Is it possible to rebuild the IMAP Inbox on the server?

TB should handle this better, obviously, but we were trying to detect corruption in our local cache by making sure the message is a valid rfc822 message from the server. I guess we need to somehow remember when the server sends us an invalid message at the protocol parsing level.

Comment 7

[mail]

So is what you are saying that as it's corrupt TB is periodically re-fetching it and that when it is re-saved locally TB always allocates more space rather than reusing existing space?

Comment 8

[David :Bienvenu]

(In reply to comment #7)
> So is what you are saying that as it's corrupt TB is periodically
> re-fetching it and that when it is re-saved locally TB always allocates more
> space rather than reusing existing space?

yes, though you can do a file | compact folders to reclaim the space.

Comment 9

[Wayne Mery (:wsmwk)]

Irving, do you recall if we've fixed such a beast in recent years?

bugs related to message size dominate list of fixes in past three years https://bugzilla.mozilla.org/buglist.cgi?f1=short_desc&list_id=11048606&o1=nowordssubstr&resolution=FIXED&classification=Client%20Software&classification=Components&chfieldto=2014-08-24&chfield=resolution&query_format=advanced&chfieldfrom=2011-07-21&f2=OP&chfieldvalue=fixed&longdesc=rfc822%20invalid%20corrupt%20malformed&component=Networking%3A%20IMAP&longdesc_type=anywordssubstr&product=MailNews%20Core&product=Thunderbird

Comment 10

[:Irving Reid (No longer working on Firefox)]

If anything I fixed affected this behaviour, it was by accident; I haven't intentionally changed the way we handle messages received from the server that fail message format checks.

It would be relatively easy to write an IMAP and/or POP fakeserver test for this.

Comment 11

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 12

[Wayne Mery (:wsmwk)]

Has testcase.
Do we have the test suggested in comment 10?

Comment 13

[gene smith]

We fixed a bug recently where bad header delimiters causes the file to grow and even compacting didn't help. But didn't add any test to verify the fix.

From comment 10:

It would be relatively easy to write an IMAP and/or POP fakeserver test for this.

At this point, not so easy. I have zero experience dealing with "fakeserver".

Comment 14

[Wayne Mery (:wsmwk)]

Do we have any reason to believe this issue still exists?