imap folders grows because a message constantly redownloads - need to remember at the protocol parsing level when the server sends us an invalid message

UNCONFIRMED
Unassigned

Status

MailNews Core
Networking: IMAP
UNCONFIRMED
7 years ago
4 months ago

People

(Reporter: mail, Unassigned)

Tracking

({testcase})

All
Other
testcase

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(1 attachment)

(Reporter)

Description

7 years ago
Created attachment 545335 [details]
Message file for message causing described behaviour.

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:6.0) Gecko/20100101 Firefox/6.0
Build ID: 20110705195857

Steps to reproduce:

I set my IMAP mail account to synchronize all messages locally.


Actual results:

I ran out of disk space and discovered that the file "C:\Documents and Settings\username\Application Data\Thunderbird\Profiles\zlsnxtyp.default\ImapMail\imap.1and1.co.uk\Sent-1" was over 15GB.

After discussion on the mozillazine support forum (http://forums.mozillazine.org/viewtopic.php?f=39&t=2248757&e=0) I discovered that when any folder in my IMAP mail account contains a particular message the OS file representing the local synchronization of that folder grows intermittently by the size of the message (554K), even when the folder only contains that message and when there is no user activity on the folder.


Expected results:

The size of the file should have remained at the size of the message.
(Reporter)

Comment 1

7 years ago
To reproduce - create a new folder in an IMAP mail account. Ensure the folder is set to synchronise locally. Drop the attached file on the folder. At apparently random intervals the OS file containing the local copy of the folder will grow in increments of 554K.
Component: General → Networking: IMAP
Keywords: testcase
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
Attachment #545335 - Attachment mime type: application/octet-stream → message/rfc822

Updated

7 years ago
Attachment #545335 - Attachment mime type: message/rfc822 → text/plain

Comment 2

7 years ago
Does the message actually come back with the first two lines:

-090206040500040500090307--
@duncanpearson.net>


that's not a valid rfc822 message, which makes us think there was an error downloading it.
(Reporter)

Comment 3

7 years ago
Those are the first two lines of the .EML file created when I saved the message.

A similar message from the same source began with the line:

Message-ID: <4D7A9CF8.6090602@duncanpearson.net>

This message was in my "Sent" folder as a result of me auto-forwarding to someone else.

Comment 4

7 years ago
(In reply to comment #3)

> 
> A similar message from the same source began with the line:
> 
> Message-ID: <4D7A9CF8.6090602@duncanpearson.net>
> 
> This message was in my "Sent" folder as a result of me auto-forwarding to
> someone else.

That's actually OK - it begins with an RFC822 header, so it's a valid message.
(Reporter)

Comment 5

7 years ago
What I meant was that the corrupt message had been forwarded by me and was in my "Sent" folder as a result. Sorry for the misunderstanding.

Comment 6

7 years ago
Oh, so the copy in your sent folder is correct, but corrupt in the IMAP inbox. Is it possible to rebuild the IMAP Inbox on the server?

TB should handle this better, obviously, but we were trying to detect corruption in our local cache by making sure the message is a valid rfc822 message from the server. I guess we need to somehow remember when the server sends us an invalid message at the protocol parsing level.
(Reporter)

Comment 7

7 years ago
So is what you are saying that as it's corrupt TB is periodically re-fetching it and that when it is re-saved locally TB always allocates more space rather than reusing existing space?

Comment 8

7 years ago
(In reply to comment #7)
> So is what you are saying that as it's corrupt TB is periodically
> re-fetching it and that when it is re-saved locally TB always allocates more
> space rather than reusing existing space?

yes, though you can do a file | compact folders to reclaim the space.

Comment 9

4 years ago
Irving, do you recall if we've fixed such a beast in recent years?

bugs related to message size dominate list of fixes in past three years https://bugzilla.mozilla.org/buglist.cgi?f1=short_desc&list_id=11048606&o1=nowordssubstr&resolution=FIXED&classification=Client%20Software&classification=Components&chfieldto=2014-08-24&chfield=resolution&query_format=advanced&chfieldfrom=2011-07-21&f2=OP&chfieldvalue=fixed&longdesc=rfc822%20invalid%20corrupt%20malformed&component=Networking%3A%20IMAP&longdesc_type=anywordssubstr&product=MailNews%20Core&product=Thunderbird
Flags: needinfo?(irving)
See Also: → bug 662979
Summary: When a folder in an IMAP mail account contains a particular message the OS file representing the local synchronization of that folder grows intermittently by the size of the message → imap folders grows because a message constantly redownloads - need to remember at the protocol parsing level when the server sends us an invalid message
If anything I fixed affected this behaviour, it was by accident; I haven't intentionally changed the way we handle messages received from the server that fail message format checks.

It would be relatively easy to write an IMAP and/or POP fakeserver test for this.
Flags: needinfo?(irving)
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.
You need to log in before you can comment on or make changes to this bug.