697635 - "From " line(unix mbox mail separator) in offline-store file of IMAP folder by "Repair Folder" is different from one by "Compact". X-Mozilla-... header shouldn't be written in offline-store file upon new mail download and "Repair Folder"

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[WADA:World Anti-bad-Duping Agency]

"From " line(unix mbox mail separator) in offline-store file of IMAP folder by "Repair Folder" is different from one by "Compact".
X-Mozilla-... header shouldn't be written in offline-store file by "Repair Folder".
X-Mozilla-Keys: header should be removed from offline-store file by "Compact".

This can't be called "bug of Tb", but it may produce unwanted problems, because mail start position after "From ", mail header part length, total mail data length, are changed by "Compact folder".
I guess some phenomena in bug reports for "corrupted mail data with IMAP" are relevant to this offset/length change in offline-store file by Compact. 

(A) Offline-store file after "Repair Folder"
From - Thu Oct 27 09:49:18 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys: $label1    
Message-ID: <[e-mail]>
Subject: test mail
From: a a <a.a@x.y.z>
To: b b <b.b@x.y.z>
Date: Sat, 01 Jan 2011 02:03:04 +0000
Content-Type: text/plain
MIME-Version: 1.0

test mail

(B) Offline-store file after "Compact"
From 
X-Mozilla-Keys: $label1    
Message-ID: <[e-mail]>
Subject: mailend
From: a a <a.a@x.y.z>
To: b b <b.b@x.y.z>
Date: Sat, 01 Jan 2011 02:03:04 +0000
Content-Type: text/plain
MIME-Version: 1.0

test mail

Comment 1

[David :Bienvenu]

When we compact offline stores, we update the offline message size here - http://mxr.mozilla.org/comm-central/source/mailnews/base/src/nsMsgFolderCompactor.cpp#981

And we account for the From header here:

http://mxr.mozilla.org/comm-central/source/mailnews/base/src/nsMsgFolderCompactor.cpp#1212

So I don't think this can be the cause of offline store corruption. We never "reparse" offline stores, and we sanity check each message in the offline store to make sure it starts with "From " before we use it.

Comment 2

[Ulf Zibis]

IMO, Compact should NOT remove the time-stamp after "From " because:
- received time info will be lost
- user could run in big problems on at least some mail servers:
-- Bug 667294
-- Bug 668482

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf.Zibis from comment #2)
> IMO, Compact should NOT remove the time-stamp after "From " because:
> - received time info will be lost

This "From " line or "From - timestmp" line is merely a separator of mails in Unix Mbox format file, and it can be seen only by viewing of Tb's offline-store file using text editor or some tools. And, timestamp in it is changed by "Repair Folder" of IMAP folder of offline-use=on. I can't understand why timestamp value in separator of Unix Mbox format is so important, although timestamp in separator of Unix Mbox format is convenient in problem analysis and I hope it's kept by Compact if timestamp is continuously written by Tb.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to David :Bienvenu from comment #1)
> When we compact offline stores, we update the offline message size here -
> And we account for the From header here:
> So I don't think this can be the cause of offline store corruption.

David :Bienvenu, is Tb torelant with "From ..." line in IMAP mail data?
Merge of "From " line(as Unix Mbox mail separator by Tb) and "From - ..." line(as mail data line) occurs by Compact of IMAP folder of offline-use=on.
It looks that original separator of "From " line is merely replaced by "From - timestamp" line in mail data, so critical mail data corruption doesn't seem to happen. However, at least inconsistent display by view/message source happens("From - ..." line as mail data is lost by Compact after Repair Folder).
(See bug 668482 comment #7)

Comment 5

[David :Bienvenu]

(In reply to WADA from comment #4)
> (In reply to David :Bienvenu from comment #1)
> > When we compact offline stores, we update the offline message size here -
> > And we account for the From header here:
> > So I don't think this can be the cause of offline store corruption.
> 
> David :Bienvenu, is Tb torelant with "From ..." line in IMAP mail data?
It doesn't care - there's no parsing of IMAP offline stores, and the From line is only used for (re)parsing local message folders, not imap offline stores.

I think the from date is only used if there's no actual date header in the message.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Problem when "merge of two From lines in offline-store by Compact" was "loss of From lines at top part in message source after Compact" only.
If multiple From lines are placed at top of uploaded mail, From lines at top part was reduced by each Compact until non From line appears as first line. However, mail end of the mail, mail start/end of following mails which is shown by View/Message Source were always correct and never affected by Compact.

Comment 7

[WADA:World Anti-bad-Duping Agency]

FYI.
X-Mozilla-Keys: $label1 in comment #0 was produced by "upload of mail from local mail folder to IMAP folder by Tb". This problem is processed by other bug.

Comment 8

[WADA:World Anti-bad-Duping Agency]

I've opeed separate bug 708941 for problem of comment #6.