668482 - emails in IMAP folder are corrupted after compact

Status: RESOLVED INVALID

(MailNews Core :: Networking: IMAP, defect)

Description

[Ulf Zibis]

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0
Build ID: 20110615151330

Steps to reproduce:

Thunderbird is online.
Copied emails from local folder to IMAP folder.
Compressed IMAP folder, marked for offline reading.


Actual results:

The
From - [timestamp]
lines in header were corrupted to:
From 
(with trailing blank)


Expected results:

From - [timestamp]
lines should have been conserved.

Comment 1

[Ulf Zibis]

A change of the account id could have been the source of the problem, see bug 667288.

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf.Zibis from comment #0)
> Actual results:
> The
> From - [timestamp]
> lines in header were corrupted to:
> From 
> (with trailing blank)
> Expected results:
> From - [timestamp]
> lines should have been conserved.

How did you see the "From - [timestamp]" line and the "From " line in message source?
By "View/Message Source" of Tb?
Or by viewing of offline-store file for the IMAP mail folder using text editor?
If latter, INVALID. You are merely looking phenomonen of bug 697635.

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf.Zibis from comment #1)
> A change of the account id could have been the source of the problem, (snip)

Are you looking phenomenon of bug 426651, aren't you?
See bugs listed in dependency tree for bug 69968. If X-Account-Key: header exists in message source, problem stated in meta bug 699681 can happen, and observed phenomenon can be affected by account number change.

Comment 4

[Ulf Zibis]

(In reply to WADA from comment #2)
> By "View/Message Source" of Tb?
> Or by viewing of offline-store file for the IMAP mail folder using text
> editor?

I see it by both.
See my comment on bug 697635.

Comment 5

[Ulf Zibis]

(In reply to WADA from comment #3)
> (In reply to Ulf.Zibis from comment #1)
> > A change of the account id could have been the source of the problem, (snip)
> 
> Are you looking phenomenon of bug 426651, aren't you?
Not really I now think, see my last comment.
But actually I have one occurrence of X-Account-Key: in my IMAP inbox of ~500 messages.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf.Zibis from comment #4)
> By "View/Message Source" of Tb?
> Or by viewing of offline-store file for the IMAP mail folder using text editor?

> I see it by both.

I still couldn't see "From - ..." nor "From " line by View/Message Source of Tb 8 for mail in folder of offline-use=on and folder of offline-use=off of Gmail IMAP account.
  (a) Mail is sent by Tb to Gmail, and arrived in Inbox of Gmail IMAP account
  (b) Mail is sent by Tb to POP3 account, and is downloaded to Inbox for the POP3
      account by Tb, and is copied to folder Gmail IMAP account by Tb.
I still could see "From - ..." or "From " line only by viewing of offline-store file for IMAP folder of offline-use=on using text editor, if mail in IMAP folder.

"From - ..." line at top of message source is shown by View/Message Source of mail held in local mail folder only, if Tb 8. 

Did you actually do View/Message Source on mail in IMAP folder of offline-use=on which is currently used by Tb? 
If offline-store file for IMAP account is copied under mail directory of POP3 account or Local Folders, separator of Unix Mbox file is shown at top of message source by View/Message Source of Tb. Are you talking about this case?
If foler pane view is Unified Folders, show account column to know mail in which account's folder, please.

Note: And, I still see phenomenon of bug 426651 on above mail (b) with Tb 8.

Comment 7

[WADA:World Anti-bad-Duping Agency]

I could see "From - Mon Dec 05 16:02:13 2011" line at top of message source by View/Message Source of mail in IMAP folder of offline-use=on at last.

(1) Copy local mail folder file which contains data of one mail only as .eml file.
.eml file content.
> From - Mon Dec 05 16:02:13 2011
> X-Account-Key: account902
> X-UIDL: 000004ab4c354af7
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
> X-Mozilla-Keys:
>(snip)

(2) Drag&Drop the .eml file to thread pane of Inbox of Gmail account(offline-use=on)
Content of offline-store file.
> From 
> From - Mon Dec 05 16:02:13 2011
> X-Account-Key: account902
> X-UIDL: 000004ab4c354af7
>(snip)
View/Message Source shows lines from "From - Mon Dec 05 ..." as message source.

(3) Compact of Inbox of Gmail account(offline-use=on)
Content of offline-store file.
> From - Mon Dec 05 16:02:13 2011
> X-Account-Key: account902
> X-UIDL: 000004ab4c354af7
>(snip)
View/Message Source shows lines from X-Account-Key:.

(4) Compact of Inbox of Gmail account(offline-use=on) again
Content of offline-store file.
> From 
> X-Account-Key: account902
> X-UIDL: 000004ab4c354af7
>(snip)
View/Message Source shows lines from X-Account-Key:.

(5) Repair Folder of Inbox of Gmail account(offline-use=on)
Content of offline-store file.
> From - Mon Dec 05 17:39:42 2011
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
> From - Mon Dec 05 16:02:13 2011
> X-Account-Key: account902
> X-UIDL: 000004ab4c354af7
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
> X-Mozilla-Keys:
"From - Mon Dec 05 16:02:13 2011" line is actually uploaded at step (2).
View/Message Source shows lines from "From - Mon Dec 05 ..." as message source.

(6) Compact of Inbox of Gmail account(offline-use=on)
Content of offline-store file.
> From - Mon Dec 05 16:02:13 2011
> X-Account-Key: account902
> X-UIDL: 000004ab4c354af7
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
> X-Mozilla-Keys:
Same as step (1).

Because IMAP offline-store file of Tb, there is no problem due to "no escaping of From line". "Rebuild-Index on local mail folder file" will never be invoked by Tb on offline-store file. Rebuil-Index related problem happens only when user used the offline-store file as local mail folder file(==used as ordinal Unix Mbox format mail data file).

Apparent problem in above is inconsistent display of message source by View/Message Source.
This is caused by merge of next two lines
> From 
> From - Mon Dec 05 16:02:13 2011
to next one line at step (3).
> From - Mon Dec 05 16:02:13 2011
Because "From - Mon Dec 05 16:02:13 2011" line is uploaded and held at IMAP server, this line is data of mail.

(In reply to Ulf.Zibis from comment #0)
> Copied emails from local folder to IMAP folder.

Is escaped "From ..." line like " From ..."(escape by space) or ">From ..."(escape by >) contained in mail in local folder?
If contained, the escaped line is probably uploaded to server after unescape by Tb.

Comment 8

[Ulf Zibis]

(In reply to WADA from comment #6)
> I still couldn't see "From - ..." nor "From " line by View/Message Source of
> Tb 8 for mail in folder of offline-use=on and folder of offline-use=off of
> Gmail IMAP account.
You are right, for TB 8. I only can see this on my old machine with TB 7.
It's not a kinda special gmail account, it's an account from german ISP 1&1 (imap.1und1.de). Additionally I can't say about INBOX. Some time ago I've copied the corrupted messages to an another IMAP folder 'From-Backup_Server' for later reference, but repaired the content of my INBOX manually to prevent from bug 667294.
If I select a message in 'From-Backup_Server' and then hit Ctrl-U, I can still see "From " in the 1st line at TB 7

> Did you actually do View/Message Source on mail in IMAP folder of
> offline-use=on which is currently used by Tb?
Yes, see above.

> If offline-store file for IMAP account is copied under mail directory of
> POP3 account or Local Folders, separator of Unix Mbox file is shown at top
> of message source by View/Message Source of Tb. Are you talking about this
> case?
No.

> If foler pane view is Unified Folders, show account column to know mail in
> which account's folder, please.
I do not use Unified Inbox folder.

> Note: And, I still see phenomenon of bug 426651 on above mail (b) with Tb 8.
IMO, this behaviour is helpful/correct to preserve the initial input account info.

Comment 9

[Ulf Zibis]

(In reply to WADA from comment #7)
> Is escaped "From ..." line like " From ..."(escape by space) or ">From
> ..."(escape by >) contained in mail in local folder?
> If contained, the escaped line is probably uploaded to server after unescape
> by Tb.

There is no escape here, but 1 space after From .

On TB 7 machine, the 'From-Backup_Server' file (22 MB, file modified on 06/30/2011) starts with:
From - Thu Jun 30 01:50:47 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
From 
Return-Path: <[e-mail]>
Delivery-Date: Mon, 23 Nov 2009 07:04:32 +0100
Received: from thailand.thailandreport.org (demiurgicsoftware.com [74.52.67.226])
	by mx.kundenserver.de (node=mxbap0) with ESMTP (Nemesis)
	id 0M0BXI-1O2MEN3QEO-00uWYZ for [e-mail]; Mon, 23 Nov 2009 07:04:32 +0100
Received: from gureck3 by thailand.thailandreport.org with local (Exim 4.69)
	(envelope-from <[e-mail]>)
	id 1NCS1v-00046s-SE
	for [e-mail]; Mon, 23 Nov 2009 07:04:07 +0100
To: [e-mail]
Subject: Thailandreport Newsletter
X-PHP-Script: www.thailandreport.org/forum/admincp/email.php for 188.60.203.5
From: "[e-mail]" <[e-mail]>
Message-ID: <[e-mail]>
MIME-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Priority: 3
X-Mailer: vBulletin Mail via PHP
Date: Mon, 23 Nov 2009 07:04:07 +0100


On TB 8 machine, the 'From-Backup_Server' file (7 MB, new download from server on 12/05/2011) starts with:
From 
Return-Path: <[e-mail]>
Delivery-Date: Thu, 25 Jun 2009 15:43:46 +0200
Received: from pmt.perfora.net (pmt2.perfora.net [212.227.126.172])
	by mx.kundenserver.de (node=mxeu5) with ESMTP (Nemesis)
	id 0MKqpg-1MJpEv48hN-00049x for [e-mail]; Thu, 25 Jun 2009 15:43:46 +0200
Received: from root by pmt.perfora.net with local (Exim 4.50)
	id 1MJpEu-00063u-Qe
	for [e-mail]; Thu, 25 Jun 2009 15:43:45 +0200
Subject: =?ISO-8859-15?Q?ABSAGE_wegen_Wasserschaden_I_KitKatClub_@_@_D=E9j=E0_Vu_Club_I_27.06.2009?=
Date: Thu, 25 Jun 2009 15:43:44 +0200
To: Me<[e-mail]>
From: [e-mail]
Reply-To: [e-mail]
X-Provags-ID: V01U2FsdGVkX1+buBjQilaXQt0qQOmK+QrxuBeSaIirzItYnqewmtvnJuv2fEWNU
 zPsPngG6n3lYECslxCr5r27jg==
MIME-Version: 1.0
Content-Type: multipart/related;
	type="multipart/alternative";
	boundary="5602750598B96AAD1C4433313B8340B80358.0000"

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf.Zibis from comment #9)
> On TB 7 machine, the 'From-Backup_Server' file (22 MB, file modified on
> 06/30/2011) starts with:
> From - Thu Jun 30 01:50:47 2011
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
> From 
> Return-Path: <[e-mail]>
>(snip)

Offline-store file content? Or View/Message Source display?

> On TB 8 machine, the 'From-Backup_Server' file (7 MB, new download from
> server on 12/05/2011) starts with:
> From 
> Return-Path: <[e-mail]>
>(snip)

Offline-store file content? Or View/Message Source display?

If View/Message Source display, it can't explain next by you in comment #8.
> You are right, for TB 8. I only can see this on my old machine with TB 7.
And you wrote "the From-Backup_Server file".
So I assume offline-store file content.

As I wrote in comment #7, offline-store file content is changed by Compact after first download(Repair Folder);
  (5) by Repair Folder -> (6)==(3) -> (4) as final
"Number of From - ... lines at topmost part of mail data shown by View/Message Source" is reduced by each Compact, and finaly lost, and they revive by next Repair Folder.

Tb7 machine data looks just after Repair Folder.
Tb8 machine data looks after Compact(s).
And, at Tb7 machine, it looks that next is phenomenon of bug 697635
> From - Thu Jun 30 01:50:47 2011
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
and followig data is shown by View/Message Source.
> From 
> Return-Path: <[e-mail]>
>(snip)

Is "From " line in TB7 before "Return Path:" sent from server(top line of mail data held at server)? Or written by Tb wrongly?
Can you check by next?

1. Create two IMAP folders : F1(offline-use=On), F2(offline-use=off)
   Copy the mail in the IMAP folder to F1 and F2. (single mail only in folder)
Because mail copy is done by "uid xx copy F1(or F2)", copy is done at server. However, Tb uses localy held data after copy in mail display. So Repair Folder is needed after mail copy by Tb.
2. F2 : Repair Folder, View/Message Source
        As offline-use=off, there is no offline-store file. 
3-1. F1 : Repair Folder, View/Message Source, check offline-store file content.
3-2. F1 : Compact, View/Message Source, check offline-store file content.
          Repeat this step several times.

Because Tb7's behavior and Tb8 behavior is same in test of comment #7, any of Tb7 and Tb8 can be used for above check, but I recommend you to check with newest Tb8.

Comment 11

[WADA:World Anti-bad-Duping Agency]

FYI. I've opened bug 708941 separately for phenomenon of comment #7.

Comment 12

[Ulf Zibis]

(In reply to WADA from comment #10)
> (In reply to Ulf.Zibis from comment #9)
> > On TB 8 machine, the 'From-Backup_Server' file (7 MB, new download from
> > server on 12/05/2011) starts with:
> >(snip)
> 
> Offline-store file content? Or View/Message Source display?
Offline-store! I said: "the 'xxx' file (size, modified date) starts with..." ;-)

> Tb7 machine data looks just after Repair Folder.
IIRC, data in 'From-Backup_Server' file on TB7 was created as follows:
1. Via web-mailer I moved all the corrupted messages from Inbox to folder From-Backup_Server
2. Started TB and synchronized folder.
3. I don't think, I used Repair Folder, but not for sure.
(It seems I did the above 3 times (after deleting), as the file is 3 times in size compared to TB 8 machine, so it was truly not compacted)

> Tb8 machine data looks after Compact(s).
This is really possible.

> Is "From " line in TB7 before "Return Path:" sent from server(top line of
> mail data held at server)? Or written by Tb wrongly?
I guess, it was written by TB before/while copying the messages to Inbox from local rescue folder (causing bug 667294 later). After, they were moved to folder From-Backup_Server on server, and then sent from server while syncing.

> Can you check by next?
> 
> 1. Create two IMAP folders : F1(offline-use=On), F2(offline-use=off)
>    Copy the mail in the IMAP folder to F1 and F2. (single mail only in
> folder)
Which mail should I copy, a correct one from Inbox or a corrupted from folder From-Backup_Server ?
Should I copy by TB or via web-mailer ?

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf.Zibis from comment #12)
> > Offline-store file content? Or View/Message Source display?
> Offline-store! I said: "the 'xxx' file (size, modified date) starts with..."
> ;-)
>(snip)
> Which mail should I copy, a correct one from Inbox or a corrupted from
> folder From-Backup_Server ?
> Should I copy by TB or via web-mailer ?

We are talking about data you provided in comment #9.
  Mail you provided data at comment #9 held in folder named 'From-Backup_Server'
You can use any of Tb and Web-mailer to copy.
If by web-mailer, first download of the mail by Tb==download by Repair Folder.
In any copy, Repair Folder is merely to force discarding of already downloaded mail data and to force re-downloaing of all mail data.

Comment 14

[Ulf Zibis]

(In reply to WADA from comment #10)
I did the following on *Tb7 machine*.
> 1. Create two IMAP folders : F1(offline-use=On), F2(offline-use=off)
>    Copy the mail in the IMAP folder to F1 and F2. (single mail only in
> folder)
> Because mail copy is done by "uid xx copy F1(or F2)", copy is done at
> server. However, Tb uses localy held data after copy in mail display. So
> Repair Folder is needed after mail copy by Tb.
I didn't Repair Folder here manually, as I understand, last sentence as Tb does it automatically.
In Profile, I saw:
F1     36 KB
F1.msf  4 KB
F2      0 KB
F2.msf  4 KB
F1 offline file content:
From - Sat Dec 10 20:58:33 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
From 
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
...

> 2. F2 : Repair Folder, View/Message Source
>         As offline-use=off, there is no offline-store file.
In Profile, I saw:
F1     36 KB
F1.msf  4 KB
F2      <deleted>
F2.msf  0 KB
Select message in F2, View/Message Source:
From 
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
...

> 3-1. F1 : Repair Folder, View/Message Source, check offline-store file
> content.
Select message in F1, View/Message Source:
From 
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
...
F1 offline file content:
From - Sat Dec 10 21:40:06 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
From 
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
...

> 3-2. F1 : Compact, View/Message Source, check offline-store file content.
>           Repeat this step several times.
Select message in F1, View/Message Source:
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
...
F1 offline file content:
From 
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
...
> Compact again
Same result.
> Compact again
Same result.

Comment 15

[Ulf Zibis]

Attachment: F1 displayed by web-mailer

Comment 16

[Ulf Zibis]

Attachment: Index of F2 after all actions

Now:
F2.msf  5 KB

Comment 17

[WADA:World Anti-bad-Duping Agency]

(A) Phenomenon you saw with the mail(first llne is "From ") is bug 708941.
>   (After Repair Folder=just after download)      (After Compact)
> ------------------------------------------     ---------------------------------------------------------------------
> (separator part or Tb)                         (new separator part or Tb after compact, "From " in mail data is used)
>   From - Sat Dec 10 21:40:06 2011                From 
>   X-Mozilla-Status: 0001                       ----------------------------------------------------------------------
>   X-Mozilla-Status2: 00000000                  (mail data, "From " of mail data is changed to separator)
> ------------------------------------             Return-Path: <[e-mail]>
> (mail data)                                      (snip)
>   From 
>   Return-Path: <[e-mail]>
>   (snip)

(B) If you do same check on ordinal mail without "From " line(s) at top of mail, you can probably see next phenomenon.
>   (After Repair Folder=just after download)      (After Compact)
> ------------------------------------------     ---------------------------------------------------------
> (separator part or Tb)                         (new separator part or Tb after compact)
>   From - Sat Dec 10 21:40:06 2011                From 
>   X-Mozilla-Status: 0001                       ---------------------------------------------------------
>   X-Mozilla-Status2: 00000000                  (mail data)
> ------------------------------------             Return-Path: <[e-mail]>
> (mail data, no "From " line)                                      (snip)
>   Return-Path: <[e-mail]>
>   (snip)
This (B) is current implementation, and this is reason why I said your report of comment #0 is INVALID.

In (B), external symptom looks; X-Mozilla-Staus:, X-Mozilla-Staus2: is removed, and "- Sat Dec 10 21:40:06 2011" is removed.
However, phenomenon may be next, which can explain above (B), (A), and bug 708941.

When Compact, Tb removes separator part, and Tb adds "From " line before mail data as new separator(or new place holder).
But if mail data start swith "From ..." line, Tb doesn't add "From " line before mail data, and Tb uses the first "From ..." line of mail data as new separator(or new place holder).

Comment 18

[WADA:World Anti-bad-Duping Agency]

Anyway, "corrupted" by you in comment #0 is still INVALID,
> Actual results:
> The
> From - [timestamp]
> lines in header were corrupted to:
> From 
> (with trailing blank)
and I believe next in comment #0 is WONTFIX for FUTURE or ENHANCEMENT.
> Expected results:
> From - [timestamp]
> lines should have been conserved.

"email corrupted" in bug summary is correct, because (A) of comment #17(phenomenon of bug 708941) is "mail data loss", even if the lost mail line is incorrectly placed "From " line(s) in mail data stream by someone.

Comment 19

[Ulf Zibis]

(In reply to WADA from comment #18)
> ... is "mail data loss", even if the lost mail line is 
> line is incorrectly placed "From " line(s) in mail data stream by someone.
I now have located "Mr. somone".
... see bug 667288 comment #15.

Comment 20

[WADA:World Anti-bad-Duping Agency]

"Mr. somone"s were;
 (a)   You who copied offline-store file as local mail folder file
 (b-1) You who copied mails from the copied local mail folder file to IMAP folder
 (b-2) Tb's bug 667288 upon mail copy by (b-1)
If any of above was not done nor existent, problem you saw didn't occur.
Even if excess "From " line was generated by above, if your final recovery procedure from the excess "From " line was one like bug 667288 comment #62, Tb's bug 667288 didn't occur, because Tb fortunately has bug 708941.

Because we already reported bug 667288, bug 708941, bug 426651, and because comment #0 is main problem of a bug at B.M.O., I believe this bug is better closed as INVALID to avoid misleading

Comment 21

[Wayne Mery (:wsmwk)]

trusting wada's comment 20. But, why not dup to bug 697635?