667288 - Broken header content after copying mails to IMAP inbox folder (if format of Unix Mbox mail separator in Tb's local mail folder file is not one Tb expects, Tb sends it to server as mail data upon copy of mail to IMAP folder)

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[Ulf Zibis]

User-Agent:       Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.18) Gecko/20110616 Thunderbird/3.1.11 ThunderBrowse/3.3.5

- At top of the email content, there is only 'From' in the first line.
- My webmailer doesn't show the real From address.
(see attachments)

Reproducible: Always

Steps to Reproduce:
I had to rescue my emails from a IMAP box, because of new mailbox setup from provider's side.
1. I copied all content to temporary folders under "Locale folders".
2. The old account went offline.
3. The new account with same email address, but different login, password etc. came online.
4. After some time, I copied the old emails back to their folders.

Comment 1

[Ulf Zibis]

Attachment: corrupted email source

Created via AddOn "ViewSourceWith"

Comment 2

[Ulf Zibis]

Attachment: Web Mailer view

The sender email addresses of those emails are no more visible. Instead only 'From' appears.

Comment 3

[Ulf Zibis]

Can you please post a workaround in the meantime?
This is critical because of: https://bugzilla.mozilla.org/show_bug.cgi?id=667294
How can I remove the wrong 'From' line from my emails?

Comment 4

[Ludovic Hirlimann [:Usul]]

Did you try rebuilding the folder's index ?

Comment 5

[Ulf Zibis]

I tried, but it didn't help.
Thinking about, why shout it help? The emails content is corrupted on both sides.
It seems, that the download frequency is little higher, if the IMAP inbox folder is selected for view.

Comment 6

[rsx11m]

That's either Database or IMAP, moving to the latter first for investigation.

Comment 7

[Ludovic Hirlimann [:Usul]]

(In reply to comment #5)
> I tried, but it didn't help.
> Thinking about, why shout it help? The emails content is corrupted on both
> sides.
> It seems, that the download frequency is little higher, if the IMAP inbox
> folder is selected for view.

Cause I failed to noticed it was corrupt server side. This bugs sounds like something got corrupted on your server. Are you thinking it was corrupted by Thunderbird ? Do you have access to your server to investigate it's log files ?

Comment 8

[Ulf Zibis]

I think, Thunderbird caused the corruption, but also possible, there was an error in the interaction with the server.
Pleas see also Bug 668482.
I don't have direct access to the logs, but I have investigated to get the from the provider. They sayed, maybe monday, I can get them.

Comment 9

[Wayne Mery (:wsmwk)]

Ulf,

Do you also see this problem (and your other two problems) if you update to version 5?  If yes, then we still need your server logs.

Did you also post a topic in getsatisfaction?

Comment 10

[Joshua Cranmer [:jcranmer]]

RESOLVED INCOMPLETE due to lack of response to previous question. If you feel this change was made in error, please respond to this bug with your reasons why.

Comment 11

[Ulf Zibis]

(In reply to Wayne Mery (:wsmwk) from comment #9)
> Ulf,
> 
> Do you also see this problem (and your other two problems) if you update to
> version 5?  If yes, then we still need your server logs.
I can't reproduce the error anymore

> Did you also post a topic in getsatisfaction?
No.

Comment 12

[Ulf Zibis]

It seems, one has found the cause for the problem:
Thunderbird adds a X-Account-Key: tag to the MBox. I can imagine, that if the id of the new account is different of the old one, back-copying the MBox files causes such problems, and/or the providers IMAP server software was irritated about this uncommon tag.
See bug 704613.

Is there a best practice tutorial for rescuing emails from the old IMAP server while changing the provider?

Comment 13

[Ulf Zibis]

See also bug 426651.

Comment 14

[Ulf Zibis]

Was changed to resolved by error.
Anyway, after closer look to bug 697635, I estimate, compacting the folders before back-copying to the IMAP server could have been the source of the error.

Comment 15

[Ulf Zibis]

(In reply to Ulf Zibis from comment #0)
> Steps to Reproduce:
I now have managed to reproduce the phenomenon with Thunderbird 7.0:

1. Create folder 'Test-Inbox' in IMAP account, offline-use=On.
2. Copy some emails from Inbox to 'Test-Inbox'.
3. Compact 'Test-Inbox'.
4. Close TB.
5. Via web-mailer delete all content of 'Test-Inbox.
   (to simulate a fresh IMAP server setup)
6. Copy files 'Test-Inbox' + 'Test-Inbox.msf' to '..\Mail\Local Folders\'.
7. Open TB.
8. Select Local Folders -> 'Test-Inbox'.
9. Ctrl-A + Copy to IMAP -> 'Test-Inbox'.
10. Open web-mailer and see the corrupted 'From' fields.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf Zibis from comment #2)
> Web Mailer view
> The sender email addresses of those emails are no more visible.
> Instead only 'From' appears.

It can be called "a bug of the Web Mailer". Perhaps (a) interprets first "From " line as "From" header, even though it's not followed by ":", (b) use this first "From" header as "From" header of the mail for which only one header is permitted.

Comment 17

[WADA:World Anti-bad-Duping Agency]

Workaround:
(1) Download the mails of "From line at top of mail data" in IMAP folder to Tb's local mail folder using Tb.
=> "From ..." lines are escaped by ">" and is changed to ">From ..." in local mail folder file of Tb because Unix Mbox format is used.
(2) If Gmail IMAP, move the mails in IMAP folder(Gmail Label) to [Gmail]/Trash.
(3) Copy(upload) the mail in local mail folder of Tb to the IMAP folder using Tb.
=> ">From ..." lines at top part of message header is not sent to IMAP server by Tb, as X-Mozilla-Status:, X-Mozilla-Status2: is not sent to IMAP server(possibly removal of corupted headr line by Tb). So excess "From ..." lines at top part of mail is removed.
=> Because Tb doesn't send X-Mozilla-Status:, X-Mozilla-Status2: to IMAP server, excess X-Mozilla-Status:, X-Mozilla-Status2: is also removed.
=> ">From ..." lines in message body is sent as-is. So "From ..." line in orinal message body is changed to ">From ...".
(4) If Gmail IMAP, erase mail data in [Gmail]/Trash by "Empty Trash".

It's never Tb's fault, and Tb won't send row "From ..." line of mail to IMAP server if uploaded mail is held in Tb's local mail folder or uploaded(copied) .eml file is saved by Tb's "Save As to .eml".
Who did generate row "From ..." line at top of the mail data? Tb? I believe NEVER.

Closing as INVALID.

Comment 18

[Ulf Zibis]

(In reply to WADA from comment #16)
> It can be called "a bug of the Web Mailer". Perhaps (a) interprets first
> "From " line as "From" header, even though it's not followed by ":", (b) use
> this first "From" header as "From" header of the mail for which only one
> header is permitted.
I would say ...
Bug of the web-mailer:
- using "From " line instead "From:..." as indicator for the senders email address
- displaying "From" as senders email address
- should better reject the wrong formatted email data with error message
Bug of Tb:
- uploading wrong formatted email data

Comment 19

[Ulf Zibis]

Correction:
Bug of the IMAP server:
- using "From " line instead "From:..." as indicator for the senders email address
- should better reject the wrong formatted email data on upload with error message
Bug of the web-mailer:
- displaying "From" as senders email address

Comment 20

[Ulf Zibis]

(In reply to WADA from comment #17)
I'm not sure, if we are talking about the same:
I don't had "From ..." lines, but one "From " line at top of each mail in local folder mbox file after step 6 in comment #15

> It's never Tb's fault, and Tb won't send row "From ..." line of mail to IMAP
> server if uploaded mail is held in Tb's local mail folder.
In step 9 of comment #15, uploaded mail, held in Tb's local mail folder, was starting with "From " line.

> Who did generate row "From ..." line at top of the mail data? Tb? I believe
> NEVER.
It was generated by Compact of Tb in step 3 of comment #15, 
and if later in step 9 Tb doesn't send that "From " line to IMAP server, where should it come from, as (a) the later downloaded mail again contained the "From " line in the mbox file and (b) if I copy "correct" mail without "From " line, but "From - timestamp" line from local folder to IMAP folder, then the mail is correctly displayed in web-mailer and additionally correctly uploaded to the client later, without containing "From " line in the mbox file.

Additionally note another maybe "From " related problem: bug 709504 comment #7.
After this, the web-mailer showed 'From' for the 1st mail, but correct email address for the 2nd mail.

Comment 21

[Ulf Zibis]

And thanks for your workaround, but I have had managed the "repair" long time ago, just after posting this bug, by manually editing the local folders temporary mbox file before back-copying to the IMAP Inbox.

Comment 22

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf Zibis from comment #20)
> > It's never Tb's fault, and Tb won't send row "From ..." line of mail to IMAP
> > server if uploaded mail is held in Tb's local mail folder.
> In step 9 of comment #15, uploaded mail, held in Tb's local mail folder, was
> starting with "From " line.

Are you talking about file of '..\Mail\Local Folders\Test-Inbox' created by YOU, never by Tb?
> 6. Copy files 'Test-Inbox' + 'Test-Inbox.msf' to '..\Mail\Local Folders\' 

Because local mail folder, "From ..." line is Unix Mbox separator, Tb splits mails at the row "From ..." line upon rebuild-index. Because .msf file is not copatible etween IMAP folder and local mail folder, rebuild-index is invoked internally upon first click(open) of Test-Inbox under 'Local Folders'. If no valid mail data under the "From ..." line, Tb doesn't look to show it as null mail.

Do you see row "From ..." line by View/Message Source of mail in 'Test-Inbox' under 'Local Folders'?

Do next.
- Create Test-Inbox-2 under 'Local Folders'
- Copy all mails in Test-Inbox under 'Local Folders' Test-Inbox-2 under 'Local Folders'
Do you see row "From " line in file of '..\Mail\Local Folders\Test-Inbox-2'?

Do next too.
- Copy '..\Mail\Local Folders\Test-Inbox-2' to '..\Mail\Local Folders\Test-Inbox-3'
- Restart Tb, click(open) Test-Inbox-3 under 'Local Folders'
- Compact Test-Inbox-3 under 'Local Folders'
Do you see row "From " line in file of '..\Mail\Local Folders\Test-Inbox-3'

Comment 23

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf Zibis from comment #20)
> > Who did generate row "From ..." line at top of the mail data? Tb? I believe
> > NEVER.
> It was generated by Compact of Tb in step 3 of comment #15, 

As you already know in your bug 66848, Tb eats first "From " line of mail data in offline-store file when Compact of IMAP offline-use=on folder is executed.
At which step in your comment#15 can Tb sends row "From " line to server?

If row "From " line(not Unix Mbox separator of local mail folder of Tb) was held in '..\Mail\Local Folders\Test-Inbox', it means that multiple row "From " lines existed at server, because you did Compact of IMAP offline-use=on folder named Test-Inbox only once in steps of comment #15.  

> and if later in step 9 Tb doesn't send that "From " line to IMAP server,
> where should it come from, as (a) the later downloaded mail again contained
> the "From " line in the mbox file and (b) if I copy "correct" mail without
> "From " line, but "From - timestamp" line from local folder to IMAP folder,
> then the mail is correctly displayed in web-mailer and additionally
> correctly uploaded to the client later, without containing "From " line in
> the mbox file.

As I wrote in previous comment, "From " line is Unix Mbox mail separator, and Tb never interpret it as mail data upon rebuild-index of local mail folder.

Even if row "From " line is contained in '..\Mail\Local Folders\Test-Inbox', I can't think there is possibility of "Tb sends row From line to server" in steps of comment #15.
Can you show step by step data of local mail folder file content and View/Message Source?

Comment 24

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf Zibis from comment #20)
> After this, the web-mailer showed 'From' for the 1st mail, but correct email
> address for the 2nd mail.

At any IMAP folder? Or specific IMAP folder only?

Even after you did recovery operation like next?
- Remove row "From " line in mail data from '..\Mail\Local Folders\Test-Inbox'
  by manual editing of the file.
- Rebuild-Index if required
  (as file size is changed, it's invoked automatically upon first open by click)
- Upload to IMAP server

If you still believe that the "From " line was uploaded by Tb from local mail folder of Tb, get IMAP log for uploading of mail by Tb, and show us evidence, please.
I can't imagine other than next cases if uploaded by Tb.
(i) Copy mails which contains row "From " lines in mail data,
    from IMAP folder to IMAP folder of other IMAP account.
(ii) Import of .eml file which contains row "From " lines into IMAP folder,
     via Drag&Drop of .eml file to IIMAP folder or thread pane of IMAP folder.
(iii) Addition of row "From " line(s) to Tb's file for local mail folder was
      done while Tb is running. Because .msf file is already opened, internal
      rebuild-index is not invoked automatically, but rebuild-index was not
      requested by user.
      So, added row "From " line is read as mail data by Tb upon mail copy
      from local mail folder to IMAP mail folder, then sent as-is to server.

Comment 25

[Ulf Zibis]

(In reply to WADA from comment #22)
> (In reply to Ulf Zibis from comment #20)
> Are you talking about file of '..\Mail\Local Folders\Test-Inbox' created by
> YOU, never by Tb?
Yes, but '...\ImapMail\xxx\Test-Inbox' was created by Tb, compacted and, after closing Tb, externally copied to '..\Mail\Local Folders\Test-Inbox'

> Because local mail folder, "From ..." line is Unix Mbox separator, Tb splits
> mails at the row "From ..." line upon rebuild-index.
In my case there was only "From " line, as copied from IMAP mbox file, but splitting the mails under Local Folders by "From " even worked without problems.

> Because .msf file is
> not copatible etween IMAP folder and local mail folder, rebuild-index is
> invoked internally upon first click(open) of Test-Inbox under 'Local
> Folders'. If no valid mail data under the "From ..." line, Tb doesn't look
> to show it as null mail.
There are no "From ..." lines, but shouldn't Tb convert the the "From " lines while rebuilding index?

> Do you see row "From ..." line by View/Message Source of mail in
> 'Test-Inbox' under 'Local Folders'?
I see neither "From ..." nor "From " lines by View/Message Source.

> Do next.
Will do later.
But I did:
1. Create folder 'Test-Inbox' in IMAP account, offline-use=On.
2. Copy 1 mail from Inbox to 'Test-Inbox'.
3. Compact 'Test-Inbox'.
4. Copy same mail again from Inbox to 'Test-Inbox'.
5. Close TB.
6. Via web-mailer delete all content of 'Test-Inbox.
   (to simulate a fresh IMAP server setup)
7. Copy file 'Test-Inbox' to '..\Mail\Local Folders\'.
8. Open TB.
9. Select Local Folders -> 'Test-Inbox'.
10. Ctrl-A + Copy to IMAP -> 'Test-Inbox'.
11. Open web-mailer and see 1 corrupted and 1 correct 'From' field.

Comment 26

[Ulf Zibis]

Attachment: Screen shot, 1 correct + 1 corrupted From field

Comment 27

[Ulf Zibis]

..\Mail\Local Folders\Test-Inbox:
From 
X-Mozilla-Keys: nonjunk                                                                         
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])
...
From - Mon Dec 12 12:38:43 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys: nonjunk                                                                         
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])


..\ImapMail\xxx\Test-Inbox:
From 
X-Mozilla-Keys: nonjunk                                                                         
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])
...
From - Mon Dec 12 12:38:43 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys: nonjunk                                                                         
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])
...
From - Sat Dec 17 23:07:49 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys: nonjunk                                                                         
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])
...
From - Sat Dec 17 23:07:49 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
From 
X-Mozilla-Keys: nonjunk                                                                         
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])

Comment 28

[Ulf Zibis]

If I delete IMAP file 'Test-Inbox', I get after new download from the IMAP server:
..\ImapMail\xxx\Test-Inbox:
From - Thu Dec 22 00:47:50 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys: nonjunk                                                                         
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])
...
From - Thu Dec 22 00:48:13 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
From 
X-Mozilla-Keys: nonjunk                                                                         
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])
...

As you can see, there is again a "From " line in the 2nd mail data.
If not before uploaded by Tb on step 4 at comment #25, how should it got to there otherwise?
Additionally I'm wondering about the "X-Mozilla-Keys: nonjunk  " lines. Shouldn't all "X-Mozilla-Xxx: " lines have been filtered out before upload to the IMAP server?

Comment 29

[Ulf Zibis]

(In reply to WADA from comment #22)
> Do next.
> - Create Test-Inbox-2 under 'Local Folders'
> - Copy all mails in Test-Inbox under 'Local Folders' Test-Inbox-2 under
> 'Local Folders'
> Do you see row "From " line in file of '..\Mail\Local Folders\Test-Inbox-2'?
Yes, there are "From " lines for each mail without "X-Mozilla-Status|2: ..." lines.

> Do next too.
> - Copy '..\Mail\Local Folders\Test-Inbox-2' to '..\Mail\Local
> Folders\Test-Inbox-3'
> - Restart Tb, click(open) Test-Inbox-3 under 'Local Folders'
> - Compact Test-Inbox-3 under 'Local Folders'
> Do you see row "From " line in file of '..\Mail\Local Folders\Test-Inbox-3'
Yes, there are "From " lines for each mail without "X-Mozilla-Status|2: ..." lines.

Comment 30

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf Zibis from comment #28)
> I'm wondering about the "X-Mozilla-Keys: nonjunk  " lines.
> Shouldn't all "X-Mozilla-Xxx: " lines have been filtered out before upload
> to the IMAP server?

You already posted comment in Bug 426651, didn't you?

Comment 31

[Ulf Zibis]

(In reply to WADA from comment #23)
> As you already know in your bug 66848,
Are you sure, you reference the correct bug no. here?

> Tb eats first "From " line of mail
> data in offline-store file when Compact of IMAP offline-use=on folder is
> executed.
> At which step in your comment#15 can Tb sends row "From " line to server?
Step 9.

> If row "From " line(not Unix Mbox separator of local mail folder of Tb) was
> held in '..\Mail\Local Folders\Test-Inbox', it means that multiple row "From
> " lines existed at server, because you did Compact of IMAP offline-use=on
> folder named Test-Inbox only once in steps of comment #15.
I'm not sure, if I understand right here. Multiple "From " lines existed in IMAP offline-use=on folder named Test-Inbox after compact (step 3 of comment 15), but they did not exist at server at that time, but they existed in '..\Mail\Local Folders\Test-Inbox after step 6 and in at server + IMAP folder after step 9.

> > and if later in step 9 Tb doesn't send that "From " line to IMAP server,
> > where should it come from, as (a) the later downloaded mail again contained
> > the "From " line in the mbox file and (b) if I copy "correct" mail without
> > "From " line, but "From - timestamp" line from local folder to IMAP folder,
> > then the mail is correctly displayed in web-mailer and additionally
> > correctly uploaded to the client later, without containing "From " line in
> > the mbox file.
> 
> As I wrote in previous comment, "From " line is Unix Mbox mail separator,
> and Tb never interpret it as mail data upon rebuild-index of local mail
> folder.
... but seems to while copying the mails to IMAP folder.

> Even if row "From " line is contained in '..\Mail\Local Folders\Test-Inbox',
> I can't think there is possibility of "Tb sends row From line to server" in
> steps of comment #15.
> Can you show step by step data of local mail folder file content and
> View/Message Source?
See comment 27.
+ View/Message Source:
From 
X-Mozilla-Keys: nonjunk                                                                         
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])
...

Comment 32

[WADA:World Anti-bad-Duping Agency]

Woops, wrong bug. it was Bug 426651.

Comment 33

[Ulf Zibis]

(In reply to WADA from comment #30)
> (In reply to Ulf Zibis from comment #28)
> > I'm wondering about the "X-Mozilla-Keys: nonjunk  " lines.
> > Shouldn't all "X-Mozilla-Xxx: " lines have been filtered out before upload
> > to the IMAP server?
> 
> You already posted comment in Bug 426651, didn't you?
I didn't mention "X-Mozilla-Xxx: " lines there, that was only in _your_ comment 1.

Comment 34

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf Zibis from comment #31)
> + View/Message Source:
> From 
> X-Mozilla-Keys: nonjunk
> Return-Path: <[e-mail]>
> Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
> Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])
> ...

Confusing but View/Message Source of Tb is different between local mail folder and IMAP folder.
If local mail folder, Tb's View/Message Source shows all lines from Unix Mbox mail separator.
In contrast to it, Unix Mbox mail separator doesn't exist if IMAP offline-use=off folder, so it's never shown. "From ..." line(... = 0 to N characters), which may be followed by X-Mozilla-Status:/X-Mozilla-Status2: if atter first download before Compact, is merely "place holder of a mail" in offline-store file of IMAP offline-use=on folder, and it's also not shown.

(In reply to Ulf Zibis from comment #29)
> Yes, there are "From " lines for each mail without "X-Mozilla-Status|2: ..." lines.

I meant next by row "From " line.
  "From " line, or line starts with "From "
   which is NOT unix mbox mail separator used by Tb
   in Tb's file for local mail folder.
i.e.
"From " line, or line starts with "From ", which is shown as message header or message body text by View/Message Source of Tb, instead of first line of message source display of mail in local mail folder(usually before X-Accout-Key:, before X-UIDL:, before X-Mozilla-Status:/X-Mozilla-Status2:, before X-Mozilla-Keys:).

I believe you already saw;
Tb escapes "From " line by ">" in Tb's local mail folder file when row "From " line is sent from IMAP server upon mail copy from IMAP folder to local mail folder.

And I believe you already saw;
Even if row "From " line was generated by manual copy of offline-store file of IMAP folder to local mail folder file, the row "From " line is treated as Unix Mbox mail separator by Tb.
i.e.
If following data of IMAP offline-store file is copied to local mail folder file by manual file copy by YOU(never by Tb), these are two different mails for Tb because of local mail folder file.
  (a) mail data of X-Mozilla-Status:/X-Mozilla-Status2: only.
      no message headers such as From:, To:, Subject:, Received:,
      no message body data.
  (b) mail data without X-Mozilla-Status:/X-Mozilla-Status2:.
      starts from X-Mozilla-Keys: header.
  Tb doesn't look to show mail (a) at thread pane.

From - Sat Dec 17 23:07:49 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
From 
X-Mozilla-Keys: nonjunk
Return-Path: <[e-mail]>
Delivery-Date: Wed, 20 Jan 2010 20:41:40 +0100
Received: from www.goatrance.de (www.goatrance.de [81.88.37.27])

Note:
After successful first Compact execution of local mail folder, X-Mozilla-Status:/X-Mozilla-Status2: is added, if X-Mozilla-Status:/X-Mozilla-Status2: doesn't exist.
And, affter second successful Compact execution of local mail folder, X-Mozilla-Keys: is added, if X-Mozilla-Keys: doesn't exist.
This is current implementation.(In reply to Ulf Zibis from comment #33)

> (In reply to WADA from comment #30)
> I didn't mention "X-Mozilla-Xxx: " lines there, that was only in _your_  comment 1.

I believe "posting comment to a bug at B.M.O" means "comment poster already read the bug at B.M.O".

Comment 35

[Ulf Zibis]

(In reply to WADA from comment #32)
> Woops, wrong bug. it was Bug 426651.
Maybe you meant bug 697635 or bug 708941?

Comment 36

[WADA:World Anti-bad-Duping Agency]

> > As you already know in your bug 66848,
> Maybe you meant bug 697635 or bug 708941?
It was your bug 668482 (last 2 was cut upon copy&paste. sorry for misoperation.)

Comment 37

[WADA:World Anti-bad-Duping Agency]

Read "row" as "raw" in my previous comments, please. Sorry for many typo's.

Comment 38

[Ulf Zibis]

(In reply to WADA from comment #24)
> (In reply to Ulf Zibis from comment #20)
> > After this, the web-mailer showed 'From' for the 1st mail, but correct email
> > address for the 2nd mail.
> 
> At any IMAP folder? Or specific IMAP folder only?
At specific 'Test-Inbox' folder on 1st try of scenario from comment 25.

> Even after you did recovery operation like next?
> - Remove row "From " line in mail data from '..\Mail\Local
> Folders\Test-Inbox'
>   by manual editing of the file.
If I would remove all "From " lines, there wouldn't remain any Unix Mbox mail separators, as there are no "From ..." lines, so multiple mails would be interpreted as _one_ mail. So does that make sense?

Comment 39

[Ulf Zibis]

Attachment: IMAP logs

(In reply to WADA from comment #24)
> If you still believe that the "From " line was uploaded by Tb from local
> mail folder of Tb, get IMAP log for uploading of mail by Tb, and show us
> evidence, please.

Comment 40

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf Zibis from comment #38)
> > Even after you did recovery operation like next?
> > - Remove row "From " line in mail data from '..\Mail\Local
> > Folders\Test-Inbox'
> >   by manual editing of the file.
> If I would remove all "From " lines, there wouldn't remain any Unix Mbox
> mail separators, as there are no "From ..." lines, so multiple mails would
> be interpreted as _one_ mail. So does that make sense?

You are correct.
Remove "Raw" "From ..." line(...=0 to N chars) which is not valid Unix Mbox mail separator, if you copied offline-store file as file for local mail folder of Tb.
i.e.
Remove "From " line in following data
  in file for local mail folder of Tb,
  which was copied from offline-store file of IMAP folder of offline-use=on. 

From - Sat Dec 17 23:07:49 2011 <= Used as Unix Mbox mail separator by Tb
X-Mozilla-Status: 0001          <= Used as correct X-Mozilla-Status:
X-Mozilla-Status2: 00000000     <= Used as correct X-Mozilla-Status2:
From                            <= '"Raw" "From " line' I called
X-Mozilla-Keys: nonjunk         <= Used as correct X-Mozilla-Keys:
Return-Path: <[e-mail]>

This is to ensure that Tb never sends(upload, append) "From " line to IMAP server if mail copy from local mail folder to IMAP folder. 

So, above can be replaced by next, because "From " only line is used as Unix Mbox mail seprator by Tb if local mail folder.

Do following before upload mails to IMAP server(copy from local mail folder to IMAP folder) to make it sure.
  1. "Repair Folder" of the local mail folder
  2. Copy a mail to the local mail folder, and delete the copied mail.
     Compact (first Compact).
  3. Copy a mail to the local mail folder, and delete the copied mail.
     Compact (second Compact).

Comment 41

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf Zibis from comment #39)
> IMAP logs

Which line of which log file is evidence that Tb sended "From " only line to IMAP server?

Comment 42

[WADA:World Anti-bad-Duping Agency]

I coul see it at Line 21592 in imap_2.log.
> 16196[a35e40]: 1dcf000:imap.1und1.de:S-Test-Inbox-4:SendData: 10 append "Test-Inbox-4" "20-Jan-2010 20:41:42 +0100" {36248+}[CR]
> [CRLF]
> 16196[a35e40]: 1dcf000:imap.1und1.de:S-Test-Inbox-4:SendData: From [CR]
> [CRLF]                                                                        
> X-Mozilla-Keys: nonjunk    (consecutive spaces)    [CR]
> [CRLF]                                                                        

Is this copy from local mail folder to IMAP mail folder of /Test-Inbox-4 ?
If copy of mail from IMAP folder to IMAP older of other IMAP server, or if import of .eml file which contains "From " line by Drag&Drop of .eml(a way to reproduce raw From line at IMAP folder), this is normal flow.

Did you explicitely execute "Repair Folder" at local mail folder after copy of imap offline-store file to local mail folder file?

Comment 43

[WADA:World Anti-bad-Duping Agency]

It may be "new line" character relevant.
Can you attach IMAP offline-store file which contains mail of "Raw" "From " only line as first/top mail and several mails additionally.

Did you copy .msf file for offline-store file directory for local mail folder too?
If so, it may be "internal rebuild-index is not automatically executed on copy of offline-store file & .msf file".

Comment 44

[WADA:World Anti-bad-Duping Agency]

Is the uploaded(append'ed) "From " line escaped in local mail folder file as ">From "(escaped by >) or " From "(escaped by space)?

Comment 45

[WADA:World Anti-bad-Duping Agency]

FYI.

Quick check result around From line with Gmail IMAP and Yahoo! IMAP.
(A) Escaped "From " line at messeage header part of mail in local mail folder.
    ">From "(escaped by >), " From "(escaped by space)
    Tb 8 sends this line as-is to IMAP server.
(A-1) Gmail IMAP  : Rejects it, because syntax error of message header
(A-2) Yahoo! IMAP : Accepts it, and ignores it.
(B) Raw "From " line at messeage header part of mail in .eml file.
    "From " only line.
    Tb 8 sends this line as-is to IMAP server upon Drag&Drop of .eml file.
(B-1) Gmail IMAP  : Accepts it, and sends it as mail data line to IMAP client.
(B-2) Yahoo! IMAP : Accepts it, and ignores it.
So, phenomenon you are looking is IMAP server dependent.

(C) If local mail folder file(call F1) content is as follows, 

++ F1 content 
From - Mon Dec 01 16:09:04 2011
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
From 
Date: Sat, 17 Dec 2011 16:08:56 +0900
From: Boyacky at RocketMail <[e-mail]>
(snip)

local mail folder file(call F2) content become as follows, after "Repair Folder" of local folder named F1, copy mail in F1 to newly created local mail folder named F2.

++ F2 content 
From 
Date: Sat, 17 Dec 2011 16:08:56 +0900
From: Boyacky at RocketMail <[e-mail]>
(snip)

"From - Mon ...", "X-Mozilla-Status: ...", "X-Mozilla-Status2: ..." is removed. "From " line in F2 is Unix Mbox mail separater of Tb's local mail folder file. Tb doesn't look to add X-Mozilla-... header when local mail copy.

Above (A)/(B)/(C) is reason why I said "Tb won't send raw From line if mail copy from local mail folder to IMAP folder".

However, if [CR] is sent by IMAP server, data in offline-store file can be one like next.
...[CRLF]
...[CR]From [CR]X-Account-Key: ...[CR]...
[CRLF]
If this offline-store file is copied by user as local mail folder file, these are perhaps sent to IMAP server as-is upon mail copy from local mail folder to IMAP folder, because "new line" of local mail folder file of Tb is [CRLF] instead of [CR].
This is reason why I asked you to attach small offline-store file in your environment.

Comment 46

[WADA:World Anti-bad-Duping Agency]

FYI.
[CR] in local mail folder file, which was originally IMAP offline-store file, may cause some problems.

(1) ++ Local mail folder F1 content
From 1[CR]
Subject: mail2[CR]
[CR] <= null line, separator of header and payload
data1[CR]
From 2[CR]
Subject: mail2[CR]
[CR] <= null line, separator of header and payload
data2[CR](End of file)

(2) After Compact of F1, these mails are accessed normally.
    Tb is basically torelant with mixture of [CRLF], [CR], [LF]
    in mail data held in local mail folder file. 

(3) Copy these two mails in F1 to local mail folder of F2.
++ Local mail folder F2 content
(mail data lines for existent mails)
From 1[CR]
Subject: mail1[CR]
[CR] <= null line, separator of mheader and payload
data1[CR]
From 2[CR]
Subject: mail2[CR]
[CR](End of file) <= null line, separator of header and payload

(3-1) data2[CR] of mail2 at bottom of F1 is lost by local mail copy.
      So, nothing is shown as message body of mail2

(4) Copy other two mails which are similar to mail data in F1(mail4 is at bottom of file) to local mail folder of F2.

++ Local mail folder F2 content
(mail data lines for existent mails)
From 1[CR]
Subject: mail1[CR]
[CR] <= null line, separator of header and payload
data1[CR]
From 2[CR]
Subject: mail2[CR]
[CR]
From 3[CR]
Subject: mail3[CR]
[CR] <= null line, separator of header and payload
data3[CR]
From 4[CR]
Subject: mail4[CR]
[CR](End of file) <= null line, separator of mheader and payload

(4-1) "From 3" for mail3 is shown as message body of mail2.

(4-2) data4[CR] of mail4 at bottom of F1 is lost by local mail copy.
      So, nothing is shown as message body of mail4

You may be looking phenomenon like (4-1).

Comment 47

[WADA:World Anti-bad-Duping Agency]

Correction of previous comment. sorry for spam.
Wrong:
  (2) After Compact of F1, these mails are accessed normally.
Correct:
  (2) After Repair Folder of F1, these mails are accessed normally.

Comment 48

[WADA:World Anti-bad-Duping Agency]

Attachment: Mail to see problem due to CR

Mail folder file consists of one mail.
  Subject: mail with CR[CR]
  1-st message body line :
       ABCDEFGHIJKLMNOPQRSTUVWXYZ[CR]
  2-nd(last) message body line, bottom of mail folder file :
      (string of 400 bytes)[CR]
  Any mail data line is terminated by [CR].

(1) Create local mail folder(call F1), or delete all mails of F1/Compact.
(2-1) Copy test mail of "mail with CR" to F1.
      ABCDEFGHIJKLMNOPQRSTUVWXYZ only is shown as message body,
      because (string of 400 bytes) is not copied to file for F1.
(2-2) Copy any mail in any mail folder to F1.
      Message body display of "mail with CR" :
        ABCDEFGHIJKLMNOPQRSTUVWXYZ is shown as first line.
        400 bytes from "From ..." line for this copied mail is shown as body,
        because mail data length of "mail with CR" is same as original.
(3) Repeat step (2) multiple times.
      
Because mail data line of (string of 400 bytes) is not copied to file for F1, ABCDEFGHIJKLMNOPQRSTUVWXYZ line only is shown after Repair Folder(rebuild-index) of F1.

Comment 49

[Ulf Zibis]

Hi WADA,
sorry for being at sleep here in Europe while you are twisting your brain about possible "new line" character relevant phenomenons.
In my files, _all_ lines end by [CR][LF], so there should not be any worry about that.
But have you recognized the 73 spaces (mark text by mouse in comment 27):
X-Mozilla-Keys: nonjunk<--73 spaces-->[CR][LF]

Comment 50

[Ulf Zibis]

(In reply to WADA from comment #45)
> FYI.
> So, phenomenon you are looking is IMAP server dependent.
That's what I wanted to say by comment 18, comment 19.

Comment 51

[Ulf Zibis]

(In reply to WADA from comment #34)
> I believe you already saw;
> Tb escapes "From " line by ">" in Tb's local mail folder file when row "From
> " line is sent from IMAP server upon mail copy from IMAP folder to local
> mail folder.
Yes.

> And I believe you already saw;
> Even if row "From " line was generated by manual copy of offline-store file
> of IMAP folder to local mail folder file, the row "From " line is treated as
> Unix Mbox mail separator by Tb.
Yes!
 
> I believe "posting comment to a bug at B.M.O" means "comment poster already
> read the bug at B.M.O".
Ah, now understood and agreed. Sorry for the spam, I had forgotten about that bug.

(In reply to WADA from comment #40)
> Do following before upload mails to IMAP server(copy from local mail folder
> to IMAP folder) to make it sure.
>   1. "Repair Folder" of the local mail folder
>   2. Copy a mail to the local mail folder, and delete the copied mail.
>      Compact (first Compact).
>   3. Copy a mail to the local mail folder, and delete the copied mail.
>      Compact (second Compact).
Thanks for the alternative work around.

(In reply to WADA from comment #42)
> Is this copy from local mail folder to IMAP mail folder of /Test-Inbox-4 ?
Yes.

> If copy of mail from IMAP folder to IMAP older of other IMAP server, or if
> import of .eml file which contains "From " line by Drag&Drop of .eml(a way
> to reproduce raw From line at IMAP folder), this is normal flow.
But incorrect as well, as IMHO, Tb should reject/repair corrupted mail data or at least only continue with a warning instead _silently_ forwarding the erroneous data to any other party e.g. IMAP server, local .eml file.

> Did you explicitely execute "Repair Folder" at local mail folder after copy
> of imap offline-store file to local mail folder file?
No, I strictly followed steps of comment 15 except copying .msf file, as irrelevant.

Comment 52

[Ulf Zibis]

(In reply to WADA from comment #44)
> Is the uploaded(append'ed) "From " line escaped in local mail folder file as
> ">From "(escaped by >) or " From "(escaped by space)?

No!

Comment 53

[WADA:World Anti-bad-Duping Agency]

Internal rebuild-index was surely invoked on the copied offline-store file, because you didn't copy .msf file.
It looks Tb sends "From " line, even though only "From ..." line as Unix Mbox mail separator exists in mail folder file(even if excess "From " line is contained, this line and following excess lines till next "From ..." line is discarded by rebuild-index), and even though there is no escaped "From ..." line(">From ..." or " From ...").
 
If not problem like [CR] relevant phenomenon(Tb uses data for next mail as mail data of this mail in this case), next header is relevant?
> From:<[e-mail]>

Tb has problem8known bug) when header like next;
> Subject: ...[CRLF]
> [Htab or Space][some Spaces][CRLF]
> [CRLF] <= separator of header and payload
In this case, Tb accesses uninitialized memory area because of "space only folded line just before separator of header and payload", so anything wrong can happen if data in uninitialize memory area is used as pointer to data.
Tb may search space after "From:", and may wrongly use "From " of "From ..." line(Unix Mbox mail separator) as mail data.

Comment 54

[WADA:World Anti-bad-Duping Agency]

Other headers without space after ":".
> X-Sender:<[e-mail]>
> X-Mailer:PHP Goabase Email Robot
> X-List:Goabase * Parties and Peopl

Comment 55

[Ulf Zibis]

In case of corrupt(=incompatible) mbox files,
my conclusion for now (in priority from top):

Tb should ...
- use same/compatible mbox format for IMAP and other folders.
- forum (and even bug) posters should never speak about manually moving/copying mail folder files around ;-) :-D
(or at least refer to detailed consequences)
- ask user about repair choices.
- verbosely automatically repair them.
- silently automatically repair them.
- reject them by error message
- silently ignore them.

Tb should NOT ...
- forward any corrupt/erroneous data _silently_ to any other party e.g. IMAP server, local .eml file.
- treat/change any corrupt/erroneous data _silently_ internally.

Comment 56

[Ulf Zibis]

Attachment: IMAP mbox after new download from server

(In reply to WADA from comment #53)
> Tb has problem8known bug) when header like next;
> > Subject: ...[CRLF]
> > [Htab or Space][some Spaces][CRLF]
> > [CRLF] <= separator of header and payload
I believe, my '"From" as sender address in web-mailer + corrupt mail data'-problem is independent from those effects, even there are some [Htab]'s in my test data, but the original 63 emails were quite different, but all failed.
So here it is (new download from server after steps in comment 25)...

Comment 57

[Ulf Zibis]

Hey, was a heavy birth!
Thanks for your patience.

+ don't forget the consequences of this bug, as they are ...
- irresponsible Windows XP (~=crash) --> bug 667294
- dataloss of all open data.

Comment 58

[WADA:World Anti-bad-Duping Agency]

(In reply to Ulf Zibis from comment #56)
> Created attachment 583775 [details]
> IMAP mbox after new download from server

Two identical mail data are contained in the file.
If they are split to two files, and if line of "From " only is removed from file for senond mail, differene was next only.
++ First mail
> From - Thu Dec 22 00:47:50 2011[CRLF]
++ Second mail
> From - Thu Dec 22 00:48:13 2011[CRLF]

++ First 7 lines of second mail ("From " only line is seen)
> From - Thu Dec 22 00:48:13 2011
> X-Mozilla-Status: 0001
> X-Mozilla-Status2: 00000000
> From 
> X-Mozilla-Keys: nonjunk     <= note: excess spaces are delete before paste
> Return-Path: <[e-mail]>

Do you have two identical mails with different UID in the IMAP mail folder?
(Show "Order Received" column. Shown value is UID of mail in IMAP folder if IMAP folder. If local mail folder, value is offset in local mail folder file)
Or you have only one mail in the IMAP mail folder?

Comment 59

[Ulf Zibis]

(In reply to WADA from comment #58)
> Do you have two identical mails with different UID in the IMAP mail folder?
Correct.
As I detailed in comment 25, I have copied the same original mail twice, so in result we have 2 duplicates, one uploaded before the compacting, and on after.

Comment 60

[WADA:World Anti-bad-Duping Agency]

Problem was simple.
Tb doesn't remove Unix Mbox mail separator line in local mail folder file upon IMAP upload, if some variants of the separator is used in local mail folder file.
i.e. "From ..." line version of Bug 426651 for X-Mozilla-Keys:.

++ Local mail folder file content

From xyz                    <= A variant of Unix Mbox mail separator
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys: tag1
Subject: mail1
(snip)

++ IMAP offline-store file content after "Copy from local mail folder", "Repair Folder"(to force re-download).

From - Fri Dec 23 09:17:29 2011 <= Place holder of Unix Mbox separator format
X-Mozilla-Status: 0001          <= Place holder line no. 2
X-Mozilla-Status2: 00000000     <= Place holder line no. 3
From xyz                  <= Data sent from server == Tb sent to server upon copy
X-Mozilla-Keys: tag1      <= Data sent from server == Tb sent to server upon copy
Subject: mail1
(snip)

Why I couldn't see above "From ..." line in my previous duplication test was;
   I used Yahoo! IMAP in test like above instead of Gmail IMAP.
   Yahoo! IMAP eated up this "From ..." line, as I stated before.

Confirming.

Unix Mbox Mail separator has many variants. For example,
(a) "From "
(b) "From ..."   (... = any non-space chars) 
(c) "From ..." + timestamp  (... = any non-space chars. Opera uses message-id) 
(d) "From -"   + timestamp  (A variant of (c) type. Tb writes this type.)
(e) "From ..." + timestamp + comments  (... = any non-space chars) 
Rebuild-Index of Tb uses rule of "line starts with From+space" for compatibility.

IMAP copy of Tb perhaps looks all mail data including Unix Mbox mail separator and removes type (d) line only upon upload to IMAP in addition to removal of X-Mozilla-Status/Status2.

Why you always saw problem of Web Mailer on first mail is;
  When multiple mails are selected at thread pane and copy is requested,
    if copy target is local mail folder, copy is done in sorted order.
    if copy target is imap folder, copy is done in reversed order.
Because mail with "From " line is second mail in your case, and sorted order is always ascending just after rebuild-index, second mail is appended first and first mail is appended second.
              
Compact of the local mail folder, Mail copy to other local mail folder, didn't alter this Unix Mbox mail separator line. So, manual removal of "From " line or manual format change of "From " line is needed in your case.
However, proper way in your case is "copy from IMAP folder to local mail folder" instead of "use IMAP offline-store file as local mail folder file" which is never officially supported.
Because "From " line is escaped and is changed to ">From " in local mail folder file, you can easily remove excess "From " by removal of ">From " line, if server rejects ">From " line in message header.

Comment 61

[WADA:World Anti-bad-Duping Agency]

FYI.

"use IMAP offline-store file as local mail folder file" is impossible in next case.
- offline-store file size > limitation of file size of local mail folder file
  current file size limitation : 4GB(Win), 2GB(Linux, Mac OS X)
  If offline-store file, 64bits integer is used for offset in file,
  thus, practically, there is no limitation in file size.
  So, unless user splits file by external utilities, file can't be used.

"use IMAP offline-store file as local mail folder file" always produces this bug's problem, if mails are uploaded again to IMAP server,
because,
  as you already know, "place holder in IMAP offline-store file" is always
  From+Space after Compact of IMAP folder of offline-use=on,
and because,
  there is no way to alter Unix Mbox mail separator in Tb's local mail folder
  file once generated by Tb or someone.

Comment 62

[WADA:World Anti-bad-Duping Agency]

FYI.
Currently available recovery procedure from excess "From " line in IMAP folder which was generated by your "copy of offline-store file as local mail folder file" + Tb's this bug is : Utilizing Tb's bug 708941.

(1) "Repair Folder" of the IMAP folder.
    => All mail data is downloaded again.
       "From " line is written in offline-store file as top line of mail data. 
(2) Compact the IMAP folder, until all "From " lines at top is eaten by Tb.
    => Due to bug 708941, excess "From " line is used as "place holder" of mail
       in IMAP offline-store file by Tb.
(3) Copy all mails to local mail folder.
    => Tb writes Unix Mbox mail separator to local mail folder
       in standard format for Tb.
       From - Sat Dec 24 02:38:40 2011
    => Excess "From " is not written in escaped format of ">From ", because
       there is no excess "From " in Tb's offline-store file due to bug 708941.
(4) Delete all mails in the IMAP folder, and Compact.
(5) Copy all mails in the local mail folder to the IMAP folder.
    => Because Unix Mbox mail separator is standard format of Tb,
       this bug won't occur.
    => Because there is no excess "From " line nor ">From " line,
       garbled data is not sent to IMAP server.

Comment 63

[gene smith]

Seem like pretty obscure bug. WADA suggests some workarounds in the last few comment.