1684639 - Multiple sent mails combined into one mail

Status: RESOLVED INCOMPLETE

(MailNews Core :: Database, defect)

Description

[Jens Müller (:tessarakt)]

TB version: 85.0b2

I just encountered a mail in my "Sent" folder which seems to be a combination of multiple sent mails.

The message source looks like this (some personal info deleted):

To: ...
From: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>
Subject: ...
Message-ID: ...
Date: Thu, 31 Dec 2020 19:28:18 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101
Thunderbird/85.0
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

https://...

From
To: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>
From: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>
...

and then some more such mails, each starting with a line with just "From".

Also the overview file seems to be broken - the subject in the message display does not match the subject in the message list.

All the messages were sent in recent days.

Comment 1

[Jens Müller (:tessarakt)]

The individual message, as far as I can still determine, were also appearing in the Sent folder ...

Comment 2

[Wayne Mery (:wsmwk)]

Is mailnews.send.jsmodule set to true?

The default was set back to false in 85.0b2 https://bugzilla.mozilla.org/show_bug.cgi?id=1680189#c6

Comment 3

[Jens Müller (:tessarakt)]

(In reply to Wayne Mery (:wsmwk) from comment #2)

Is mailnews.send.jsmodule set to true?

The default was set back to false in 85.0b2 https://bugzilla.mozilla.org/show_bug.cgi?id=1680189#c6

It's on the default value false.

Comment 4

[Alfred Peters [:infofrommozilla]]

(In reply to Jens Müller (:tessarakt) from comment #0)

TB version: 85.0b2

https://...

From
To: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>

Just "From " with no additional content?

In Bug 1681487, the MBox seperator was tweaked.
But has the patch even made it into TB85b yet?

Comment 5

[Jens Müller (:tessarakt)]

(In reply to Alfred Peters from comment #4)

(In reply to Jens Müller (:tessarakt) from comment #0)

TB version: 85.0b2

https://...

From
To: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>

Just "From " with no additional content?

Actually, it's "From " with an additional space.

That's probably the reason why in the message display itself it's combined with the next "To:" line (format=flowed rules, if I remember correctly).

The problem has not occurred again since ...

Comment 6

[Wayne Mery (:wsmwk)]

(In reply to Alfred Peters from comment #4)

...
In Bug 1681487, the MBox seperator was tweaked.
But has the patch even made it into TB85b yet?

No, it is not.

Comment 7

[gene smith]

Is each message OK when you read it normally or is the problem only with "show source"?

If you look at the "Sent" mbox file, each message should start with a From delimiter. More typically you will see something like this:

From - Sun Dec 13 22:15:13 2020
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
RealHeader: some info   // This should be the first item visible in "show source" for the message.

as the delimiter between messages. Does you sent folder mbox file have the correct delimiters?

Comment 8

[Jens Müller (:tessarakt)]

(In reply to gene smith from comment #7)

Is each message OK when you read it normally or is the problem only with "show source"?

There is ONE broken message:

• it shows with a subject in the message list that is different from the actual subject shown in the message pane. Also the Date is different.
• When I read the message normally it is also shown wrongly.
• A lot of other messages are "added" at the end of it - apparently all messages added to the Sent folder up to some point, maybe when the Sent folder was compressed.

If you look at the "Sent" mbox file,

I was at first a bit confused which file. But the folder properties tell me it's 228 MB big, which allows me to identify the file.

I have these files on disk:

• Sent.msf, last modified in August 2020, which is 2 kB big.
• Sent-1, last modified yesterday, which is 232714 kB big ==> so that is the file
• Sent-1.msf, last modified today, which is 1470 kB big.

each message should start with a From delimiter. More typically you will see something like this:

From - Sun Dec 13 22:15:13 2020
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
RealHeader: some info   // This should be the first item visible in "show source" for the message.

as the delimiter between messages. Does you sent folder mbox file have the correct delimiters?

One remark: I haven't compressed the folder, so it still contains a lot of messages that have since been moved to a different folder.

I searched backwards for "From".

The messages that is not displayed correctly is the latest (most to the bottom) message with not just "From ":

From - Fri Jan 1 14:18:55 2021
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000

And as already written earlier, the Date: header is different from that: "Date: Thu, 31 Dec 2020 19:28:18 +0100".

Searching further upwards, the "From" delimiters seem to be ok:

The next one upwards is:

From - Fri Jan 1 14:18:55 2021
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000

and then the next one:

From - Fri Jan 1 14:18:55 2021
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000

Comment 9

[gene smith]

From
To: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>
From: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>
...

You showed this in comment 0. Does this show up in the "bad" message in the Sent-1 mbox file? Maybe this is somehow another emails that is an attachment to another email, not sure. Lines in email body starting with "From " are supposed be "escaped" so that that aren't interpreted as a false delimiter and start a new message. Did the message you sent that triggered this contain another messages either inline or as an attachment?

Hard to know exactly what is causing this. Emails as attachments shouldn't cause a problem but I guess it's possible.

You might be able to "fix" the problem by compacting the folder and/or repairing the folder. The repair function is accessed in the folder properties. But this may destroy the evidence of whatever caused this.

Comment 10

[Alfred Peters [:infofrommozilla]]

I was wondering where the separator line without the date came from in the first place. When I delete an email and then call "Compact Folder" all dates are indeed deleted from the separator lines.

{@Gene: Can you tell if this is intended behavior?}

... But here they are then deleted from all lines!

(In reply to Jens Müller (:tessarakt) from comment #8)

The messages that is not displayed correctly is the latest (most to the bottom) message with not just "From ":
From - Fri Jan 1 14:18:55 2021

And as already written earlier, the Date: header is different from that: "Date: Thu, 31 Dec 2020 19:28:18 +0100".

The next one upwards is:
From - Fri Jan 1 14:18:55 2021

and then the next one:
From - Fri Jan 1 14:18:55 2021

Yes, that's remarkable.
Do all separator lines have the same date?

This would mean that a "Repair Folder" was in progress at that time (this restores the date). If this was running at the same time as "Compact Folder", this could be an explanation for the destroyed MBOX.

Comment 11

[Jens Müller (:tessarakt)]

(In reply to Alfred Peters from comment #10)

I was wondering where the separator line without the date came from in the first place. When I delete an email and then call "Compact Folder" all dates are indeed deleted from the separator lines.

{@Gene: Can you tell if this is intended behavior?}

... But here they are then deleted from all lines!

(In reply to Jens Müller (:tessarakt) from comment #8)

The messages that is not displayed correctly is the latest (most to the bottom) message with not just "From ":
From - Fri Jan 1 14:18:55 2021

And as already written earlier, the Date: header is different from that: "Date: Thu, 31 Dec 2020 19:28:18 +0100".

The next one upwards is:
From - Fri Jan 1 14:18:55 2021

and then the next one:
From - Fri Jan 1 14:18:55 2021

Yes, that's remarkable.
Do all separator lines have the same date?

This would mean that a "Repair Folder" was in progress at that time (this restores the date). If this was running at the same time as "Compact Folder", this could be an explanation for the destroyed MBOX.

(In reply to gene smith from comment #9)

From
To: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>
From: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>
...

You showed this in comment 0. Does this show up in the "bad" message in the Sent-1 mbox file? Maybe this is somehow another emails that is an attachment to another email, not sure. Lines in email body starting with "From " are supposed be "escaped" so that that aren't interpreted as a false delimiter and start a new message. Did the message you sent that triggered this contain another messages either inline or as an attachment?

Hard to know exactly what is causing this. Emails as attachments shouldn't cause a problem but I guess it's possible.

You might be able to "fix" the problem by compacting the folder and/or repairing the folder. The repair function is accessed in the folder properties. But this may destroy the evidence of whatever caused this.

(In reply to gene smith from comment #9)

From
To: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>
From: =?UTF-8?Q?Jens_M=c3=bcller?= <[e-mail]>
...

You showed this in comment 0. Does this show up in the "bad" message in the Sent-1 mbox file? Maybe this is somehow another emails that is an attachment to another email, not sure. Lines in email body starting with "From " are supposed be "escaped" so that that aren't interpreted as a false delimiter and start a new message. Did the message you sent that triggered this contain another messages either inline or as an attachment?

No, the broken message (which "contains" other messages) was just one simple line of text with

Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hard to know exactly what is causing this. Emails as attachments shouldn't cause a problem but I guess it's possible.

You might be able to "fix" the problem by compacting the folder and/or repairing the folder. The repair function is accessed in the folder properties. But this may destroy the evidence of whatever caused this.

I made a copy of Sent-1, and will try this next.

Comment 12

[Jens Müller (:tessarakt)]

(In reply to Jens Müller (:tessarakt) from comment #8)

The messages that is not displayed correctly is the latest (most to the bottom) message with not just "From ":
From - Fri Jan 1 14:18:55 2021

And as already written earlier, the Date: header is different from that: "Date: Thu, 31 Dec 2020 19:28:18 +0100".

The next one upwards is:
From - Fri Jan 1 14:18:55 2021

and then the next one:
From - Fri Jan 1 14:18:55 2021

Yes, that's remarkable.
Do all separator lines have the same date?

Actually, no. A bit further up I have

From - Fri Jan 1 14:18:42 2021
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000

(twice)

And then above that again something with just "From " ...

This would mean that a "Repair Folder" was in progress at that time (this restores the date). If this was running at the same time as "Compact Folder", this could be an explanation for the destroyed MBOX.

Interesting ...

Comment 13

[Jens Müller (:tessarakt)]

I now compressed and then repaired, and the broken message is now gone - I had anyway already moved it to a different folder ... It was apparently also downloading mails again from IMAP when while/after repairing.

As written earlier, I made a backup of the original Sent-1, so if there are further questions, I can look into the backup.

(Interestingly, two very old messages showed up again with the current date, because they have a broken date header:

Date: So, 21 Mrz 2010 17:44:02 +0800
Message-ID: <[e-mail]>

Apparently the mail client on my very first HTC Hero smartphone was quite broken, localizing the month names in the Date header ...)

Comment 14

[gene smith]

Alfred wrote:

I was wondering where the separator line without the date came from in the first place. When I delete an email and then call "Compact Folder" all dates are indeed deleted from the separator lines.

{@Gene: Can you tell if this is intended behavior?}

... But here they are then deleted from all lines!

I don't know if it's intended. However, I do know that the dash and date are really "don't care" fields that aren't used. As long as the line starts with "From " it is interpreted as the start of a new message when the mbox file is parsed.

(In reply to Jens Müller (:tessarakt) from comment #13)

I now compressed and then repaired, and the broken message is now gone - I had anyway already moved it to a different folder ... It was apparently also downloading mails again from IMAP when while/after repairing.

Yes, repair will completely resync with the server so everything tends to be re-downloaded.

As written earlier, I made a backup of the original Sent-1, so if there are further questions, I can look into the backup.

Not sure if it would help but I might need to see the full Sent-1 file if it is possible to redact any private stuff (and the size isn't too big).

(Interestingly, two very old messages showed up again with the current date, because they have a broken date header:

Date: So, 21 Mrz 2010 17:44:02 +0800
Message-ID: <[e-mail]>

Apparently the mail client on my very first HTC Hero smartphone was quite broken, localizing the month names in the Date header ...)

Yes, that would probably confuse tb.

Comment 15

[Jens Müller (:tessarakt)]

(In reply to gene smith from comment #14)

(In reply to Jens Müller (:tessarakt) from comment #13)

I now compressed and then repaired, and the broken message is now gone - I had anyway already moved it to a different folder ...

To clarify this a bit:

• The message list entry that "pointed" to the "broken" message now leads to a message that indeed matches what is shown in the message list.
• The "broken" message had already been moved to another folder.

So it looks to me as if one overview entry got messed up and pointed to an already deleted message.

It was apparently also downloading mails again from IMAP when while/after repairing.

Yes, repair will completely resync with the server so everything tends to be re-downloaded.

As written earlier, I made a backup of the original Sent-1, so if there are further questions, I can look into the backup.

Not sure if it would help but I might need to see the full Sent-1 file if it is possible to redact any private stuff (and the size isn't too big).

It's very big (200 MB), and contains lot of private stuff.

And I start to think that it might rather be caused by a broken .msf file - which I did not backup unfortunately.

Sorry, it seems I won't be able to help much more here ...

Comment 16

[Wayne Mery (:wsmwk)]

It sounds like we aren't going to find the cause of the folder corruption?