339143 - Content of POP account messages that passed through filter(s) gets mangled very badly

Status: RESOLVED INCOMPLETE

(Thunderbird :: General, defect)

Description

[Xander]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3

This one is hard to describe.
I am running an IBM Thinkpad R31 with Windows XP Home SP2 with Thunderbird version 1.5.0.2 (20060308).
The problem occurred right after reviving the machine from Hibernation, and bringing Thunderbird up from the taskbar, after which it started to receive email from all accounts.
What happened was, that all mail was transferred from the IMAP and POP servers into Thunderbird, and passed through some message filters I had set earlier this week to sort messages into specific folders.
What also happened was some legitimate messages were marked as Junk.. Upon examination of these messages, I observed the headers of these messages were not matching the content of the messages, and the "From" address was mangled to something similar to "zxckldsjf aopiulkfj" (Hovering mouse over mangled address reveils valid email address, though this address might -not- belong there but in another message!)
Further investigation revealed that ALL messages that were retrieved, had their headers and body mixed up with each other, and some headers mangled, some messages were duplicated several times with different mixed-up content.
All affected messages are completely mangled and have become human-unreadable, although Thunderbird can open them just fine. One message could not be saved without renaming the filename in the Save-As dialog because of the colon character not being stripped from the filename by Thunderbird, other messages with the colon character in the title were processed properly upon Save-As.

What I suspect has happened is; that while the messages were being retrieved, the Junk Mail Controls interfered with my message filters, or possibly that another factor is into play where one of the actual, real junk messages with invalid hidden content made the Junk Mail Control trip and mess up the entire retrieve/junk/filter checking process.

During the mangling process, Thunderbird did -not- crash or display visible breakage in the GUI. No error messages were produced.
Messages that were marked as Junk could be marked as Not Junk, but would not be automatically moved back to Inbox.
Closing and re-opening Thunderbird did not restore the messages to their original state.

Reproducible: Didn't try

Steps to Reproduce:
This is a -possible- way of reproducing the problem; I cannot however test it without losing confidence in the safety of my mail, I need them to be readable.

1. Send yourself emails from Firefox with the "File > Send Link..." option through Microsoft Outlook 2002 and 2003 (I have 2002 on POP at home, 2003 on Exchange 2003 at work) to one of the POP mail accounts.
2. Revive laptop from Hibernation mode and maximize Thunderbird, wait for it to start retrieving mail automatically after X minutes (as set in account preferences)
3. Check received and processed messages for correctness.

It is also possible for my Outlook 2002 client to be at fault, as it will -not- send a message unless you create and send the exact same message -twice-. A 'dead' duplicate remains in the Outbox indefinitly, the copy that gets sent -may- contain invalid code, I can't confirm this, but it's something that I noticed happening yesterday evening (May 23).
Actual Results:  
All mail received through the POP account(s) has been mangled and mixed together, and split back into what looks like separate, valid messages, although header and body are no longer correct or readable. Message header "From" field mangled to invalid email address.

Expected Results:  
Retrieved messages should not have been mangled to death.
Perhaps an error message such as "Thunderbird encountered an error processing [message]. Examine/Continue/Skip/Abort?" might have been useful, if appliccable.

I am using Thunderbird version 1.5.0.2 (20060308) with the Enigmail extension version 0.94.0.0 (20060110), enabled and configured on ALL (POP and SSL IMAP) accounts with a single identity set up for all accounts.
The laptop is configured with Thunderbird as the default email client, Enigmail was installed and the accounts + messages were imported afterwards from Outlook Express version 6.00.2900.2180 (xpsp_sp2_rtm.040803-2185) into the empty Thunderbird client, then account settings were checked and updated for correctness.
After several days of using the new Thunderbird setup, added the following message filters under "Tools > Message Filters...":

--- Filters for: pop.cablewanadoo.nl ---
#1. Garfield; Match any of the following:
"Subject contains 'And now, a word from Garfield'"
"From contains garfield.com"
Perform these actions:
"Move Message to Garfield Comics on Local Folders"

#2. Mandrake; Match any of the following:
"From contains Mandrake"
"From contains Mandriva Team"
Perform these actions:
"Move Message to Mandrake on Local Folders"

#3. PayPal; Match all of the following:
"From contains PayPal"
"To contains sander-overvliet"
Perform these actions:
"Move Message to PayPal on Local Folders"

#4. Openpsion; Match any of the following:
"Subject contains [openpsion]"
Perform these actions:
"Move Message to PsiLinux @ Sourceforge.net on Local Folders"

#5. Router Logs; Match any of the following:
"Subject contains LogFull"
"Subject contains AttackLog"
Perform these actions:
Move Message to Router Logs on Local Folders"

"Run selected filter(s) on:" is set to Inbox of Local Folders, although i'm not sure if that's relevant for autonomous operation of the filters.
Unfortunately I had the Filter Log option turned off, so I can't provide a log of the disaster :/

But I can provide the mangled messages for review if required.

Comment 1

[Xander]

Attachment: Samples of mangled messages saved from Thunderbird

ZIP File was scanned with ClamWin Free Antivirus, report below for correctness:

--------------------------------------
Scan started: Wed May 24 22:02:25 2006


-- summary --
Known viruses: 56380
Engine version: 0.88.2
Scanned directories: 0
Scanned files: 1
Infected files: 0

Data scanned: 0.18 MB
Time: 16.804 sec (0 m 16 s)
--------------------------------------
Completed
--------------------------------------

Comment 2

[Wayne Mery (:wsmwk)]

Xander, reporter writes: 

I do recall having some problems while away on summer holiday, using a
wireles GPRS connection via Bluetooth once; but I need to do a thorough
search for that month's mail to be sure there weren't any occurrences.

The bugreport I filed back in 2006 was from an installation running on a
laptop has since been nuked, but I moved the mailbox store from that
machine to the one I am using right now, a new laptop.
Same mailbox store, Thunderbird version 2.0.0.6 (20070728).

Will report back when I have completed the search

Comment 3

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Does the issue still occur in the latest 2.0.0.x / trunk nightlies?

Comment 4

[Xander]

I can no longer confirm the bug as of branch 2.0.0.xx (right up to 2.0.0.14 - 20080421), and I have not been able to find any more corrupted emails in my mailboxes since.
Also unfortunately I have not been able to do many tests between the reported version and this latest version due to the lack of signal availability while on the road, at present there seems to be a war between GSM/GPRS and 3G/UMTS among providers causing a clash in availability.. I can't make any mobile connection at all right now to test things on. :(
Sorry this was dragged on so long.
How does one go about closing a bug correctly?

Comment 5

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Resolving incomplete due to lack of information on how to test on, would be an appropriate resolution.