Closed Bug 360409 Opened 18 years ago Closed 18 years ago

mails with messed up references seem to disappear

Categories

(Thunderbird :: General, defect)

Product:
Thunderbird
Component:
General
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ksoule, Assigned: Bienvenu)

References

Details

(Keywords: fixed1.8.0.9, regression, verified1.8.1.1)

Attachments

(2 files, 1 obsolete file)

Test5 folder (6 problematic messages), plus related .msf file
7.48 KB, application/x-zip-compressed
Details
do a one time detection of this case, and regenerate summary files that appear to have this problem.
4.09 KB, patch
mscott
: superreview+
jay
: approval1.8.0.9+
Details | Diff | Splinter Review 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0
Build Identifier: Thunderbird/1.5.0.8 (as well as the Trunk nightly builds)

After upgrading from 1.5.0.7 to 1.5.0.8, I determined that *certain* new mail (specifically, new mail sent by the X-Mailer "Chilkat Software Inc (http://www.chilkatsoft.com)") did not appear in either the Inbox, or in the subfolder where it was sent by my filter (even though the subfolder had an "arrow" indicating new mail).  I also found that mail previously (pre-upgrade) received from a Chilkat X-Mailer *or* from an X-Mailer of either "JMail 4.4 by Dimac" or "JMail 4.5 by Dimac" was no longer visible in 1.5.0.8 (I haven't received any mail sent via JMail post-upgrade, so I haven't been able to test that).  So far, I have only seen the problem affect mail sent from one of these 2 X-Mailers, though my testing is by no means exhaustive.

It seems that the crux of the problem is an inability to process the headers of e-mail generated by these 2 X-Mailers, and that inability results in a corrupt *.msf file (which then causes the messages to be invisible and/or lost).

Reproducible: Always

Steps to Reproduce:
1   Close TB 1.5.0.8.
2.  Copy the attached "test5" file (created in 1.5.0.7, by copying 6 messages into a new folder called "Test5") into the Inbox.sbd folder.  The Test5 file contains 6 e-mail messages, 3 generated by the JMail X-Mailer, and 3 generated by the Chilkat X-Mailer. FOR THIS TEST, DO NOT COPY THE INCLUDED TEST5.MSF FILE.
3.  Restart Thunderbird (which will cause generation of a new .msf file), and navigate to the Test5 folder, under Inbox.
4.  Look to see how many messages appear (there should be 6; I see 1).
5.  Attempt to copy/move the file(s) in the Test 5 folder to another folder.

Actual Results:  
Although the Test5 folder should display 6 messages, it displays only 1.  Attempts to copy/move that message result in a perpetual hourglass.  There is no way to access/see the other 5 messages in the file/folder.  There is no way to "manage" any of the messages in the file/folder.  If the folder is compacted, the missing messages will be permanently lost.

Expected Results:  
I expect 1.5.0.8 to behave in the same manner as 1.5.0.7.  After restarting TB, I expect to see all 6 messages in the Test5 folder, and I expect to be able to copy/move them easily and without incident.  I do not expect compaction to cause mail database corruption.

The attached file (TB1.5.0.7.zip) contain the Test5 file (6 messages), and the related Test5.msf file ... both created in TB 1.5.0.7.

As a reference point, copy *both* files into Inbox.sbd in TB 1.5.0.7.  Restart TB.   You will see 6 messages, and they can be copied, moved, managed, etc. as one would expect.

Now copy both files into Inbox.sbd in TB 1.5.0.8.  Restart TB.  You will see the same 6 messages.  However, if you attempt to copy/move all 6 of them, it will appear that only 1 message was copied/moved (and the operation will generate a perpetual hourglass).  If you look at the underlying data files, you will see that all of the selected messages were, in fact, copied, but that the *.msf file in the destination folder is corrupt, so it appears that but a single message was copied (and, if a "move" operation is performed, the files in the source folder will *not* be deleted).  If you delete the *.msf file in the Test5 folder, and allow TB 1.5.0.8 to regenerate it, only a single message will appear in the folder (though if you look at the data file, you will see that it contains all 6 messages).

If you copy the Test5.msf file generated in TB 1.5.0.8 into the Inbox.sbd folder in 1.5.0.7 (overwriting the equivalent file generated by 1.5.0.7), and then restart TB, you will see but a single message in the folder (though the data file contains all 6).  If you then delete the Test5.msf file in TB 1.5.0.7, and restart TB, allowing TB to regenerate the file, you will see all 6 messages again.

More details of the problem, together with comments/experiences of others, can be found in the thread: http://forums.mozillazine.org/viewtopic.php?t=485752 I created the topic, and the analysis (a bit rough at first) becomes much more focused as the testing progresses.
The attachment contains 2 files, (i) Test5 ... a message folder with 6 messages, 3 generated by the Chilket X-Mailer, and 3 generated by the JMail X-Mailer, and (ii) Test5.msf, the related .msf file.  Both files were created in TB 1.5.0.7, by copying the 6 selected messages to a new folder called "Test5".
My interim solution for this bug has been to reinstall TB 1.5.0.7, and disable all automatic program upgrades.
Version: unspecified → 1.5
As clarification, not only does this problem occur in TB 1.5.0.8, I have also been able to reproduce it in TB version 2.0, beta 1 (20061111).
Flags: blocking1.8.0.9?
Flags: blocking-thunderbird2?
Confirmed on Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20061112 Thunderbird/3.0a1 ID:2006111204, branch too.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: clean-report, regression
OS: Windows XP → All
Hardware: PC → All
*** Bug 360340 has been marked as a duplicate of this bug. ***
cc'ing David in case this was caused by bug 351685 and bug
332883 on the security branch.
*** Bug 360530 has been marked as a duplicate of this bug. ***
All the messages that do not show up have Message-ID, References, Resent-Message-ID set to the same value, like
Message-ID: <CRID_7bbabaab57ba41e69a354d392b2d6e74@centercode.com>
References: <CRID_7bbabaab57ba41e69a354d392b2d6e74@centercode.com>
Resent-Message-ID: <CRID_7bbabaab57ba41e69a354d392b2d6e74@centercode.com>

So it looks like a messed up threading issue. 

Doing a local backout of attachment 239252 [details] [diff] [review] in bug 332883 fixed it for me.

Bumping severity, seems like this is affecting a lot of people (from discussion e.g. in http://forums.mozillazine.org/viewtopic.php?t=487248). And it really is rather nasty.
Severity: critical → blocker
Depends on: 332883
Summary: Mail received from Chilkat and JMail X-Mailers is lost or disappears → mails with messed up references seem to disappear
I agree that it's nasty, but it's not a blocker...most users in normal usage won't get multiple messages with the same message-id...I've been looking at this but I don't think I'll have a solution before  get back from vacation.
Severity: blocker → critical
Maybe not "most" users, but seems to be happening more than you would think from the discussions about it.
Assignee: mscott → bienvenu
Blocks: 332883
Severity: critical → blocker
No longer depends on: 332883
Flags: blocking1.8.0.9? → blocking1.8.0.9+
Regarding David's comment about "most users", when he reduced the severity from "blocker" to "critical".

I didn't report this bug as an intellectual exercise, or a contrived test case.  I reported it because one of my mail folders contained 343 messages in 1.5.0.7, and the number was immediately reduced to 271 (a loss of 72 messages) upon installing 1.5.0.8.  Further, all *new* messages from a particular major software house where immediately lost in 1.5.0.8 ... without being displayed in TB at all (and permanently lost if you compact the folders ... which I did).

I was able to debug this issue because all of my mail now traverses GMail before going to TB, and GMail retains an archive.  Fortunately, my drive is imaged every night, so I was also able to recover all of the older lost messages that didn't go through GMail.

How nasty is it, and should it be a blocker?  I guess that depends upon whether it is your mail that is being lost ... or mine.
No longer blocks: 332883
Severity: blocker → critical
Depends on: 332883
Flags: blocking1.8.0.9+ → blocking1.8.0.9?
I had the same problem.. i could still see the affected messages when viewing the Inbox file using a text editor, but not in TB. Tried compacting, and that didn't help, neither did deleting the msf file. Below you can see the header for one of the affected messages, directly copied from the 'Inbox' file. I'm afraid compacting actually deleted the messages, because after downgrading to TB .7, i only got back a few of them that were dated today, i think from after my last 'compact'.

In other words: this makes TB completely useless, and i can't believe that this version is still being distributed, because it must be hurting many people. I had messages disappear almost every time i got new messages, probably at least 30% of them. Not sure if they were all spam, because i only started really investigating after compacting to see if this would fix it, and that process seems to have deleted most of them.

I just hope it has something to do with spam filtering, because in that case at least i know that i probably didn't lose any important messages. Can anyone confirm this?

Anyway, when i get Vista, i'll be sure not to use TB again - and i'm sure many others will feel the same way. I've been using it for years, but it's unacceptable to lose mail, and i completely lost my confidence with TB, especially since it's still being distributed, when it clearly shouldn't be. The bug report makes it look like this is some isolated issue having to do with a few obscure mail agents, which obviously isn't the case (look below, X-Mailer: Microsoft Outlook Express 6.00.2900.2869)

Good luck everyone, and at least downgrade for now, as a quick fix! (and switch off Automatic updates)

If this is not a "blocker", i don't know what is... (you'll feel the same way if you're waiting for an important email, your mail program says "new mail!", and you discover an empty inbox...)

-- Niek


From - Mon Nov 13 17:35:55 2006
X-Account-Key: account2
X-UIDL: UID15741-1103787658
X-Mozilla-Status: 0000
X-Mozilla-Status2: 00000000
Return-Path: <ljveeflbi>
Delivered-To: 3-niek@xxxmydomainxxx.com
Received: (qmail 24804 invoked from network); 13 Nov 2006 16:27:05 -0000
Received: from dsl-189-166-104-20.prod-infinitum.com.mx (189.166.104.20)
by u15175353.onlinehome-server.com with SMTP; 13 Nov 2006 16:27:05 -0000
Message-ID: <000601c7073f>
From: "Were" <ljveeflbi>
To: niek@xxxmydomainxxx.com
References: <000601c7073f>
Subject: Re: glory World Service
Date: Mon, 13 Nov 2006 10:16:14 -0600
MIME-Version: 1.0
Content-Type: multipart/related;
type="multipart/alternative";
boundary="----=_NextPart_000_0004_01C7070C.BFB6D1E0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.2869
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2962
Attached patch proposed fix (obsolete) — — Splinter Review 
ignore references of headers that claim to be a reply to themselves.
Attachment #245484 - Flags: superreview?(mscott)
If this bug is technically a duplicate of Bug 360252, I suggest marking that bug a duplicate of this, as this entry has far more information.

Oh, and add one more person is affected by this and is currently downgrading to 1.5.0.7
I'm sorry for that regression, and I fixed it as quickly as I could while I was on vacation. Compacting was not related to this bug, since compaction ignores message threading...

I did not realize that Outlook Express was so broken...

If you sort by date, or any other column, and are non-threaded, you will see all your messages (you have to be in a flat sort when you open the folder, so if you're threaded, you have to sort by date, and then click away from the folder, and back).
*** Bug 360252 has been marked as a duplicate of this bug. ***
Re: Comment 9, suggesting that the problem is the result of Message-ID, References, Resent-Message-ID set to the same value (and someone changed the description of the bug accordingly), and the proposed fix in Comment 14.  Please look at Comment 13, where there is but a Message-ID (i.e., the message is not purporting to be a reply to itself).

I'm not a software developer, but I am able to discern the factual inconsistency, and therefore question the conclusion set forth in Comment 9, and the related change in the description of this bug.

Of course, there may be more than a single issue involved here.
From comment 13:

Message-ID: <000601c7073f>
From: "Were" <ljveeflbi>
To: niek@xxxmydomainxxx.com
References: <000601c7073f>

Note that the Message-ID and the References header are precisely the same, which means that the message says it's a reply to itself. Does that make sense?
David, what do you mean with "If you sort by date, or any other column, and are non-threaded, you will see all your messages". My messages weren't threaded, and they were sorted by date, and i could not see all my messages. (View -> Sort by -> Date / Descending / Unthreaded). The only thing i could do to make them show up was downgrade to 1.5.0.7 and delete the msf file.

By the way: why is this bug's status still "NEW"?

Regards,
-- Niek
Re: David's Comment 19 ... my bad, I just missed it (the References).  Of course, I understand.
With the attached test folder, if you're threaded, you only see one message. If you sort by date, then click away and click back, you'll see all six messages again.

If you were ever threaded in the folder, you'll miss messages until you open the folder such that it opens non-threaded (i.e., it was sorted, flat, when you closed the folder, before you re-opened it). That's the case for this test case, at least. Perhaps other test cases behave differently - are you saying the folder was never threaded? In that case, we might want a new test case that demonstrates that...
Status: NEW → ASSIGNED
Example in Bug 360252 has "Exhibit B" never showing.  I always work unthreaded (View > Sort By > Unthreaded) and sorted by date (View > Sort By > Date).  Descending, for what it's worth.

I have now downgraded to 1.5.0.7 (can't very well have e-mail be there but not showing up!) so I can't do any further testing.
David ... I created the test folder, and I've never used TB in threaded mode (by default, I sort by date).
By the way, again: i'm pretty sure that compacting deleted the invisible messages, even after returning to 1.5.0.7. The only messages i got back were the few (3) that i received after my last 'compact' action.

Please read the following threads for some more user experiences, also referring to the compacting effect:

http://forums.mozillazine.org/viewtopic.php?t=487248

http://forums.mozillazine.org/viewtopic.php?p=2594779

These two threads seem to be *by far* the most active ones in their respective categories over the last couple of days, suggesting the problem is far bigger than some of you would want to think. Which again warrants a "blocker" rating, i would hope...

As for being threaded or not: as far as i remember, my inbox was never threaded. But i might have tested this option a year ago or so, i don't know. At least i haven't made any changes to the sorting/threading in the last couple of months: it has always been the way i described it (View ->
Sort by -> Date / Descending / Unthreaded) although i sometimes click on 'date' again to sort the other way, or click on the View -> Unread tab when i get spam with an old date and i want to quickly find and delete it. 

Regards,
-- Niek

Sorry, I thought Dan had changed it back to blocker, my mistake. Marking as blocker.

I used the test case on the trunk/2.0 builds - perhaps the 1.5.0.8 build behaves differently...but as the original test case describes, you
Severity: critical → blocker
We try to say that the regression bug "blocks" the original bug, that is, correctly fixing the original bug requires fixing this bug. Helps in keeping track should we later want to port the original bug to an old branch.
Blocks: 332883
No longer depends on: 332883
Flags: blocking1.8.0.9? → blocking1.8.0.9+
Attachment #245484 - Flags: superreview?(mscott) → superreview+
I did not mean to set the severity to "Blocker", which is reserved for bugs that stop development or testing (https://bugzilla.mozilla.org/page.cgi?id=fields.html#bug_severity). I don't think this bug qualifies, critical is more accurate.

However it does appear to be a significant regression and should "block" the next release (which is a flag, not a severity).
Severity: blocker → critical
Ok - i just meant to say that it should "block" the next release. In my opinion, it should even block the 1.5.0.8 release: this will affect many people. Just look at how these are by far the most popular items in both the Mozilla Bugs and the Support categories (see links i provided earlier). And those are just the people who were **** enough to Google the problem and write about it - most users probably don't even notice, since it happens almost invisibly (you never see the mails come in). I only noticed it after a few days, when i happened to pay attention to how many mails it was downloading, and how many it actually displayed (and considering usually at least 50% is spam, you aren't always very surprised if you don't see all the mails that are downloaded because they're automatically redirected to Junk). After realizing it, i noticed it happened almost every time i downloaded a few mails...
fixed on trunk and branch - will nominate the patch for 1.5.0.9 in just a sec.
Status: ASSIGNED → RESOLVED
Closed: 18 years ago
Keywords: fixed1.8.1
Resolution: --- → FIXED
Comment on attachment 245484 [details] [diff] [review]
proposed fix

requesting 1.8.0.9 approval
Attachment #245484 - Flags: approval1.8.0.9?
To stave off any potential confusion, marking it fixed means it's fixed on the trunk of our source code repository, not that the fix is in an available build yet.
Ok, so does this patch fix the issue of the disappearing messages? Your patch seems to be related mostly to threading issues (?), but please note again that it doesn't seem to have much to do with that. At least, in my case, i never used threading, and the messages still disappeared. Also clearly, this should be a bug that was introduced in 1.5.0.8, and wasn't there in earlier versions (at least not any recent ones).
This bug manifests itself differently on the trunk and 2.0 branch, vs the 1.5.0.x branch. Unfortunately, it's much worse on the 1.5.0.x branch - on that branch, when you get multiple messages that have the same message id, and the same reference id, and claim to be replies to themselves, the subsequent msg hdrs don't actually get added to the db. (on the trunk, they get added, but don't get displayed when in threaded view).

So, for the 1.5.0.x branch, this fix only fixes the problem for new incoming messages, or if you delete the .msf file. To fix it for the 1.5.0.x branch, we'd need to force the regeneration of all local folder .msf files.  This is a big yuck. But I can't think of any other way around it, other than perhaps to detect the case where an existing msg hdr claims to be a reply to itself, and only regenerate the db in that case.  I'll try to come up with a patch for that.
The bug may manifest itself differently (from a technical standpoint) on the trunk, branches, etc.  However, before I filed the bug, I tested it on the 2.0 nightly, and I got the same results that I got in 1.5.0.8 (of the 6 test messages, when the nightly generated the .msf file, only 1 message displayed ... the first one).  The important point I want to make is that I NEVER used/tested threaded view (I only have 6 messages in the test file ... and they aren't threaded anyway), so I am a bit troubled by the distinction between flat and threaded view, and the subsequent attribution of bug manifestation to threaded view.
Kent, I'm not sure - when I tried it with a 2.0 build, I saw the six messages when in flat view, and only one when in threaded view. I can try it again later, but it's really not that important right now - if you delete your .msf file, and open the folder with tomorrow's 2.0 nightly build, it should work fine. The really critical thing is fixing 1.5.0.8 builds right now, and somehow regenerating .msf files when we detect this problem.
... as for 1.5.0.8 ... if you delete all of the .msf files, you get all of the messages back (assuming they haven't been lost permanently via compaction), but you also lose all of the folder-by-folder sort customizations.  That's gonna be hella confusing to peeps like my mom, me thinks (though I don't have an alternate solution).
If I can detect the corruption and mark the summary file invalid programmatically, we might be able to maintain the sort info, etc.
*** Bug 360643 has been marked as a duplicate of this bug. ***
David, just out of curiosity, and to get a better idea of what i (and others) lost after compacting: do you think these 'self-replying' messages are mostly spam? Or do you know of any legitimate email programs / websites (Outlook, Yahoo mail, etc.) that create messages like this? Is this even legal according to the RFC's in question? The issue would be slightly less damaging if you know that the disappearing messages are mainly spam and/or generated by some poorly written mailer.

Thanks.
Niek, it's definitely illegal. I had never encountered messages like this before. Some are definitely spam (I suspect that having a reference header decreases the spam score with some spam filters, so they just stick one in, and the message-id happens to be handy).

However, in the Test5 folder attachment, there are several messages with the same problem that don't seem to be spam. They have a "Resent-From" header, and all the message-ids are from centercode.com - this makes me think there's some server-side software rewriting the message-ids - I don't think it's the original sending program generating the bad message-id+reference combination.
When I created the Test5 folder, I selected 6 messages out of a total of 72 that "disappeared" from one of my folders when I upgraded to 1.5.0.8 from 1.5.0.7.  None was spam.  All were received from a very large Silicon Valley-based software vendor in connection with beta testing (mass mails to all of the beta group, or personalized e-mails generated by their bug reporting system).
I can confirm Kent's finding (comment #42). No spam, just beta test group messages.

Essentially two things were happening: 

1) In builds before 1.5.0.8, (1.5.0.7 for example), using the sort by thread command in a folder with messages sent using the Chilkat X-mailer (from another large California-based software company for beta testing) resulted in all but the first message disappearing.  This could only be fixed by deleting the relevant .msf file

2) In build 1.5.0.8, all messages moved to the folder using a filter would disappear. [I did not leave any of these messages in the Inbox so I can't confirm them missing there also].  Removing the .msf file would not reconstitute the message index list.

ie the problems (probably the same problem) appeared to become more extreme with 1.5.0.8

I didn't compact the folder with the hidden messages, so I lost no messages, but I cannot upgrade to 1.5.0.8 or later unless this bug is killed because I need to read the incoming list messages.  Other users on the same beta test list as me have reported the same problem with the list messages.  Worked around by reinstalling 1.5.0.7.

DN
this patch makes it so we'll do a one time run through of the msg hdrs in a db, and if we find messages with the reference header set to the message-id, we'll regenerate the db.
Attachment #245484 - Attachment is obsolete: true
Attachment #245652 - Flags: superreview?(mscott)
Attachment #245484 - Flags: approval1.8.0.9?
this will deal with missing messages, if the user hasn't compacted folders...
will the "one time detection" occur only if the upgrade is from 1.5.0.8, or will it run on any installed version of TB?

will any user preferences (such as sort order) be captured and programmed into the newly-generated .msf file?
any installed version of TB that has the patch in it will run the run-time check, if it hasn't been run before. The one-time-ness is stored in the .msf files, and new msf files are created knowing they don't need to do the one time check.

Sort info should be maintained...
This shows my ignornace of how TB works, but I presume it's necessary to await an updated version of TB installer with the patch applied?  Or is there some way to apply the patch file manually?
This bug affects SeaMonkey as well, including SeaMonkey 1.1b.  It is not present in the latest 1.1 nightlies, though, due to the checkin for the patch in comment #14 (Tested using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061115 SeaMonkey/1.1b).
Comment on attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

I'd really like to get this onto the trunk soon, in order to get it into 1.5.0.9, so switching review to Seth.
Attachment #245652 - Flags: superreview?(mscott) → superreview?(sspitzer)
Is there an estimated time on getting this fix made into a .exe file?  Is there a way to escalate this?  I have been without email for a week now (won't move down to 1.5.0.7 as there is a security bug in it per 1.5.0.8 release notes).  Thanks much!
I need to get it reviewed; then I need to get approval to check it in; then it needs to be released in a build. After I check it in, you could pull a nightly 1.5.0.9 build before it's actually released...the 1.5.0.9 release is supposed to be in December, afaik. 
Sorry, but i don't understand why you guys are waiting until December for such a critical problem. In the mean time, most of your users will have emails disappear (probably without even realizing until their contacts get upset at them because they haven't replied to something), and if they adhere to TB's suggested 'good practice' of compacting their inbox on a 'regular basis', this means that those emails will be lost forever. This should be totally unacceptable, and on equal footing (if not higher) with whatever security updates you may distribute from time to time. If i were still using 1.5.0.8, and i found out a month from now that you guys knew about this problem for over a month, and i happened to have lost several important emails, i would be seriously **** off and sure never to use TB again. If this leaks to the media, i'm sure you'll get a lot of seriously bad press...
I'm inclined to agree with Niek.  This bug has the potential to do TB a lot of damage and the patched version should be released outside the standard sequence eg as 1.5.0.8a
Comment on attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

switch review back to Scott
Attachment #245652 - Flags: superreview?(sspitzer) → superreview?(mscott)
Comment on attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

I wonder if we should check the error code before checking the return value:

+          if (msgHdr && NS_SUCCEEDED(err))

Looks good to me.
Attachment #245652 - Flags: superreview?(mscott) → superreview+
Comment on attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

this landed on the trunk and 2.0 branch today.

I also looked through a lot of my old mail and didn't find any messages like this, except for a test case folder for the bug that caused this regression. So I suspect that most users are not running into this.
Attachment #245652 - Flags: approval1.8.0.9?
And for another anecdotal case: I would be hit by this issue every. single. day.

Does it really matter if 100,000 , 10,000, or 1,000 users are hit by this?  An e-mail application that fails to show some e-mail, and worse yet -deletes- it without chance of recovery after the user performs an operation (compacting) that is recommended by the authors?  Unacceptable.

Perhaps the powers that be do indeed deem this insignificant enough to warrant an immediate new release - but perhaps one of the following could at least take place:
- 1.5.0.8 pulled
- a warning placed on the download page
- a warning issued through the automatic updates (is this possible?)
- an advisory issued through select media; Slashdot, ZDnet, the usual.

This problem should, at the very least, not be silently lingering in this bug report (and its dupes) as if nothing is wrong, while users may be missing and completely losing e-mails, for no good reason, every single day.  It's bad enough that there's a whole page on "Missing Mails" as it is (why should mail -ever- go missing?), it's much worse that it does not address this issue.
( I notice it's a wiki - if anybody has editing rights; http://kb.mozillazine.org/Disappearing_mail )

Just my 2cts - sticking with 1.5.0.7 here.. I'll take my chances with the vulnerabilities.
I was alerted to this problem by my wife who noticed that the number of emails being downloaded appeared to be different to the number of new messages received. Messages coming from a Yahoo Groups! mail alias were being lost. She knows how to google to find the problem but was unable to resolve it (downgrading to 1.5.0.7 does require some skill and, more importantly, confidence).

So I believe that this bug could be affecting anyone using Yahoo Groups! and a lot of them may not realise. The impact of this bug could be devastating to the Thunderbird project and I vote for getting a fix out as soon as possible.

Please let me know what the proper means of escallation is if a person would like to see this fix released out-of-schedule.  Also, could you please list the name and email (I would have to open a yahoo / gmail account though) or contact method of the person who has the authority to release this fix out-of-schedule.  Perhaps they are have not been made adequately aware of the issue.  Thanks.
Can you attach the headers of a yahoo groups message that disappeared? My yahoo groups messages look fine.

Cc'ing Jay to see if there's any way we can get an update out there sooner.
Keywords: fixed1.8.1 → fixed1.8.1.1
Comment on attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

Approved for 1.8.0 branch.  a=jay for drivers, please land this asap.
Attachment #245652 - Flags: approval1.8.0.9? → approval1.8.0.9+
Keywords: fixed1.8.0.9
Yahoo is sending an empty row (line) in the header,
mailman does bounce replies to the list when the send
from yahoo.
Are your shure that it is a problem with the email client
and not a bug made by yahoo? 
Builds which contain this fix are available here:
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-mozilla1.8.0/
Ignore the version number, which hasn't been updated yet, just look at the date. The fix was checked in on the relevant 1.8.0 branch on 2006-11-21 13:04 PST. So any builds made after that contain the fix.
Please see (and fix) bug #295208 , which is a similar or related problem.
Well guys nice to see this fixed, but when will there be an update for 1.5.0.8 ?
You're not waiting till december, right ?
-Jose
.. i'm afraid it'll be in December. Unfortunately Mozilla isn't giving high priority to this issue, so it's probably time for all of us to look for a new mail client. Or step to the press and let them know - i'm pretty sure that would also speed up the fix :)
a well-known german press wrote in its online-news about this bug...i follow my previous posters there is no much time for the mozilla-team to fix this problem without loosing many users....this bug can cost thunderbirds image, i'm sure. Nobody wants to use a mailclient with the fear loosing his mails...

http://www.heise.de/newsticker/meldung/81529

btw. is there a german fix available?

regards frank
(In reply to comment #68)
>
> btw. is there a german fix available?

Hi Frank

the fix is downgrading to TB 1.5.0.7 and deleting of all .msf files in your mailbox. TB will create them new and the hidden mails re-appear (as long you have not compacted your mail folders). Make a backup of all you mail database before doing this!

Here you can get the older TB:
http://releases.mozilla.org/pub/mozilla.org/thunderbird/releases/
What odd for me is that neither deleting the .msf files and downgrading to 1.5.0.7 or using the nightly build brought back the non-spam messages in my Inbox.  However the non-spam messages are still in the Inbox file itself, as I can see them by using mbox-viewer from http://sourceforge.net/projects/mbox-viewer to look at the Inbox file.  Is anyone else experiencing a similar situation?

Is there some other file I need to delete besides the .msf files?  For me, several hundred megs of non-spam still do not show up.  Thanks.
Re: Comment #68

You might try ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2006-11-25-12-mozilla1.8.0-l10n/thunderbird-1.5.0.8.de.win32.installer.exe

The time of the build indicates that it was compiled after the patch landed (even though it still says 1.5.0.8).

Of course, back up your mail data first, and then uninstall your present TB, and install this one (i.e., don't install this on top of your existing installation).
I followed your directions exactly but the non-spam still does not show up.  Thanks for giving me something to try though.
I found a way to bring the missing non-spam back:
qgrep -v -X "^X-Mozilla-" Inbox

Qgrep is a free downloadable Microsoft utility similar to the UNIX "grep" command that with the command line arguments above will remove the extra mail header lines.

The darn thing is that I will have to run this manually on all my mail files to make sure messages haven't disappeared that I don't know about.  But, at least I will have my old email messages back :)
Correction:
qgrep -v -X "^X-Mozilla-" Inbox > Inbox2
move /Y Inbox2 Inbox
OK, folks, I can live with the December update, since I haven't compacted folders for a year or so and I have many folders with >300MB and TB is running smoothly.

But I'd really appreciate a procedure on how to proceed when the update is out:
* just  update ?
* delete the MSF files ? What do I lose by deleting the MSF files ?
* something else ?

Thanks,
Jose.
most likely, just update - we do a one-time check of your .msf file to see if you have at least one instance of a message with identical message-id and reference headers - if so, we regenerate the .msf file automatically. The idea is that if you ever got a message like that before 1.5.0.8, you might have gotten more since 1.5.0.8.

If you delete the .msf file yourself, you will lose settings like the current sort order for the folder, but it's not a big deal - people delete .msf files all the time.
(In reply to comment #76)
> sort order for the folder, but it's not a big deal - people delete .msf files
> all the time.

I can't agree with this point. No-one deletes .msf files regularly except some developers. And you also lose any label you set before. So in my opinion it's a really big deal to come back (if you really can do that) to the former state.

It's nice to hear that we rebuild the summary file automatically. That is the only way for people who are just users of a software. Thanks.
David, thanks for the info.

As for MSF file deleting/regeneration:
Does that mean I just lose sort order, BUT NOT read/unread/label flags ?
That is actually the most important thing for me. If I lose sort order,
I don't care, but if read/unread/label flags are lost, that is a big
problem for me and I bet for the majority of other users aswell.

Thanks,
Jose.
(In reply to comment #78)
> That is actually the most important thing for me. If I lose sort order,
> I don't care, but if read/unread/label flags are lost, that is a big
> problem for me and I bet for the majority of other users aswell.

The read or unread status won't be affected by deleting the mail summary file because it is stored inside the MBOX file (X-Mozilla-Status). But as I already described in comment 77 you will lose any of your labels. You can have a try. But remember to make a backup of the msf first. Afterwards you can restore the old state by copying it back.
Erh, I'm new to the bugzilla forum, so maybe I've missed something.  I spent a good few hours searching this and other relevant sites for information about this reference header problem.

However, I'm at a total loss here, what do I get if I install Thunderbird 1.5.0.8?  Information is abound, but not very consistent.  This report says the bug is fixed (albeit with regression); still the current version number has no indication of this - almost makes you wonder if there is something to hide?  The release notes do address the issue, but settles it with a recommendation to "not compact your folder" until the 1.5.0.9 release.  Clearly this indicates that you should not expect the bug to be fixed.

David wrote this in Comment #57:
(From update of attachment 245652 [details] [diff] [review] [edit])
this landed on the trunk and 2.0 branch today.

- does this mean a patch has been included in the current Thunderbird build (1.5.0.8)?

Sorry if what I just wrote is a boring drag of messed up information, but this is what you get when you search the forums and the official web site.  I won't ever dare to use rel 1.5.0.8, unless I am certain the reference header problem has been dealt with.  After all, having mails deleted spontaniously is just about the worst kind of error you can come up with in an email client - and for the Microsoft clan this is just the kind of story they'd love to have a good laugh at with their clients.  

Right now, it would be great to know that you don't risk loosing your emails if you install the current TB version.  If this is not possible, I think people should be advised to install 1.5.0.7 until a proper fix has been included in the release.

And thanks for providing a great open source program!

- Rasmus, who will never compact a folder

The 1.5.0.8 release notes were changed to say that the bug exists in 1.5.0.8. The bugzilla status of FIXED means the bug is fixed in the latest code in our CVS repository. It has nothing to do with whether the bug is fixed in any particular release, though of course many many people get confused about that.  We use keywords to say which additional releases a fix will be in, hence the fixed1.8.0.9 keyword above.  We really don't expect end-users to use bugzilla in this way - it's really more for developers and testers. 
Oh no, I hoped this was not the case.  As I read your reply, the current available release may actually delete some of your mail (!), still it is being offered for download (!!).  

I understand this is not an end-user site - consider me a tester then ;-)  The reason for my being here, is that this is where it happens, and sadly I cannot rely on information about Thunderbird from the product page or the user forums. Did you know there are still moderator posts in the forums advisning people to compact folders in order to solve the problem?  Not new ones but still.  This is like a recipe for disaster if you are trying out Mozilla and installing a fresh copy...

For the TB project I really, really, really hope this bug does not lead to widespread dataloss among end users, as this way of patching-without-reversioning could lead to a major loss of credibility.  

I think, most people can accept that a system might be marked by some bugs, provided they can trust that at least their data is not being corrupted.  As was posted before, if loosing the data of an IT system is not a blocker, then what is?  Data loss corrupts confidence like nothing else.  Crash the system, freeze the computer etc., users will put up with this as long as you can 'fix it' and get back to normal.  But data loss?  Ouch.

Hope you understand my concern.  Personally, I will consider always running (at least) one version below current release, but this can hardly be a satisfying strategy for the Mozilla project.  Maybe some bugs are even worse (like?), but  several people have noted here that this kind of bug never occurred before.  With this in mind, I think maybe the development team should reconsider the strategy for addressing data loss bugs.  All said in great respect for a cool product made by some cool people ;-)

Regards!
Rasmus, i totally agree with you, and so do most people i think. For some reason the guys in charge of Mozilla don't think this is a very serious situation. They seem to take the Microsoft approach of shipping the patch on their pre-determined schedule, rather than fixing it right away. This MS late-patch thing was exactly the incentive that made me start using Firefox rather than IE, but now i'm sure i will at least not use Thunderbird anymore once i install Vista and get the new Windows Mail or Outlook. You're right: losing mail is probably the worst thing that an email program can do, and i would rate it much higher than the usual security issues, which you often rarely run into. Also, i think many users will get really upset once they do upgrade to 1.5.0.9, because suddenly a lot of mail will re-appear from out of nowhere (provided they haven't compacted their mailbox), making the problem very visible. So at that point i'm sure many users like you and me will lose all confidence in this product, and rightly so. I already wrote a story on slashdot.org about this in the hope that it would get some more publicity and users would be warned so that hopefully they could prevent their own mail from disappearing. Unfortunately it wasn't published, i assume because either it had already been talked about before, or because an opensource-fanatic was reviewing it. (A similar issue in a Microsoft product usually gets lots of attention there, so it does at least make you wonder.) Feel free to post your experiences there too, maybe you'll be more successful.

By the way: i have nothing against Mozilla - i've been a happy user of their products for at least a year, and of course it is to be expected that there are bugs in any software product from time to time. However, i think they handled this whole thing extremely poorly. I can't think of a situation where users wouldn't lose their confidence after losing their mail while the issue was known by the developers... :(
niek: go away.

this is not the way we work. you're apparently not actually permanently losing data, just not being able to see it. for this, i'm sorry. when you upgrade to 1.5.0.9, your problems should be automatically corrected.

note that the "forums" for the most part are not designed to delete things, including bad/outdated advice. we can't and won't "fix" problems by deleting references to them. just as we can't and won't remove your objectionable comment from bugzilla.

please be aware that releasing a new version of any product is complicated. and releasing a version of our branch releases requires us to have satisfied whatever security concerns we have, not just non dataloss bugfixes. if we release with the wrong set of security fixes, we get a lot of bad press and hurt our users substantially more.

politics of any sort do not belong in bugs. and hopefully you'll leave them in the bitbucket instead of tryiing to air them anywhere else.
"go away"?  That's a little harsh, isn't it?

This isn't a forum, so it's certainly not the right venue for either of you to vent your opinions on/of either side of the argument.  But I do share Niek's concern; I won't re-iterate why, just read above.

However, you do state that..
"you're apparently not actually permanently losing
data, just not being able to see it."

In the interest of the bug description's integrity it should be noted that although the direct effect of this bug does not cause data loss, effects from another common action taken does.  Compacting folders is a recommended practice by 'Mozilla', on pages at MozillaZine -and- is often the answer even by ThunderBird developers for the description of "I can't see my e-mail".
So stating that a person is not actually permanently losing data is based on the assumption that those affected by this bug would not compact their folders, which is an assumption which, given the aforementioned, is horribly flawed.

I think we all understand that updating any program with the scope of ThunderBird requires great care to be taken, including taking the interest of security at heart.  However, I'm not sure why you do fear bad press from any security issues but do not fear bad press from losing e-mail; In fact, 'ThunderBird' as a project is very lucky that the press has, so far, not picked up on this ( Tweakers.net aside ). Moreover, and back on topic, there have been other suggestions made that do not involve updating the program at all, but simply warn the users that they may currently be losing e-mail.  Is that truly too much to ask?  If not - who, exactly, do we ask who is (apparently) not monitoring this bug?  I suppose it is December and we may see the official update pretty soon anyway, making it a moot question.

Just my, last, 2cts in this bug / discussion.
Timeless Anonymous:

"niek: go away."

Always a great way of starting a discussion ;)


"you're apparently not actually permanently losing
data"

I'm sorry to say, but you apparently don't understand what this bug is about. Yes, i have permanently lost data, and so have many others. Please read this bug report (including its comments) before you write something like this, especially when you start your ramblings the way you started them. 


"when you upgrade to 1.5.0.9, your problems should be automatically corrected"

No. Not when you follow Mozilla's recommended practice.


"politics of any sort do not belong in bugs."

I agree, but i didn't see another way of getting the attention of the Mozilla's in charge of this situation, and i was trying to do something in order to prevent others from facing data loss. I think i (and others) pointed out some valid suggestions that would prevent alienating a lot of users down the road. This is in everyone's interest. Ignoring the end user is never a good thing. The simple fact that everyone who knows about this issue seems to have downgraded to 1.5.0.7 should tell you that this is generally considered a lot more serious than whatever security bug may have been fixed in .8, and so this bug can "get a lot of bad press and hurt our users substantially more", to put it in your own words. Especially by not warning about it. Also, this suggests that distributing 1.5.0.8 is probably not a thing that users would want if they knew the details, and distributing 1.5.0.7 would be the best thing to do until it's properly fixed. And you already have 1.5.0.7, so there's nothing complicated about that. Also, being a software developer myself, i fail to see why 1.5.0.8 can't simply be updated with just this fix, and shipped out. Give me one good reason why this hasn't been done.


"note that the "forums" for the most part are not designed to delete things, including bad/outdated advice. we can't and won't "fix" problems by deleting references to them. just as we can't and won't remove your objectionable comment from bugzilla."

Again, this shows you haven't read this bug report. But, could you tell me what you found so 'objectionable' about my comments? You know, even when you really like a product and are passionate about it (like you seem to be), that doesn't mean you shouldn't be allowed to criticize, in order to improve. Or does it? 
I am still experiencing this issue on nightly 1.5.0.9 2006-12-04-14-mozilla1.8.0-l10n

If anything it has become even worse.

Moderators are still advising people to compact their folders over in the forum.

I have lost all faith in Mozilla - you can no longer ensure the integrity of my mail, and you have shown contempt for your users by leaving this bug in the open. Shame on you Mozilla.
"still having this problem" meaning new incoming messages with identical reference and message-id header's aren't getting displayed?
(In reply to comment #88)
> "still having this problem" meaning new incoming messages with identical
> reference and message-id header's aren't getting displayed?
> 
I was affected by the original bug in 1.5.0.8. I subsequently installed a nightly.

Sorry I cant answer that - all I know is that email is being downloaded, I receive notification that new mail has arrived, and I can find no new mail in any folder. This has hapened multiple times since installing the nightly.

If you can offer some guidance on how to test I will try to test properly. I am on a deadline at the minute and cannot test currently.
I went back and verified that a 1.5.0.9 build handles parsing the Test5 folder attached to this bug, such that all the messages show up. That's pretty much the test case I have since I haven't found a way to get the problematic type of messages sent to me via a mail server. If anyone has subscription instructions for a mailing list that exhibits the problem, that would be helpful.

First, you should double check that you're really running the 1.5.0.9 build using help | about.

Next, if you haven't already, check that the mail isn't ending up in the junk folder.

Then, you should open your Inbox in a text editor, scroll to the bottom, and look for messages that you haven't seen in the message list - if you find any, you could copy+paste the headers and e-mail them to me, or attach them to the bug. You'll also see deleted messages in the text editor, but they'll usually have an x-mozilla-status header line with a value of 8 or 9 (the last hex byte will have the high bit set, indicating it was deleted)
Flags: blocking-thunderbird2?
Can anyone confirm that this bug has been fixed in the official TB 2.0 beta 1 (20061206)?

I already wrote an article on all the big improvements, but don't want to recommend testing the new beta when users could loose their mails.
Verified fixed using todays nightly 2.0 branch build. (And yes, beta1 is ok.)
Keywords: fixed1.8.1.1 → verified1.8.1.1
Any update update on this for TB1.5 folks ? It's mid-december!
Thanks,
Jose
there was a delay in 1.5.0.x - doing a release is a very complicated, time consuming process, but our release team is working hard to get it out ASAP.
David: why couldn't you guys update 1.5.0.8 with *just this fix*, immediately after you made it? The way you stick to your update schedule seems worse than Microsoft's patch-tuesday scheduling. This episode has really made me dislike this product, when to me it seems that it could have been so easily prevented. Not only did i lose trust in Thunderbird after permanently losing mail, but i now also lost trust in the way Mozilla handles these kinds of issues. What's your opinion? It'll be interesting to see what happens when the update does arrive, and users suddenly see their inbox stuffed with mail from a month ago.
This problem is not mentioned in the release notes ?
http://www.mozilla.org/projects/security/known-vulnerabilities.html#thunderbird1.5.0.9
So I assume this bug is not fixed, right ?

It's fixed. (Guess it falls under "Improvements to product stability.")
http://en-us.www.mozilla.com/en-US/thunderbird/releases/1.5.0.9.html
Thanks Magnus!

Now I'd be most grateful, and I'm sure a bunch of other users too,
if someone (preferably from the Mozilla team) could post a brief
"guide" for upgrading to 1.5.0.9:
I.e. is it possible to just upgrade and possible hidden mails
just appear ? Or what must we do to make possible hidden mails
appear ? I'm sure, deleting the MSF files is no option for many
users, since the flags are lost, so is there a way without deleting
the MSF files or is this handled automatically ?

Thanks a lot!
Just upgraded

No new mails appeared. I know that I have suffered from this problem and I have not compacted folders for some time (since I realised that compacting deleted the hidden mails).

Please advise on method for retriving hidden mails.
Try deleting your inbox.msf file - but if no messages appear, then you weren't actually suffering from this problem...
the flags aren't lost when deleting the .msf file, unless you've got e-mail migrated from an other e-mail client with TB 1.0 and you've never ever compacted your folders with 1.5
One other thing to try:  I had the same issue, I chalked it up to possibly a weird side effect of the bug (?) with normal usage of the corrupt Inbox.  Since then I fixed it as per message #73/#74: https://bugzilla.mozilla.org/show_bug.cgi?id=360409#c73 .  Qgrep is available if you go to microsoft.com and type in "qgrep" in their search box.  Make a backup copy of your Inbox first!  Of course, this is a shotgun approach which deletes some other message settings and brings back all other deleted mail too.
what does that Qgrep statement actually do? I don't see how it could be related to this bug since it seems to deal with x-mozilla-status lines...
It was something I tried that worked for me after I upgraded to the daily build that had the fix applied, and manually deleting the .msf files didn't bring my regular mail back.  The intent on my part was that if the invisible message got marked as deleted (one of the X-Mozilla- line features) somehow unknown to me after they became invisible (don't know how but figured it might be a side effect - with previous normal mail usage in the buggy version / other message range deletions?).  Unfortunately my Inbox was way too big  (hundreds of megs) to view/edit with Microsoft text-based tools, so I couldn't do further analysis, but it did fix the issue for my Inbox so I offered it up as a suggestion if all else failed.
After rebuilding the .msf files I still have no new mail re-appearing. I was under the impression that this fix would rebuild my mailboxes and automatically fix this issue? 

I am quite certain that I have been affected by this bug. I am quickly losing confidence in TB. I do not feel that I should have to use other tools to affect the required changes.

I will try to manually look through the inbox files to look for messages that may be affected by this and report back here.

Rgds

Richard
Re: Comment #106 ... did you ever compact your folders with TB 1.5.0.8?  That is the kiss of death.
Unless you actually have evidence of receiving messages with common reference and message-id  headers, you don't know that you've been affected by this bug. 
(In reply to comment #108)
> Unless you actually have evidence of receiving messages with common reference
> and message-id  headers, you don't know that you've been affected by this bug. 
> 

I was affected by 1.5.0.8. Messages re-appeared on downgrade. Since moving to nightly I have seen a variance between reported new messages in system tray Vs actual new messages in TB.

I did compact previously (as per the dreadful advice still being given in the TB forums). But I have not compacted for many weeks now to avoid any further potential loss.

Is there any simple way to check for common headers without manual inspection?
Manual inspection is the only way to be sure that I can think of, other than writing some fancy perl script...
Some fancy perl script ?
Here is my quick hack and I already detected some hidden
mails here (well, all but one are SPAM anyway):
http://www.captain.at/howto-thunderbird-mailbox-checker.php

cool - note that the resent-message-id is irrelevant - Thunderbird ignores it.

Where was the non-junk message from?
(In reply to comment #112)
> cool - note that the resent-message-id is irrelevant - Thunderbird ignores it.

ACK - I will modify the script.

> Where was the non-junk message from?

I just verified it and it is actually a false detection.
It is a return receipt, where the original mail headers are attached.
I try to circumvent this, but better have a few false detections than
reviewing a half-gig mailbox by hand.


Thunderbird Mailbox Checker V0.2 was just released:
http://www.captain.at/howto-thunderbird-mailbox-checker.php

Changelog:
V0.2: removed check for Resent-Message-ID since it is ignored by TB. Added detection of end of header to avoid false detections at i.e. "return receipts" where the header of the original email is attached.
Hi Hannes,

> Here is my quick hack and I already detected some hidden
> mails here

So do you mean that the updated Thunderbird (1.5.0.9) still suffers from the disappearing message issue? Or did you write the script for 1.5.0.8 only? In other words, should i update from my current 1.5.0.7 (which i downgraded to) to 1.5.0.9, or not yet? I want to make triple sure this time that my messages won't be disappearing again. Richard still seems to be having problems?

Thanks,
-- Niek
(In reply to comment #115)
> Hi Hannes,
> 
> > Here is my quick hack and I already detected some hidden
> > mails here
> 
> So do you mean that the updated Thunderbird (1.5.0.9) still suffers from the
> disappearing message issue? Or did you write the script for 1.5.0.8 only? In
> other words, should i update from my current 1.5.0.7 (which i downgraded to) to
> 1.5.0.9, or not yet? I want to make triple sure this time that my messages
> won't be disappearing again. Richard still seems to be having problems?

Hi Niek!

I don't know...
I have not tried 1.5.0.9 yet. I will probably clone my Inbox (with the
SPAMs that have those headers and are hidden) and give it a try with
1.5.0.9 on another machine. I keep ya'll posted about the result.

My script is only to check mailboxes for mails with problematic headers.
It is basically TB-version independent. You can run the script on any
MBOX, whether it is a TB mailbox or not. The script just shows basic
information about emails with equal "Message-ID" and "References" headers.



For what it's worth, I had written a similar script (in a whole different language, though), and since 2006/Nov/07* I would have lost 792 e-mails.  The vast majority of those would be from our mailing system, 29 of them however woulc come from outside mailing systems.  8 of which came from a Packard Bell mail server, which seems to be fixed since at least 2006/Nov/19.
(* Note that this excludes e-mail received since 2006/Jan/01.  E-mail prior to that date is archived away and no longer in the mailbox files, so wouldn't be affected.)

I'll be installing 1.5.0.9 shortly.. if there's still some manner of issue, I'll find out soon enough - the hard way :)

I'd like to express my disappointment that this issue is not explicitly mentioned in the release notes, though.  Although I am confident that it is addressed (why claim otherwise?), it does mean that the general public will remain unaware that they may have lost e-mail due to this bug.  Perhaps it is better that way.  Time to get paranoid yet?
looks fixed here
(In reply to comment #118)
> looks fixed here

Richard,

do you mean that the hidden mails automatically appeared after
upgrading to 1.5.0.9 ?


I should have made myself clearer :)

I don't know if it makes the previously invisible e-mails visible again, as I have been using 1.5.0.7 while waiting for a bugfix.

However, none of the e-mails that would automatically be 'invisible' with 1.5.0.8 are invisible now.

--

What I was referring to, however, was more with new malformed e-mails coming in not automatically disappearing - which was my primary concern.
Richard,

thanks for the clarification.
I guess we can conclude:
* bug fixed in 1.5.0.9
* hidden emails in 1.5.0.8 do not automatically appear in 1.5.0.9

As for hidden mails I'd suggest the following (without losing
any flags or whatsoever by deleting the MSF file):
In 1.5.0.9: Just copy the affected mailbox and give it another name.
So it will have no MSF file, and TB 1.5.0.9 will rebuild the MSF for
the "new" mailbox.
Look for the emails detected with
Thunderbird Mailbox Checker
http://www.captain.at/howto-thunderbird-mailbox-checker.php
in the "new" folder and move them to the original mailbox.
WARNING: Only try this with 1.5.0.9! TB 1.5.0.8 will hang when moving
such mails to the original mailbox.
This bug does still exist in TB 2.0.0.0. 

http://forums.mozillazine.org/viewtopic.php?t=542398

When I move (or copy) certain messages manually or by means of a filter from the inbox to a subfolder, sometimes these messages disappear, sometimes not.
I use Windows 2000 And discovererd this bug in 2.0.0.3

It seemd (but I am not 100% sure) that the new Thunderbird viewed some messages as spam, I uncheck the spam-marker and put the mails back in the Inbox.

Rusult the mails were gone!?

These were importand mails for me.. it scared the hell out of me, that mails could get completely lost.. I made a backup of my inbox and looked for the lost mails.

They were still there then I did a compact folders => the mails were REALY gone. Then I restored the backup, resulting in a rebuild of the MSF-file, and the mails were back!

I thought returning to my previous version (1.5.0.4) would solve the problem, but what I have read here, it looks like the problem may exist in that version also? :( I will keep an eye out, checking if mails in this version (1.5.0.4) also get lost..

My preliminary conclusion, there is something wrong with the msf-file-structure...

Concluding.. reading this thread, it looks like this problem had been going on for some time, without beeing solved.. this scares the hell out of me, I have an internet company, and need to be able to rely on mails NOT disappearing :(

This realy needs to get resolved! I have been a loyal user since Netscape Messenger 4.something, but this isn't just a bug, this bug is huge and verry scarry!
What you see is bug 368112. This is already fixed in current branch nighlies.
(In reply to comment #124)
> What you see is bug 368112. This is already fixed in current branch nighlies.
Do I read it correctly that this should be fixed with 2.0.0.4?
(In reply to comment #125)
> > What you see is bug 368112. This is already fixed in current branch nighlies.
> Do I read it correctly that this should be fixed with 2.0.0.4?

Yes, TB 2.0.0.4 will be the next official release where the issue from bug 368112 will be fixed. So let's stop us here.
I don't think it's been decided it will be called "2.0.0.4", it might be time to let Thunderbird go it's own numbering way and call it "2.0.0.1" since 2.0 was just released.

But yeah, the "next release" will have the fix.

Question, it is said here that with version 2.0.0.4 this problem would be fixed. Where can I see if this problem has been fixed in the latest version, I can only find version 2.0.0.0 ? 
This bug is fixed in 2.0.0.0. The other issue - bug 368112 will be in the next point release. For pre-release builds see bug 368112 comment 52.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: