Last Comment Bug 360409 - mails with messed up references seem to disappear
: mails with messed up references seem to disappear
Status: RESOLVED FIXED
: fixed1.8.0.9, regression, verified1.8.1.1
Product: Thunderbird
Classification: Client Software
Component: General (show other bugs)
: 1.5
: All All
: -- critical with 7 votes (vote)
: ---
Assigned To: David :Bienvenu
:
:
Mentors:
: 360252 360340 360530 360643 (view as bug list)
Depends on:
Blocks: 332883
  Show dependency treegraph
 
Reported: 2006-11-11 17:36 PST by Kent S.
Modified: 2007-05-31 02:37 PDT (History)
33 users (show)
dveditz: blocking1.8.0.9+
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Test5 folder (6 problematic messages), plus related .msf file (7.48 KB, application/x-zip-compressed)
2006-11-11 17:43 PST, Kent S.
no flags Details
proposed fix (1.09 KB, patch)
2006-11-13 12:41 PST, David :Bienvenu
mscott: superreview+
Details | Diff | Splinter Review
do a one time detection of this case, and regenerate summary files that appear to have this problem. (4.09 KB, patch)
2006-11-15 07:47 PST, David :Bienvenu
mscott: superreview+
jaymoz: approval1.8.0.9+
Details | Diff | Splinter Review

Description Kent S. 2006-11-11 17:36:40 PST
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0
Build Identifier: Thunderbird/1.5.0.8 (as well as the Trunk nightly builds)

After upgrading from 1.5.0.7 to 1.5.0.8, I determined that *certain* new mail (specifically, new mail sent by the X-Mailer "Chilkat Software Inc (http://www.chilkatsoft.com)") did not appear in either the Inbox, or in the subfolder where it was sent by my filter (even though the subfolder had an "arrow" indicating new mail).  I also found that mail previously (pre-upgrade) received from a Chilkat X-Mailer *or* from an X-Mailer of either "JMail 4.4 by Dimac" or "JMail 4.5 by Dimac" was no longer visible in 1.5.0.8 (I haven't received any mail sent via JMail post-upgrade, so I haven't been able to test that).  So far, I have only seen the problem affect mail sent from one of these 2 X-Mailers, though my testing is by no means exhaustive.

It seems that the crux of the problem is an inability to process the headers of e-mail generated by these 2 X-Mailers, and that inability results in a corrupt *.msf file (which then causes the messages to be invisible and/or lost).

Reproducible: Always

Steps to Reproduce:
1   Close TB 1.5.0.8.
2.  Copy the attached "test5" file (created in 1.5.0.7, by copying 6 messages into a new folder called "Test5") into the Inbox.sbd folder.  The Test5 file contains 6 e-mail messages, 3 generated by the JMail X-Mailer, and 3 generated by the Chilkat X-Mailer. FOR THIS TEST, DO NOT COPY THE INCLUDED TEST5.MSF FILE.
3.  Restart Thunderbird (which will cause generation of a new .msf file), and navigate to the Test5 folder, under Inbox.
4.  Look to see how many messages appear (there should be 6; I see 1).
5.  Attempt to copy/move the file(s) in the Test 5 folder to another folder.

Actual Results:  
Although the Test5 folder should display 6 messages, it displays only 1.  Attempts to copy/move that message result in a perpetual hourglass.  There is no way to access/see the other 5 messages in the file/folder.  There is no way to "manage" any of the messages in the file/folder.  If the folder is compacted, the missing messages will be permanently lost.

Expected Results:  
I expect 1.5.0.8 to behave in the same manner as 1.5.0.7.  After restarting TB, I expect to see all 6 messages in the Test5 folder, and I expect to be able to copy/move them easily and without incident.  I do not expect compaction to cause mail database corruption.

The attached file (TB1.5.0.7.zip) contain the Test5 file (6 messages), and the related Test5.msf file ... both created in TB 1.5.0.7.

As a reference point, copy *both* files into Inbox.sbd in TB 1.5.0.7.  Restart TB.   You will see 6 messages, and they can be copied, moved, managed, etc. as one would expect.

Now copy both files into Inbox.sbd in TB 1.5.0.8.  Restart TB.  You will see the same 6 messages.  However, if you attempt to copy/move all 6 of them, it will appear that only 1 message was copied/moved (and the operation will generate a perpetual hourglass).  If you look at the underlying data files, you will see that all of the selected messages were, in fact, copied, but that the *.msf file in the destination folder is corrupt, so it appears that but a single message was copied (and, if a "move" operation is performed, the files in the source folder will *not* be deleted).  If you delete the *.msf file in the Test5 folder, and allow TB 1.5.0.8 to regenerate it, only a single message will appear in the folder (though if you look at the data file, you will see that it contains all 6 messages).

If you copy the Test5.msf file generated in TB 1.5.0.8 into the Inbox.sbd folder in 1.5.0.7 (overwriting the equivalent file generated by 1.5.0.7), and then restart TB, you will see but a single message in the folder (though the data file contains all 6).  If you then delete the Test5.msf file in TB 1.5.0.7, and restart TB, allowing TB to regenerate the file, you will see all 6 messages again.

More details of the problem, together with comments/experiences of others, can be found in the thread: http://forums.mozillazine.org/viewtopic.php?t=485752 I created the topic, and the analysis (a bit rough at first) becomes much more focused as the testing progresses.
Comment 1 Kent S. 2006-11-11 17:43:24 PST
Created attachment 245365 [details]
Test5 folder (6 problematic messages), plus related .msf file

The attachment contains 2 files, (i) Test5 ... a message folder with 6 messages, 3 generated by the Chilket X-Mailer, and 3 generated by the JMail X-Mailer, and (ii) Test5.msf, the related .msf file.  Both files were created in TB 1.5.0.7, by copying the 6 selected messages to a new folder called "Test5".
Comment 2 Kent S. 2006-11-11 17:47:54 PST
My interim solution for this bug has been to reinstall TB 1.5.0.7, and disable all automatic program upgrades.
Comment 3 Kent S. 2006-11-11 19:59:04 PST
As clarification, not only does this problem occur in TB 1.5.0.8, I have also been able to reproduce it in TB version 2.0, beta 1 (20061111).
Comment 4 Magnus Melin 2006-11-12 10:18:17 PST
Confirmed on Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20061112 Thunderbird/3.0a1 ID:2006111204, branch too.
Comment 6 dan 2006-11-12 18:11:32 PST
*** Bug 360340 has been marked as a duplicate of this bug. ***
Comment 7 Scott MacGregor 2006-11-12 19:38:28 PST
cc'ing David in case this was caused by bug 351685 and bug
332883 on the security branch.
Comment 8 Magnus Melin 2006-11-13 06:41:33 PST
*** Bug 360530 has been marked as a duplicate of this bug. ***
Comment 9 Magnus Melin 2006-11-13 11:05:32 PST
All the messages that do not show up have Message-ID, References, Resent-Message-ID set to the same value, like
Message-ID: <CRID_7bbabaab57ba41e69a354d392b2d6e74@centercode.com>
References: <CRID_7bbabaab57ba41e69a354d392b2d6e74@centercode.com>
Resent-Message-ID: <CRID_7bbabaab57ba41e69a354d392b2d6e74@centercode.com>

So it looks like a messed up threading issue. 

Doing a local backout of attachment 239252 [details] [diff] [review] in bug 332883 fixed it for me.

Bumping severity, seems like this is affecting a lot of people (from discussion e.g. in http://forums.mozillazine.org/viewtopic.php?t=487248). And it really is rather nasty.
Comment 10 David :Bienvenu 2006-11-13 12:03:04 PST
I agree that it's nasty, but it's not a blocker...most users in normal usage won't get multiple messages with the same message-id...I've been looking at this but I don't think I'll have a solution before  get back from vacation.
Comment 11 Daniel Veditz [:dveditz] 2006-11-13 12:13:04 PST
Maybe not "most" users, but seems to be happening more than you would think from the discussions about it.
Comment 12 Kent S. 2006-11-13 12:22:32 PST
Regarding David's comment about "most users", when he reduced the severity from "blocker" to "critical".

I didn't report this bug as an intellectual exercise, or a contrived test case.  I reported it because one of my mail folders contained 343 messages in 1.5.0.7, and the number was immediately reduced to 271 (a loss of 72 messages) upon installing 1.5.0.8.  Further, all *new* messages from a particular major software house where immediately lost in 1.5.0.8 ... without being displayed in TB at all (and permanently lost if you compact the folders ... which I did).

I was able to debug this issue because all of my mail now traverses GMail before going to TB, and GMail retains an archive.  Fortunately, my drive is imaged every night, so I was also able to recover all of the older lost messages that didn't go through GMail.

How nasty is it, and should it be a blocker?  I guess that depends upon whether it is your mail that is being lost ... or mine.
Comment 13 Niek 2006-11-13 12:38:11 PST
I had the same problem.. i could still see the affected messages when viewing the Inbox file using a text editor, but not in TB. Tried compacting, and that didn't help, neither did deleting the msf file. Below you can see the header for one of the affected messages, directly copied from the 'Inbox' file. I'm afraid compacting actually deleted the messages, because after downgrading to TB .7, i only got back a few of them that were dated today, i think from after my last 'compact'.

In other words: this makes TB completely useless, and i can't believe that this version is still being distributed, because it must be hurting many people. I had messages disappear almost every time i got new messages, probably at least 30% of them. Not sure if they were all spam, because i only started really investigating after compacting to see if this would fix it, and that process seems to have deleted most of them.

I just hope it has something to do with spam filtering, because in that case at least i know that i probably didn't lose any important messages. Can anyone confirm this?

Anyway, when i get Vista, i'll be sure not to use TB again - and i'm sure many others will feel the same way. I've been using it for years, but it's unacceptable to lose mail, and i completely lost my confidence with TB, especially since it's still being distributed, when it clearly shouldn't be. The bug report makes it look like this is some isolated issue having to do with a few obscure mail agents, which obviously isn't the case (look below, X-Mailer: Microsoft Outlook Express 6.00.2900.2869)

Good luck everyone, and at least downgrade for now, as a quick fix! (and switch off Automatic updates)

If this is not a "blocker", i don't know what is... (you'll feel the same way if you're waiting for an important email, your mail program says "new mail!", and you discover an empty inbox...)

-- Niek


From - Mon Nov 13 17:35:55 2006
X-Account-Key: account2
X-UIDL: UID15741-1103787658
X-Mozilla-Status: 0000
X-Mozilla-Status2: 00000000
Return-Path: <ljveeflbi>
Delivered-To: 3-niek@xxxmydomainxxx.com
Received: (qmail 24804 invoked from network); 13 Nov 2006 16:27:05 -0000
Received: from dsl-189-166-104-20.prod-infinitum.com.mx (189.166.104.20)
by u15175353.onlinehome-server.com with SMTP; 13 Nov 2006 16:27:05 -0000
Message-ID: <000601c7073f>
From: "Were" <ljveeflbi>
To: niek@xxxmydomainxxx.com
References: <000601c7073f>
Subject: Re: glory World Service
Date: Mon, 13 Nov 2006 10:16:14 -0600
MIME-Version: 1.0
Content-Type: multipart/related;
type="multipart/alternative";
boundary="----=_NextPart_000_0004_01C7070C.BFB6D1E0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.2869
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2962
Comment 14 David :Bienvenu 2006-11-13 12:41:30 PST
Created attachment 245484 [details] [diff] [review]
proposed fix

ignore references of headers that claim to be a reply to themselves.
Comment 15 bugmoz 2006-11-13 12:47:04 PST
If this bug is technically a duplicate of Bug 360252, I suggest marking that bug a duplicate of this, as this entry has far more information.

Oh, and add one more person is affected by this and is currently downgrading to 1.5.0.7
Comment 16 David :Bienvenu 2006-11-13 12:47:21 PST
I'm sorry for that regression, and I fixed it as quickly as I could while I was on vacation. Compacting was not related to this bug, since compaction ignores message threading...

I did not realize that Outlook Express was so broken...

If you sort by date, or any other column, and are non-threaded, you will see all your messages (you have to be in a flat sort when you open the folder, so if you're threaded, you have to sort by date, and then click away from the folder, and back).
Comment 17 David :Bienvenu 2006-11-13 12:48:40 PST
*** Bug 360252 has been marked as a duplicate of this bug. ***
Comment 18 Kent S. 2006-11-13 12:53:48 PST
Re: Comment 9, suggesting that the problem is the result of Message-ID, References, Resent-Message-ID set to the same value (and someone changed the description of the bug accordingly), and the proposed fix in Comment 14.  Please look at Comment 13, where there is but a Message-ID (i.e., the message is not purporting to be a reply to itself).

I'm not a software developer, but I am able to discern the factual inconsistency, and therefore question the conclusion set forth in Comment 9, and the related change in the description of this bug.

Of course, there may be more than a single issue involved here.
Comment 19 David :Bienvenu 2006-11-13 12:56:31 PST
From comment 13:

Message-ID: <000601c7073f>
From: "Were" <ljveeflbi>
To: niek@xxxmydomainxxx.com
References: <000601c7073f>

Note that the Message-ID and the References header are precisely the same, which means that the message says it's a reply to itself. Does that make sense?
Comment 20 Niek 2006-11-13 12:57:49 PST
David, what do you mean with "If you sort by date, or any other column, and are non-threaded, you will see all your messages". My messages weren't threaded, and they were sorted by date, and i could not see all my messages. (View -> Sort by -> Date / Descending / Unthreaded). The only thing i could do to make them show up was downgrade to 1.5.0.7 and delete the msf file.

By the way: why is this bug's status still "NEW"?

Regards,
-- Niek
Comment 21 Kent S. 2006-11-13 13:01:12 PST
Re: David's Comment 19 ... my bad, I just missed it (the References).  Of course, I understand.
Comment 22 David :Bienvenu 2006-11-13 13:05:35 PST
With the attached test folder, if you're threaded, you only see one message. If you sort by date, then click away and click back, you'll see all six messages again.

If you were ever threaded in the folder, you'll miss messages until you open the folder such that it opens non-threaded (i.e., it was sorted, flat, when you closed the folder, before you re-opened it). That's the case for this test case, at least. Perhaps other test cases behave differently - are you saying the folder was never threaded? In that case, we might want a new test case that demonstrates that...
Comment 23 bugmoz 2006-11-13 13:09:49 PST
Example in Bug 360252 has "Exhibit B" never showing.  I always work unthreaded (View > Sort By > Unthreaded) and sorted by date (View > Sort By > Date).  Descending, for what it's worth.

I have now downgraded to 1.5.0.7 (can't very well have e-mail be there but not showing up!) so I can't do any further testing.
Comment 24 Kent S. 2006-11-13 13:13:23 PST
David ... I created the test folder, and I've never used TB in threaded mode (by default, I sort by date).
Comment 25 Niek 2006-11-13 13:13:38 PST
By the way, again: i'm pretty sure that compacting deleted the invisible messages, even after returning to 1.5.0.7. The only messages i got back were the few (3) that i received after my last 'compact' action.

Please read the following threads for some more user experiences, also referring to the compacting effect:

http://forums.mozillazine.org/viewtopic.php?t=487248

http://forums.mozillazine.org/viewtopic.php?p=2594779

These two threads seem to be *by far* the most active ones in their respective categories over the last couple of days, suggesting the problem is far bigger than some of you would want to think. Which again warrants a "blocker" rating, i would hope...

As for being threaded or not: as far as i remember, my inbox was never threaded. But i might have tested this option a year ago or so, i don't know. At least i haven't made any changes to the sorting/threading in the last couple of months: it has always been the way i described it (View ->
Sort by -> Date / Descending / Unthreaded) although i sometimes click on 'date' again to sort the other way, or click on the View -> Unread tab when i get spam with an old date and i want to quickly find and delete it. 

Regards,
-- Niek
Comment 26 David :Bienvenu 2006-11-13 13:20:32 PST
Sorry, I thought Dan had changed it back to blocker, my mistake. Marking as blocker.

I used the test case on the trunk/2.0 builds - perhaps the 1.5.0.8 build behaves differently...but as the original test case describes, you

Comment 27 Daniel Veditz [:dveditz] 2006-11-13 14:46:38 PST
We try to say that the regression bug "blocks" the original bug, that is, correctly fixing the original bug requires fixing this bug. Helps in keeping track should we later want to port the original bug to an old branch.
Comment 28 Daniel Veditz [:dveditz] 2006-11-13 14:56:44 PST
I did not mean to set the severity to "Blocker", which is reserved for bugs that stop development or testing (https://bugzilla.mozilla.org/page.cgi?id=fields.html#bug_severity). I don't think this bug qualifies, critical is more accurate.

However it does appear to be a significant regression and should "block" the next release (which is a flag, not a severity).
Comment 29 Niek 2006-11-13 15:25:21 PST
Ok - i just meant to say that it should "block" the next release. In my opinion, it should even block the 1.5.0.8 release: this will affect many people. Just look at how these are by far the most popular items in both the Mozilla Bugs and the Support categories (see links i provided earlier). And those are just the people who were **** enough to Google the problem and write about it - most users probably don't even notice, since it happens almost invisibly (you never see the mails come in). I only noticed it after a few days, when i happened to pay attention to how many mails it was downloading, and how many it actually displayed (and considering usually at least 50% is spam, you aren't always very surprised if you don't see all the mails that are downloaded because they're automatically redirected to Junk). After realizing it, i noticed it happened almost every time i downloaded a few mails...
Comment 30 David :Bienvenu 2006-11-13 16:28:59 PST
fixed on trunk and branch - will nominate the patch for 1.5.0.9 in just a sec.
Comment 31 David :Bienvenu 2006-11-13 16:29:40 PST
Comment on attachment 245484 [details] [diff] [review]
proposed fix

requesting 1.8.0.9 approval
Comment 32 David :Bienvenu 2006-11-13 16:32:42 PST
To stave off any potential confusion, marking it fixed means it's fixed on the trunk of our source code repository, not that the fix is in an available build yet.
Comment 33 Niek 2006-11-13 16:58:08 PST
Ok, so does this patch fix the issue of the disappearing messages? Your patch seems to be related mostly to threading issues (?), but please note again that it doesn't seem to have much to do with that. At least, in my case, i never used threading, and the messages still disappeared. Also clearly, this should be a bug that was introduced in 1.5.0.8, and wasn't there in earlier versions (at least not any recent ones).
Comment 34 David :Bienvenu 2006-11-13 17:36:10 PST
This bug manifests itself differently on the trunk and 2.0 branch, vs the 1.5.0.x branch. Unfortunately, it's much worse on the 1.5.0.x branch - on that branch, when you get multiple messages that have the same message id, and the same reference id, and claim to be replies to themselves, the subsequent msg hdrs don't actually get added to the db. (on the trunk, they get added, but don't get displayed when in threaded view).

So, for the 1.5.0.x branch, this fix only fixes the problem for new incoming messages, or if you delete the .msf file. To fix it for the 1.5.0.x branch, we'd need to force the regeneration of all local folder .msf files.  This is a big yuck. But I can't think of any other way around it, other than perhaps to detect the case where an existing msg hdr claims to be a reply to itself, and only regenerate the db in that case.  I'll try to come up with a patch for that.
Comment 35 Kent S. 2006-11-13 17:46:13 PST
The bug may manifest itself differently (from a technical standpoint) on the trunk, branches, etc.  However, before I filed the bug, I tested it on the 2.0 nightly, and I got the same results that I got in 1.5.0.8 (of the 6 test messages, when the nightly generated the .msf file, only 1 message displayed ... the first one).  The important point I want to make is that I NEVER used/tested threaded view (I only have 6 messages in the test file ... and they aren't threaded anyway), so I am a bit troubled by the distinction between flat and threaded view, and the subsequent attribution of bug manifestation to threaded view.
Comment 36 David :Bienvenu 2006-11-13 17:55:19 PST
Kent, I'm not sure - when I tried it with a 2.0 build, I saw the six messages when in flat view, and only one when in threaded view. I can try it again later, but it's really not that important right now - if you delete your .msf file, and open the folder with tomorrow's 2.0 nightly build, it should work fine. The really critical thing is fixing 1.5.0.8 builds right now, and somehow regenerating .msf files when we detect this problem.
Comment 37 Kent S. 2006-11-13 18:07:08 PST
... as for 1.5.0.8 ... if you delete all of the .msf files, you get all of the messages back (assuming they haven't been lost permanently via compaction), but you also lose all of the folder-by-folder sort customizations.  That's gonna be hella confusing to peeps like my mom, me thinks (though I don't have an alternate solution).
Comment 38 David :Bienvenu 2006-11-13 18:10:48 PST
If I can detect the corruption and mark the summary file invalid programmatically, we might be able to maintain the sort info, etc.
Comment 39 David :Bienvenu 2006-11-14 06:29:14 PST
*** Bug 360643 has been marked as a duplicate of this bug. ***
Comment 40 Niek 2006-11-14 06:31:25 PST
David, just out of curiosity, and to get a better idea of what i (and others) lost after compacting: do you think these 'self-replying' messages are mostly spam? Or do you know of any legitimate email programs / websites (Outlook, Yahoo mail, etc.) that create messages like this? Is this even legal according to the RFC's in question? The issue would be slightly less damaging if you know that the disappearing messages are mainly spam and/or generated by some poorly written mailer.

Thanks.
Comment 41 David :Bienvenu 2006-11-14 06:42:41 PST
Niek, it's definitely illegal. I had never encountered messages like this before. Some are definitely spam (I suspect that having a reference header decreases the spam score with some spam filters, so they just stick one in, and the message-id happens to be handy).

However, in the Test5 folder attachment, there are several messages with the same problem that don't seem to be spam. They have a "Resent-From" header, and all the message-ids are from centercode.com - this makes me think there's some server-side software rewriting the message-ids - I don't think it's the original sending program generating the bad message-id+reference combination.
Comment 42 Kent S. 2006-11-14 08:09:17 PST
When I created the Test5 folder, I selected 6 messages out of a total of 72 that "disappeared" from one of my folders when I upgraded to 1.5.0.8 from 1.5.0.7.  None was spam.  All were received from a very large Silicon Valley-based software vendor in connection with beta testing (mass mails to all of the beta group, or personalized e-mails generated by their bug reporting system).
Comment 43 David Nicholls 2006-11-15 04:27:00 PST
I can confirm Kent's finding (comment #42). No spam, just beta test group messages.

Essentially two things were happening: 

1) In builds before 1.5.0.8, (1.5.0.7 for example), using the sort by thread command in a folder with messages sent using the Chilkat X-mailer (from another large California-based software company for beta testing) resulted in all but the first message disappearing.  This could only be fixed by deleting the relevant .msf file

2) In build 1.5.0.8, all messages moved to the folder using a filter would disappear. [I did not leave any of these messages in the Inbox so I can't confirm them missing there also].  Removing the .msf file would not reconstitute the message index list.

ie the problems (probably the same problem) appeared to become more extreme with 1.5.0.8

I didn't compact the folder with the hidden messages, so I lost no messages, but I cannot upgrade to 1.5.0.8 or later unless this bug is killed because I need to read the incoming list messages.  Other users on the same beta test list as me have reported the same problem with the list messages.  Worked around by reinstalling 1.5.0.7.

DN
Comment 44 David :Bienvenu 2006-11-15 07:47:40 PST
Created attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

this patch makes it so we'll do a one time run through of the msg hdrs in a db, and if we find messages with the reference header set to the message-id, we'll regenerate the db.
Comment 45 David :Bienvenu 2006-11-15 07:50:01 PST
this will deal with missing messages, if the user hasn't compacted folders...
Comment 46 Kent S. 2006-11-15 08:14:03 PST
will the "one time detection" occur only if the upgrade is from 1.5.0.8, or will it run on any installed version of TB?

will any user preferences (such as sort order) be captured and programmed into the newly-generated .msf file?
Comment 47 David :Bienvenu 2006-11-15 11:00:50 PST
any installed version of TB that has the patch in it will run the run-time check, if it hasn't been run before. The one-time-ness is stored in the .msf files, and new msf files are created knowing they don't need to do the one time check.

Sort info should be maintained...
Comment 48 David Nicholls 2006-11-15 14:36:15 PST
This shows my ignornace of how TB works, but I presume it's necessary to await an updated version of TB installer with the patch applied?  Or is there some way to apply the patch file manually?
Comment 49 Krang 2006-11-15 21:08:34 PST
This bug affects SeaMonkey as well, including SeaMonkey 1.1b.  It is not present in the latest 1.1 nightlies, though, due to the checkin for the patch in comment #14 (Tested using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061115 SeaMonkey/1.1b).
Comment 50 David :Bienvenu 2006-11-16 11:29:09 PST
Comment on attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

I'd really like to get this onto the trunk soon, in order to get it into 1.5.0.9, so switching review to Seth.
Comment 51 Chris 2006-11-18 08:00:30 PST
Is there an estimated time on getting this fix made into a .exe file?  Is there a way to escalate this?  I have been without email for a week now (won't move down to 1.5.0.7 as there is a security bug in it per 1.5.0.8 release notes).  Thanks much!
Comment 52 David :Bienvenu 2006-11-18 08:31:10 PST
I need to get it reviewed; then I need to get approval to check it in; then it needs to be released in a build. After I check it in, you could pull a nightly 1.5.0.9 build before it's actually released...the 1.5.0.9 release is supposed to be in December, afaik. 
Comment 53 Niek 2006-11-18 18:09:15 PST
Sorry, but i don't understand why you guys are waiting until December for such a critical problem. In the mean time, most of your users will have emails disappear (probably without even realizing until their contacts get upset at them because they haven't replied to something), and if they adhere to TB's suggested 'good practice' of compacting their inbox on a 'regular basis', this means that those emails will be lost forever. This should be totally unacceptable, and on equal footing (if not higher) with whatever security updates you may distribute from time to time. If i were still using 1.5.0.8, and i found out a month from now that you guys knew about this problem for over a month, and i happened to have lost several important emails, i would be seriously **** off and sure never to use TB again. If this leaks to the media, i'm sure you'll get a lot of seriously bad press...
Comment 54 David Nicholls 2006-11-18 19:13:33 PST
I'm inclined to agree with Niek.  This bug has the potential to do TB a lot of damage and the patched version should be released outside the standard sequence eg as 1.5.0.8a
Comment 55 David :Bienvenu 2006-11-20 08:33:54 PST
Comment on attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

switch review back to Scott
Comment 56 Scott MacGregor 2006-11-20 10:22:28 PST
Comment on attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

I wonder if we should check the error code before checking the return value:

+          if (msgHdr && NS_SUCCEEDED(err))

Looks good to me.
Comment 57 David :Bienvenu 2006-11-20 13:34:50 PST
Comment on attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

this landed on the trunk and 2.0 branch today.

I also looked through a lot of my old mail and didn't find any messages like this, except for a test case folder for the bug that caused this regression. So I suspect that most users are not running into this.
Comment 58 bugmoz 2006-11-20 14:59:32 PST
And for another anecdotal case: I would be hit by this issue every. single. day.

Does it really matter if 100,000 , 10,000, or 1,000 users are hit by this?  An e-mail application that fails to show some e-mail, and worse yet -deletes- it without chance of recovery after the user performs an operation (compacting) that is recommended by the authors?  Unacceptable.

Perhaps the powers that be do indeed deem this insignificant enough to warrant an immediate new release - but perhaps one of the following could at least take place:
- 1.5.0.8 pulled
- a warning placed on the download page
- a warning issued through the automatic updates (is this possible?)
- an advisory issued through select media; Slashdot, ZDnet, the usual.

This problem should, at the very least, not be silently lingering in this bug report (and its dupes) as if nothing is wrong, while users may be missing and completely losing e-mails, for no good reason, every single day.  It's bad enough that there's a whole page on "Missing Mails" as it is (why should mail -ever- go missing?), it's much worse that it does not address this issue.
( I notice it's a wiki - if anybody has editing rights; http://kb.mozillazine.org/Disappearing_mail )

Just my 2cts - sticking with 1.5.0.7 here.. I'll take my chances with the vulnerabilities.
Comment 59 Paul 2006-11-21 03:45:14 PST
I was alerted to this problem by my wife who noticed that the number of emails being downloaded appeared to be different to the number of new messages received. Messages coming from a Yahoo Groups! mail alias were being lost. She knows how to google to find the problem but was unable to resolve it (downgrading to 1.5.0.7 does require some skill and, more importantly, confidence).

So I believe that this bug could be affecting anyone using Yahoo Groups! and a lot of them may not realise. The impact of this bug could be devastating to the Thunderbird project and I vote for getting a fix out as soon as possible.
Comment 60 Chris 2006-11-21 07:46:23 PST
Please let me know what the proper means of escallation is if a person would like to see this fix released out-of-schedule.  Also, could you please list the name and email (I would have to open a yahoo / gmail account though) or contact method of the person who has the authority to release this fix out-of-schedule.  Perhaps they are have not been made adequately aware of the issue.  Thanks.
Comment 61 David :Bienvenu 2006-11-21 08:08:00 PST
Can you attach the headers of a yahoo groups message that disappeared? My yahoo groups messages look fine.

Cc'ing Jay to see if there's any way we can get an update out there sooner.
Comment 62 Jay Patel [:jay] 2006-11-21 11:30:38 PST
Comment on attachment 245652 [details] [diff] [review]
do a one time detection of this case, and regenerate summary files that appear to have this problem.

Approved for 1.8.0 branch.  a=jay for drivers, please land this asap.
Comment 63 bugzilla 2006-11-24 02:59:32 PST
Yahoo is sending an empty row (line) in the header,
mailman does bounce replies to the list when the send
from yahoo.
Are your shure that it is a problem with the email client
and not a bug made by yahoo? 
Comment 64 Steffen Wilberg 2006-11-24 03:28:21 PST
Builds which contain this fix are available here:
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-mozilla1.8.0/
Ignore the version number, which hasn't been updated yet, just look at the date. The fix was checked in on the relevant 1.8.0 branch on 2006-11-21 13:04 PST. So any builds made after that contain the fix.
Comment 65 Hadmut Danisch 2006-11-24 04:23:33 PST
Please see (and fix) bug #295208 , which is a similar or related problem.
Comment 66 Jose Hugaño 2006-11-24 13:42:33 PST
Well guys nice to see this fixed, but when will there be an update for 1.5.0.8 ?
You're not waiting till december, right ?
-Jose
Comment 67 Niek 2006-11-25 13:15:39 PST
.. i'm afraid it'll be in December. Unfortunately Mozilla isn't giving high priority to this issue, so it's probably time for all of us to look for a new mail client. Or step to the press and let them know - i'm pretty sure that would also speed up the fix :)
Comment 68 Frank 2006-11-25 15:13:08 PST
a well-known german press wrote in its online-news about this bug...i follow my previous posters there is no much time for the mozilla-team to fix this problem without loosing many users....this bug can cost thunderbirds image, i'm sure. Nobody wants to use a mailclient with the fear loosing his mails...

http://www.heise.de/newsticker/meldung/81529

btw. is there a german fix available?

regards frank
Comment 69 Rotis 2006-11-25 18:16:22 PST
(In reply to comment #68)
>
> btw. is there a german fix available?

Hi Frank

the fix is downgrading to TB 1.5.0.7 and deleting of all .msf files in your mailbox. TB will create them new and the hidden mails re-appear (as long you have not compacted your mail folders). Make a backup of all you mail database before doing this!

Here you can get the older TB:
http://releases.mozilla.org/pub/mozilla.org/thunderbird/releases/
Comment 70 Chris 2006-11-25 18:56:37 PST
What odd for me is that neither deleting the .msf files and downgrading to 1.5.0.7 or using the nightly build brought back the non-spam messages in my Inbox.  However the non-spam messages are still in the Inbox file itself, as I can see them by using mbox-viewer from http://sourceforge.net/projects/mbox-viewer to look at the Inbox file.  Is anyone else experiencing a similar situation?

Is there some other file I need to delete besides the .msf files?  For me, several hundred megs of non-spam still do not show up.  Thanks.
Comment 71 Kent S. 2006-11-25 18:59:16 PST
Re: Comment #68

You might try ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2006-11-25-12-mozilla1.8.0-l10n/thunderbird-1.5.0.8.de.win32.installer.exe

The time of the build indicates that it was compiled after the patch landed (even though it still says 1.5.0.8).

Of course, back up your mail data first, and then uninstall your present TB, and install this one (i.e., don't install this on top of your existing installation).
Comment 72 Chris 2006-11-25 21:34:02 PST
I followed your directions exactly but the non-spam still does not show up.  Thanks for giving me something to try though.
Comment 73 Chris 2006-11-25 22:18:24 PST
I found a way to bring the missing non-spam back:
qgrep -v -X "^X-Mozilla-" Inbox

Qgrep is a free downloadable Microsoft utility similar to the UNIX "grep" command that with the command line arguments above will remove the extra mail header lines.

The darn thing is that I will have to run this manually on all my mail files to make sure messages haven't disappeared that I don't know about.  But, at least I will have my old email messages back :)
Comment 74 Chris 2006-11-25 22:21:47 PST
Correction:
qgrep -v -X "^X-Mozilla-" Inbox > Inbox2
move /Y Inbox2 Inbox
Comment 75 Jose Hugaño 2006-11-29 06:39:45 PST
OK, folks, I can live with the December update, since I haven't compacted folders for a year or so and I have many folders with >300MB and TB is running smoothly.

But I'd really appreciate a procedure on how to proceed when the update is out:
* just  update ?
* delete the MSF files ? What do I lose by deleting the MSF files ?
* something else ?

Thanks,
Jose.
Comment 76 David :Bienvenu 2006-11-29 07:56:41 PST
most likely, just update - we do a one-time check of your .msf file to see if you have at least one instance of a message with identical message-id and reference headers - if so, we regenerate the .msf file automatically. The idea is that if you ever got a message like that before 1.5.0.8, you might have gotten more since 1.5.0.8.

If you delete the .msf file yourself, you will lose settings like the current sort order for the folder, but it's not a big deal - people delete .msf files all the time.
Comment 77 Henrik Skupin (:whimboo) 2006-11-29 08:25:12 PST
(In reply to comment #76)
> sort order for the folder, but it's not a big deal - people delete .msf files
> all the time.

I can't agree with this point. No-one deletes .msf files regularly except some developers. And you also lose any label you set before. So in my opinion it's a really big deal to come back (if you really can do that) to the former state.

It's nice to hear that we rebuild the summary file automatically. That is the only way for people who are just users of a software. Thanks.
Comment 78 Jose Hugaño 2006-11-29 09:05:23 PST
David, thanks for the info.

As for MSF file deleting/regeneration:
Does that mean I just lose sort order, BUT NOT read/unread/label flags ?
That is actually the most important thing for me. If I lose sort order,
I don't care, but if read/unread/label flags are lost, that is a big
problem for me and I bet for the majority of other users aswell.

Thanks,
Jose.
Comment 79 Henrik Skupin (:whimboo) 2006-11-29 10:18:17 PST
(In reply to comment #78)
> That is actually the most important thing for me. If I lose sort order,
> I don't care, but if read/unread/label flags are lost, that is a big
> problem for me and I bet for the majority of other users aswell.

The read or unread status won't be affected by deleting the mail summary file because it is stored inside the MBOX file (X-Mozilla-Status). But as I already described in comment 77 you will lose any of your labels. You can have a try. But remember to make a backup of the msf first. Afterwards you can restore the old state by copying it back.
Comment 80 RasmusHJ 2006-11-30 17:53:15 PST
Erh, I'm new to the bugzilla forum, so maybe I've missed something.  I spent a good few hours searching this and other relevant sites for information about this reference header problem.

However, I'm at a total loss here, what do I get if I install Thunderbird 1.5.0.8?  Information is abound, but not very consistent.  This report says the bug is fixed (albeit with regression); still the current version number has no indication of this - almost makes you wonder if there is something to hide?  The release notes do address the issue, but settles it with a recommendation to "not compact your folder" until the 1.5.0.9 release.  Clearly this indicates that you should not expect the bug to be fixed.

David wrote this in Comment #57:
(From update of attachment 245652 [details] [diff] [review] [edit])
this landed on the trunk and 2.0 branch today.

- does this mean a patch has been included in the current Thunderbird build (1.5.0.8)?

Sorry if what I just wrote is a boring drag of messed up information, but this is what you get when you search the forums and the official web site.  I won't ever dare to use rel 1.5.0.8, unless I am certain the reference header problem has been dealt with.  After all, having mails deleted spontaniously is just about the worst kind of error you can come up with in an email client - and for the Microsoft clan this is just the kind of story they'd love to have a good laugh at with their clients.  

Right now, it would be great to know that you don't risk loosing your emails if you install the current TB version.  If this is not possible, I think people should be advised to install 1.5.0.7 until a proper fix has been included in the release.

And thanks for providing a great open source program!

- Rasmus, who will never compact a folder
Comment 81 David :Bienvenu 2006-11-30 20:02:35 PST
The 1.5.0.8 release notes were changed to say that the bug exists in 1.5.0.8. The bugzilla status of FIXED means the bug is fixed in the latest code in our CVS repository. It has nothing to do with whether the bug is fixed in any particular release, though of course many many people get confused about that.  We use keywords to say which additional releases a fix will be in, hence the fixed1.8.0.9 keyword above.  We really don't expect end-users to use bugzilla in this way - it's really more for developers and testers. 
Comment 82 RasmusHJ 2006-12-01 03:01:01 PST
Oh no, I hoped this was not the case.  As I read your reply, the current available release may actually delete some of your mail (!), still it is being offered for download (!!).  

I understand this is not an end-user site - consider me a tester then ;-)  The reason for my being here, is that this is where it happens, and sadly I cannot rely on information about Thunderbird from the product page or the user forums. Did you know there are still moderator posts in the forums advisning people to compact folders in order to solve the problem?  Not new ones but still.  This is like a recipe for disaster if you are trying out Mozilla and installing a fresh copy...

For the TB project I really, really, really hope this bug does not lead to widespread dataloss among end users, as this way of patching-without-reversioning could lead to a major loss of credibility.  

I think, most people can accept that a system might be marked by some bugs, provided they can trust that at least their data is not being corrupted.  As was posted before, if loosing the data of an IT system is not a blocker, then what is?  Data loss corrupts confidence like nothing else.  Crash the system, freeze the computer etc., users will put up with this as long as you can 'fix it' and get back to normal.  But data loss?  Ouch.

Hope you understand my concern.  Personally, I will consider always running (at least) one version below current release, but this can hardly be a satisfying strategy for the Mozilla project.  Maybe some bugs are even worse (like?), but  several people have noted here that this kind of bug never occurred before.  With this in mind, I think maybe the development team should reconsider the strategy for addressing data loss bugs.  All said in great respect for a cool product made by some cool people ;-)

Regards!
Comment 83 Niek 2006-12-01 11:07:30 PST
Rasmus, i totally agree with you, and so do most people i think. For some reason the guys in charge of Mozilla don't think this is a very serious situation. They seem to take the Microsoft approach of shipping the patch on their pre-determined schedule, rather than fixing it right away. This MS late-patch thing was exactly the incentive that made me start using Firefox rather than IE, but now i'm sure i will at least not use Thunderbird anymore once i install Vista and get the new Windows Mail or Outlook. You're right: losing mail is probably the worst thing that an email program can do, and i would rate it much higher than the usual security issues, which you often rarely run into. Also, i think many users will get really upset once they do upgrade to 1.5.0.9, because suddenly a lot of mail will re-appear from out of nowhere (provided they haven't compacted their mailbox), making the problem very visible. So at that point i'm sure many users like you and me will lose all confidence in this product, and rightly so. I already wrote a story on slashdot.org about this in the hope that it would get some more publicity and users would be warned so that hopefully they could prevent their own mail from disappearing. Unfortunately it wasn't published, i assume because either it had already been talked about before, or because an opensource-fanatic was reviewing it. (A similar issue in a Microsoft product usually gets lots of attention there, so it does at least make you wonder.) Feel free to post your experiences there too, maybe you'll be more successful.

By the way: i have nothing against Mozilla - i've been a happy user of their products for at least a year, and of course it is to be expected that there are bugs in any software product from time to time. However, i think they handled this whole thing extremely poorly. I can't think of a situation where users wouldn't lose their confidence after losing their mail while the issue was known by the developers... :(
Comment 84 timeless 2006-12-03 06:25:27 PST
niek: go away.

this is not the way we work. you're apparently not actually permanently losing data, just not being able to see it. for this, i'm sorry. when you upgrade to 1.5.0.9, your problems should be automatically corrected.

note that the "forums" for the most part are not designed to delete things, including bad/outdated advice. we can't and won't "fix" problems by deleting references to them. just as we can't and won't remove your objectionable comment from bugzilla.

please be aware that releasing a new version of any product is complicated. and releasing a version of our branch releases requires us to have satisfied whatever security concerns we have, not just non dataloss bugfixes. if we release with the wrong set of security fixes, we get a lot of bad press and hurt our users substantially more.

politics of any sort do not belong in bugs. and hopefully you'll leave them in the bitbucket instead of tryiing to air them anywhere else.
Comment 85 bugmoz 2006-12-03 12:21:00 PST
"go away"?  That's a little harsh, isn't it?

This isn't a forum, so it's certainly not the right venue for either of you to vent your opinions on/of either side of the argument.  But I do share Niek's concern; I won't re-iterate why, just read above.

However, you do state that..
"you're apparently not actually permanently losing
data, just not being able to see it."

In the interest of the bug description's integrity it should be noted that although the direct effect of this bug does not cause data loss, effects from another common action taken does.  Compacting folders is a recommended practice by 'Mozilla', on pages at MozillaZine -and- is often the answer even by ThunderBird developers for the description of "I can't see my e-mail".
So stating that a person is not actually permanently losing data is based on the assumption that those affected by this bug would not compact their folders, which is an assumption which, given the aforementioned, is horribly flawed.

I think we all understand that updating any program with the scope of ThunderBird requires great care to be taken, including taking the interest of security at heart.  However, I'm not sure why you do fear bad press from any security issues but do not fear bad press from losing e-mail; In fact, 'ThunderBird' as a project is very lucky that the press has, so far, not picked up on this ( Tweakers.net aside ). Moreover, and back on topic, there have been other suggestions made that do not involve updating the program at all, but simply warn the users that they may currently be losing e-mail.  Is that truly too much to ask?  If not - who, exactly, do we ask who is (apparently) not monitoring this bug?  I suppose it is December and we may see the official update pretty soon anyway, making it a moot question.

Just my, last, 2cts in this bug / discussion.
Comment 86 Niek 2006-12-05 08:21:59 PST
Timeless Anonymous:

"niek: go away."

Always a great way of starting a discussion ;)


"you're apparently not actually permanently losing
data"

I'm sorry to say, but you apparently don't understand what this bug is about. Yes, i have permanently lost data, and so have many others. Please read this bug report (including its comments) before you write something like this, especially when you start your ramblings the way you started them. 


"when you upgrade to 1.5.0.9, your problems should be automatically corrected"

No. Not when you follow Mozilla's recommended practice.


"politics of any sort do not belong in bugs."

I agree, but i didn't see another way of getting the attention of the Mozilla's in charge of this situation, and i was trying to do something in order to prevent others from facing data loss. I think i (and others) pointed out some valid suggestions that would prevent alienating a lot of users down the road. This is in everyone's interest. Ignoring the end user is never a good thing. The simple fact that everyone who knows about this issue seems to have downgraded to 1.5.0.7 should tell you that this is generally considered a lot more serious than whatever security bug may have been fixed in .8, and so this bug can "get a lot of bad press and hurt our users substantially more", to put it in your own words. Especially by not warning about it. Also, this suggests that distributing 1.5.0.8 is probably not a thing that users would want if they knew the details, and distributing 1.5.0.7 would be the best thing to do until it's properly fixed. And you already have 1.5.0.7, so there's nothing complicated about that. Also, being a software developer myself, i fail to see why 1.5.0.8 can't simply be updated with just this fix, and shipped out. Give me one good reason why this hasn't been done.


"note that the "forums" for the most part are not designed to delete things, including bad/outdated advice. we can't and won't "fix" problems by deleting references to them. just as we can't and won't remove your objectionable comment from bugzilla."

Again, this shows you haven't read this bug report. But, could you tell me what you found so 'objectionable' about my comments? You know, even when you really like a product and are passionate about it (like you seem to be), that doesn't mean you shouldn't be allowed to criticize, in order to improve. Or does it? 
Comment 87 Richard Hearne 2006-12-08 09:08:59 PST
I am still experiencing this issue on nightly 1.5.0.9 2006-12-04-14-mozilla1.8.0-l10n

If anything it has become even worse.

Moderators are still advising people to compact their folders over in the forum.

I have lost all faith in Mozilla - you can no longer ensure the integrity of my mail, and you have shown contempt for your users by leaving this bug in the open. Shame on you Mozilla.
Comment 88 David :Bienvenu 2006-12-08 09:16:25 PST
"still having this problem" meaning new incoming messages with identical reference and message-id header's aren't getting displayed?
Comment 89 Richard Hearne 2006-12-08 10:00:00 PST
(In reply to comment #88)
> "still having this problem" meaning new incoming messages with identical
> reference and message-id header's aren't getting displayed?
> 
I was affected by the original bug in 1.5.0.8. I subsequently installed a nightly.

Sorry I cant answer that - all I know is that email is being downloaded, I receive notification that new mail has arrived, and I can find no new mail in any folder. This has hapened multiple times since installing the nightly.

If you can offer some guidance on how to test I will try to test properly. I am on a deadline at the minute and cannot test currently.
Comment 90 David :Bienvenu 2006-12-08 10:12:09 PST
I went back and verified that a 1.5.0.9 build handles parsing the Test5 folder attached to this bug, such that all the messages show up. That's pretty much the test case I have since I haven't found a way to get the problematic type of messages sent to me via a mail server. If anyone has subscription instructions for a mailing list that exhibits the problem, that would be helpful.

First, you should double check that you're really running the 1.5.0.9 build using help | about.

Next, if you haven't already, check that the mail isn't ending up in the junk folder.

Then, you should open your Inbox in a text editor, scroll to the bottom, and look for messages that you haven't seen in the message list - if you find any, you could copy+paste the headers and e-mail them to me, or attach them to the bug. You'll also see deleted messages in the text editor, but they'll usually have an x-mozilla-status header line with a value of 8 or 9 (the last hex byte will have the high bit set, indicating it was deleted)
Comment 91 Rene L. 2006-12-17 06:11:57 PST
Can anyone confirm that this bug has been fixed in the official TB 2.0 beta 1 (20061206)?

I already wrote an article on all the big improvements, but don't want to recommend testing the new beta when users could loose their mails.
Comment 92 Magnus Melin 2006-12-17 08:07:08 PST
Verified fixed using todays nightly 2.0 branch build. (And yes, beta1 is ok.)
Comment 93 Jose Hugaño 2006-12-17 14:48:58 PST
Any update update on this for TB1.5 folks ? It's mid-december!
Thanks,
Jose
Comment 94 David :Bienvenu 2006-12-17 14:55:03 PST
there was a delay in 1.5.0.x - doing a release is a very complicated, time consuming process, but our release team is working hard to get it out ASAP.
Comment 95 Niek 2006-12-17 17:08:03 PST
David: why couldn't you guys update 1.5.0.8 with *just this fix*, immediately after you made it? The way you stick to your update schedule seems worse than Microsoft's patch-tuesday scheduling. This episode has really made me dislike this product, when to me it seems that it could have been so easily prevented. Not only did i lose trust in Thunderbird after permanently losing mail, but i now also lost trust in the way Mozilla handles these kinds of issues. What's your opinion? It'll be interesting to see what happens when the update does arrive, and users suddenly see their inbox stuffed with mail from a month ago.
Comment 96 test 2006-12-19 14:34:46 PST
1.5.0.9 is availiable:

ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/1.5.0.9/
Comment 97 Jose Hugaño 2006-12-20 00:30:04 PST
This problem is not mentioned in the release notes ?
http://www.mozilla.org/projects/security/known-vulnerabilities.html#thunderbird1.5.0.9
So I assume this bug is not fixed, right ?
Comment 98 Magnus Melin 2006-12-20 01:34:12 PST
It's fixed. (Guess it falls under "Improvements to product stability.")
http://en-us.www.mozilla.com/en-US/thunderbird/releases/1.5.0.9.html
Comment 99 Jose Hugaño 2006-12-20 04:06:38 PST
Thanks Magnus!

Now I'd be most grateful, and I'm sure a bunch of other users too,
if someone (preferably from the Mozilla team) could post a brief
"guide" for upgrading to 1.5.0.9:
I.e. is it possible to just upgrade and possible hidden mails
just appear ? Or what must we do to make possible hidden mails
appear ? I'm sure, deleting the MSF files is no option for many
users, since the flags are lost, so is there a way without deleting
the MSF files or is this handled automatically ?

Thanks a lot!
Comment 100 Richard Hearne 2006-12-20 04:16:40 PST
Just upgraded

No new mails appeared. I know that I have suffered from this problem and I have not compacted folders for some time (since I realised that compacting deleted the hidden mails).

Please advise on method for retriving hidden mails.
Comment 101 David :Bienvenu 2006-12-20 06:54:21 PST
Try deleting your inbox.msf file - but if no messages appear, then you weren't actually suffering from this problem...
Comment 102 David :Bienvenu 2006-12-20 06:56:23 PST
the flags aren't lost when deleting the .msf file, unless you've got e-mail migrated from an other e-mail client with TB 1.0 and you've never ever compacted your folders with 1.5
Comment 103 Chris 2006-12-20 18:17:36 PST
One other thing to try:  I had the same issue, I chalked it up to possibly a weird side effect of the bug (?) with normal usage of the corrupt Inbox.  Since then I fixed it as per message #73/#74: https://bugzilla.mozilla.org/show_bug.cgi?id=360409#c73 .  Qgrep is available if you go to microsoft.com and type in "qgrep" in their search box.  Make a backup copy of your Inbox first!  Of course, this is a shotgun approach which deletes some other message settings and brings back all other deleted mail too.
Comment 104 David :Bienvenu 2006-12-20 18:33:24 PST
what does that Qgrep statement actually do? I don't see how it could be related to this bug since it seems to deal with x-mozilla-status lines...
Comment 105 Chris 2006-12-20 18:56:33 PST
It was something I tried that worked for me after I upgraded to the daily build that had the fix applied, and manually deleting the .msf files didn't bring my regular mail back.  The intent on my part was that if the invisible message got marked as deleted (one of the X-Mozilla- line features) somehow unknown to me after they became invisible (don't know how but figured it might be a side effect - with previous normal mail usage in the buggy version / other message range deletions?).  Unfortunately my Inbox was way too big  (hundreds of megs) to view/edit with Microsoft text-based tools, so I couldn't do further analysis, but it did fix the issue for my Inbox so I offered it up as a suggestion if all else failed.
Comment 106 Richard Hearne 2006-12-20 20:45:48 PST
After rebuilding the .msf files I still have no new mail re-appearing. I was under the impression that this fix would rebuild my mailboxes and automatically fix this issue? 

I am quite certain that I have been affected by this bug. I am quickly losing confidence in TB. I do not feel that I should have to use other tools to affect the required changes.

I will try to manually look through the inbox files to look for messages that may be affected by this and report back here.

Rgds

Richard
Comment 107 Kent S. 2006-12-20 20:54:52 PST
Re: Comment #106 ... did you ever compact your folders with TB 1.5.0.8?  That is the kiss of death.
Comment 108 David :Bienvenu 2006-12-20 21:46:37 PST
Unless you actually have evidence of receiving messages with common reference and message-id  headers, you don't know that you've been affected by this bug. 
Comment 109 Richard Hearne 2006-12-21 00:11:13 PST
(In reply to comment #108)
> Unless you actually have evidence of receiving messages with common reference
> and message-id  headers, you don't know that you've been affected by this bug. 
> 

I was affected by 1.5.0.8. Messages re-appeared on downgrade. Since moving to nightly I have seen a variance between reported new messages in system tray Vs actual new messages in TB.

I did compact previously (as per the dreadful advice still being given in the TB forums). But I have not compacted for many weeks now to avoid any further potential loss.

Is there any simple way to check for common headers without manual inspection?
Comment 110 David :Bienvenu 2006-12-21 08:49:20 PST
Manual inspection is the only way to be sure that I can think of, other than writing some fancy perl script...
Comment 111 Hannes Mayer 2006-12-21 10:50:46 PST
Some fancy perl script ?
Here is my quick hack and I already detected some hidden
mails here (well, all but one are SPAM anyway):
http://www.captain.at/howto-thunderbird-mailbox-checker.php
Comment 112 David :Bienvenu 2006-12-21 10:59:43 PST
cool - note that the resent-message-id is irrelevant - Thunderbird ignores it.

Where was the non-junk message from?
Comment 113 Hannes Mayer 2006-12-21 14:25:21 PST
(In reply to comment #112)
> cool - note that the resent-message-id is irrelevant - Thunderbird ignores it.

ACK - I will modify the script.

> Where was the non-junk message from?

I just verified it and it is actually a false detection.
It is a return receipt, where the original mail headers are attached.
I try to circumvent this, but better have a few false detections than
reviewing a half-gig mailbox by hand.

Comment 114 Hannes Mayer 2006-12-21 14:48:29 PST
Thunderbird Mailbox Checker V0.2 was just released:
http://www.captain.at/howto-thunderbird-mailbox-checker.php

Changelog:
V0.2: removed check for Resent-Message-ID since it is ignored by TB. Added detection of end of header to avoid false detections at i.e. "return receipts" where the header of the original email is attached.
Comment 115 Niek 2006-12-21 17:02:51 PST
Hi Hannes,

> Here is my quick hack and I already detected some hidden
> mails here

So do you mean that the updated Thunderbird (1.5.0.9) still suffers from the disappearing message issue? Or did you write the script for 1.5.0.8 only? In other words, should i update from my current 1.5.0.7 (which i downgraded to) to 1.5.0.9, or not yet? I want to make triple sure this time that my messages won't be disappearing again. Richard still seems to be having problems?

Thanks,
-- Niek
Comment 116 Hannes Mayer 2006-12-22 00:14:16 PST
(In reply to comment #115)
> Hi Hannes,
> 
> > Here is my quick hack and I already detected some hidden
> > mails here
> 
> So do you mean that the updated Thunderbird (1.5.0.9) still suffers from the
> disappearing message issue? Or did you write the script for 1.5.0.8 only? In
> other words, should i update from my current 1.5.0.7 (which i downgraded to) to
> 1.5.0.9, or not yet? I want to make triple sure this time that my messages
> won't be disappearing again. Richard still seems to be having problems?

Hi Niek!

I don't know...
I have not tried 1.5.0.9 yet. I will probably clone my Inbox (with the
SPAMs that have those headers and are hidden) and give it a try with
1.5.0.9 on another machine. I keep ya'll posted about the result.

My script is only to check mailboxes for mails with problematic headers.
It is basically TB-version independent. You can run the script on any
MBOX, whether it is a TB mailbox or not. The script just shows basic
information about emails with equal "Message-ID" and "References" headers.


Comment 117 bugmoz 2006-12-22 01:49:36 PST
For what it's worth, I had written a similar script (in a whole different language, though), and since 2006/Nov/07* I would have lost 792 e-mails.  The vast majority of those would be from our mailing system, 29 of them however woulc come from outside mailing systems.  8 of which came from a Packard Bell mail server, which seems to be fixed since at least 2006/Nov/19.
(* Note that this excludes e-mail received since 2006/Jan/01.  E-mail prior to that date is archived away and no longer in the mailbox files, so wouldn't be affected.)

I'll be installing 1.5.0.9 shortly.. if there's still some manner of issue, I'll find out soon enough - the hard way :)

I'd like to express my disappointment that this issue is not explicitly mentioned in the release notes, though.  Although I am confident that it is addressed (why claim otherwise?), it does mean that the general public will remain unaware that they may have lost e-mail due to this bug.  Perhaps it is better that way.  Time to get paranoid yet?
Comment 118 bugmoz 2006-12-22 03:14:42 PST
looks fixed here
Comment 119 Hannes Mayer 2006-12-22 03:30:55 PST
(In reply to comment #118)
> looks fixed here

Richard,

do you mean that the hidden mails automatically appeared after
upgrading to 1.5.0.9 ?

Comment 120 bugmoz 2006-12-22 03:58:30 PST
I should have made myself clearer :)

I don't know if it makes the previously invisible e-mails visible again, as I have been using 1.5.0.7 while waiting for a bugfix.

However, none of the e-mails that would automatically be 'invisible' with 1.5.0.8 are invisible now.

--

What I was referring to, however, was more with new malformed e-mails coming in not automatically disappearing - which was my primary concern.
Comment 121 Hannes Mayer 2006-12-22 06:04:37 PST
Richard,

thanks for the clarification.
I guess we can conclude:
* bug fixed in 1.5.0.9
* hidden emails in 1.5.0.8 do not automatically appear in 1.5.0.9

As for hidden mails I'd suggest the following (without losing
any flags or whatsoever by deleting the MSF file):
In 1.5.0.9: Just copy the affected mailbox and give it another name.
So it will have no MSF file, and TB 1.5.0.9 will rebuild the MSF for
the "new" mailbox.
Look for the emails detected with
Thunderbird Mailbox Checker
http://www.captain.at/howto-thunderbird-mailbox-checker.php
in the "new" folder and move them to the original mailbox.
WARNING: Only try this with 1.5.0.9! TB 1.5.0.8 will hang when moving
such mails to the original mailbox.
Comment 122 Ralf Siegel 2007-04-22 08:24:57 PDT
This bug does still exist in TB 2.0.0.0. 

http://forums.mozillazine.org/viewtopic.php?t=542398

When I move (or copy) certain messages manually or by means of a filter from the inbox to a subfolder, sometimes these messages disappear, sometimes not.
Comment 123 E. Toering 2007-04-27 01:48:45 PDT
I use Windows 2000 And discovererd this bug in 2.0.0.3

It seemd (but I am not 100% sure) that the new Thunderbird viewed some messages as spam, I uncheck the spam-marker and put the mails back in the Inbox.

Rusult the mails were gone!?

These were importand mails for me.. it scared the hell out of me, that mails could get completely lost.. I made a backup of my inbox and looked for the lost mails.

They were still there then I did a compact folders => the mails were REALY gone. Then I restored the backup, resulting in a rebuild of the MSF-file, and the mails were back!

I thought returning to my previous version (1.5.0.4) would solve the problem, but what I have read here, it looks like the problem may exist in that version also? :( I will keep an eye out, checking if mails in this version (1.5.0.4) also get lost..

My preliminary conclusion, there is something wrong with the msf-file-structure...

Concluding.. reading this thread, it looks like this problem had been going on for some time, without beeing solved.. this scares the hell out of me, I have an internet company, and need to be able to rely on mails NOT disappearing :(

This realy needs to get resolved! I have been a loyal user since Netscape Messenger 4.something, but this isn't just a bug, this bug is huge and verry scarry!
Comment 124 Henrik Skupin (:whimboo) 2007-04-27 06:27:46 PDT
What you see is bug 368112. This is already fixed in current branch nighlies.
Comment 125 E. Toering 2007-04-27 07:45:15 PDT
(In reply to comment #124)
> What you see is bug 368112. This is already fixed in current branch nighlies.
Do I read it correctly that this should be fixed with 2.0.0.4?
Comment 126 Henrik Skupin (:whimboo) 2007-04-27 10:39:47 PDT
(In reply to comment #125)
> > What you see is bug 368112. This is already fixed in current branch nighlies.
> Do I read it correctly that this should be fixed with 2.0.0.4?

Yes, TB 2.0.0.4 will be the next official release where the issue from bug 368112 will be fixed. So let's stop us here.
Comment 127 Daniel Veditz [:dveditz] 2007-04-27 17:51:00 PDT
I don't think it's been decided it will be called "2.0.0.4", it might be time to let Thunderbird go it's own numbering way and call it "2.0.0.1" since 2.0 was just released.

But yeah, the "next release" will have the fix.
Comment 128 E. Toering 2007-05-31 01:15:55 PDT
Question, it is said here that with version 2.0.0.4 this problem would be fixed. Where can I see if this problem has been fixed in the latest version, I can only find version 2.0.0.0 ? 
Comment 129 Magnus Melin 2007-05-31 02:37:57 PDT
This bug is fixed in 2.0.0.0. The other issue - bug 368112 will be in the next point release. For pre-release builds see bug 368112 comment 52.

Note You need to log in before you can comment on or make changes to this bug.