Closed Bug 81141 Opened 24 years ago Closed 16 years ago

[RFE] Ability to rebuild msf files on demand

Categories

(MailNews Core :: Database, enhancement, P5)

Product:
MailNews Core
Component:
Database
Type:
enhancement

Tracking

(Not tracked)

Status:
VERIFIED WONTFIX

People

(Reporter: gandalf, Assigned: Bienvenu)

References

Details

(Whiteboard: [add-on idea])

From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux 2.2.19 i586; en-US; rv:0.9+) Gecko/20010514 BuildID: 2001051415 Maybe, will be a good idea to give mozilla mail client an option to build msf files again. I dont know, how exactly works the option "compact folders", but msf file is an index of messages in folder. And when this index is broken (por example - crash of client) - is necessary to build a new index. At this time I know only the way of close mail client, erase broken msf file by hand and re-run the mail client. Reproducible: Always Steps to Reproduce: 1.Give to mozilla mail client an option to re-build the msf file for actual mail folder. 2.Give to mozilla mail client an option to re-build the msf files for all folders. 3.
Marking NEW.
Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Linux → All
Hardware: PC → All
Summary: Possibility to build msf files again → [RFE] Ability to rebuild msf files on demand
I've got a workaround for this, or at least to make mozilla rebuild all the indexes once you've deleted them. steps: 1. delete *.msf, easiest to do with a find, if you've got nested mailboxes as I have. 2. do a mailnews search on some random string as subject line. go and make a cup of tea ( ;) ). when you get back, all your .msfs will have been regenerated.
setting priority
Status: NEW → ASSIGNED
Priority: -- → P5
*** Bug 152869 has been marked as a duplicate of this bug. ***
*** Bug 106246 has been marked as a duplicate of this bug. ***
*** Bug 211736 has been marked as a duplicate of this bug. ***
Does this bug also handle the rebuild of msf files for news or has this to covered by another bug?
for news, it's just a matter of deleting the .msf file - it will get rebuilt when you click on the newsgroup. So, yeah, we can let this stand for that.
But the rebuilt .msf file for news does not reflect whether postings were already downloaded for offline use. This is something what I fought was subject of this bug to compare .msf file and mail/news database and rewrite the .msf file with the actual available datas.
that's a different issue which probably deserves its own bug - something like including the offline store information in a regenerated .msf file for a newsgroup or offline imap folder.
Filed bug 230213 for above mentioned feature.
Sometimes I have to compose a message, choose [Send Later], and edit the "Unsent Message" file with a text editor myself, due to Mozilla's limited functionality. (Ever tried to attach a text file to a message body where they have different charsets? Ever wanted to edit MIME's Content-* headers?) With Netscape 4, it was not a problem because after I edited the "Unsent Messages" file myself Netscape detected that the file was newer than the .snm file and automatically regenerated the .snm file, keeping it up-to-date. Mozilla doesn't seem to detect that the mail file has changed. It goes on with the out-of-date .msf file when actually sending the messages, so I have to make .msf file regenerated. It can be done by deleting it, but the problem is that as long as Mozilla is running it holds the .msf file so that it cannot be deleted. I have to close every Mozilla window (not only the mailnews window but the entire program), delete the .msf file, and run Mozilla again to make it regenerate the .msf file. This is very cumbersome. I wish Mozilla would be wise enough to detect that the mail file had changed after the corresponding .msf file had been built, or that at least it would regenerate the .msf file on demand.
*** Bug 269916 has been marked as a duplicate of this bug. ***
Product: MailNews → Core
That works for the .msf files, but if you have folders with files in them, the folders still aren't visible as you need to regenerate files with the same names as the folders. So that would need to be addressed as well. I have just spent a reasonable amount of time rebuilding a users mail structure because of some corruption that occurred, specifically caused by System Mechanic wiping a whole heap of zero byte files and other files it deemed to be junk, which were in the users mail store only for the purposes of making folders visible. It was user error, sure, but somehow along the line the index files were left in an extremely inconsistent state and no automatic rebuild happened, so we were left with a heap of invisible mailboxes (visible in explorer, but invisible in TB) which TB didn't rebuild indexes for. Wiping all the .msf files within explorer and triggering a full rebuild manually would have been good, especially if it also had recreated the right files to make the folders visible. It's probably the subject of another bug report that TB even allowed all these files to be deleted while it was open (let alone user error approving the delete in the first place), but the ability to recover from other bugs is fairly critical. A maintenance option of something like "clear all indexes" and "rebuild all indexes" from within TB would be very handy. At least until we can be absolutely certain there are no bugs whatsoever in the store (which we can't ever be, realistically).
This has been being kicked around since 2001 when Linux kernel 2.2.19 was considered not to be laughed at!?!?!?!? Come on guys, this is most disappointing. What needs to happen for this to get attention and a PROPER resolution. The solution needs to be simple, one process (menu selection, or other) to Compact, rebuild .msf files, etc... What can I do to help make this happen?
In 2.0 - bring up folder properties, click rebuild index.
But that rebuilds the index for that folder ONLY. Could we please stop ignoring this deficiency and suggesting features that are not a total solution for this RFE.
You'll notice (or actually, you didn't notice) that I didn't mark the bug fixed.
I did notice that the ticket remained open. I also noticed that this was first reported in 2001... six years ago! Should I assume that mailbox DB corruption is not a "cool" enough topic to warrant a solution be developed? The hard work of actually developing technology to recover mailbox DB corruption has been developed. It need only be extended to automatically process all folders for one account, or for all accounts. It seems a simple enough request, so why has it not gotten done?
Running Windows Disk Defrag on the drive containing the TBird data files results in a few folders rebuilding their MSF files / need to click the folder before new mail filtered to the folder will turn the folder text BOLD / etc... I made a backup of the TBird data files before I ran disk defrag, closed all running programs, and reopened TBird after the defrag had ended. On a few folders, it was as if I was clicking on a folder without an MSF file: It built an index upon clicking the folder, and my sort order was forgotten. Just now I discovered by clicking on a folder that has a filter associated with it, that messages had been filtered to that folder since running defrag. The index was rebuilt, and four unread messages "magically" showed up. So... anticipating the need to "click each folder" yet again. gggrrrr.....
Since migrating to TBird, I have been most annoyed that there is no "re-index all folders" option. However, last night I discovered if I do a search for messages starting with the name of the account (root of the account) and search all sub-folders, it has the added benefit of rebuilding .msf files when it encounters one that it does not like. That in turn makes messages in those folders that have been filed there by a filter suddenly show up as unread, and once again the folder indicates that unread messages are within.
This is something I really need right now. I've been having big trouble with Tbird having to rebuild summary files to let me go to a folder, so I have to do a search to force it to rebuild the summary files. I also have 14 POP accounts so sometimes I need to do the search 14 times to fix them all. It would be real nice to be able to just tell the program to rebuild all of them while I go grab a bite to eat.
QA Contact: esther → database
Product: Core → MailNews Core
In TB 3 RC1, the trick of doing a search no longer rebuilds the summary files. As far as I can tell, there is now no way at all to trigger a rebuild -- besides manually clicking on each folder that needs it. If one's profile has many folders, that's a problem. (Merely clicking on the bottom or top of the list and then holding down an arrow key doesn't even work, since the system doesn't keep up, and many folders are skipped.) Can we have some kind of real fix, e.g. the "reindex all" option or button mentioned above. By the way, there is documentation on the web that claims that Tbird automatically rebuilds missing msf files on start-up, but as far as I know, that has never been true, at least as long as I've been using the program (about 4 years now).
This is either WFM or INVALID. The "rebuild a single folder's index" case has been covered since Thunderbird 2.0 as previously noted. The "rebuild all folders' indices" case is extension fodder; there is no justification for it in core given the reliability of msf files in Thunderbird 3.0 and the destructive side-effects of rebuilding on IMAP offline stores. If a user constantly finds themselves needing to have their .msf files rebuilt, it suggests either a (different) bug in Thunderbird or badly behaved system software that is improperly touching the timestamps on files. Thunderbird support should likely be consulted in those cases: http://www.mozillamessaging.com/en-US/support/ Given the bug's subject only covers the singular case, marking WFM.
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Resolution: --- → WORKSFORME
I share my profile between my laptop and my desktop machine, which means my msf files need to be rebuilt after copying. (Copying the msf files themselves does not work.) It would be nice to have a better way of doing this, but I haven't seen one, and at some point I did my best to look for suggestions in the various forums and bug reports. What has been reliable is to do a search through the folders on first starting up on the new machine, which rebuilds all necessary folders. Without even this possibility, there is a problem. Of course, the real problem is how to sync the files between the two machines, but there seems to be no other solution for that. (I do not want to keep all my mail on the imap server, and I file things into the various folders quite regularly.)
I also don't see how this meets the criteria for either "resolved" or "worksforme".
You have a very unusual use case which Thunderbird is not designed to handle. I would strongly suggest just keeping your mail on the IMAP server. Failing that, I would suggest moving to use of local IMAP servers; there are various tools that can synchronize IMAP folders/accounts available. We use WORKSFORME in Thunderbird/MailNews to indicate that the bug has been fixed but that we don't know what bug fixed it (or some other bug fixed it). I am unclear if the bugzilla link used to agree with us but got clobbered by the recent bugzilla upgrade or if this has always been a Thunderbird/MailNews-specific idiom.
Thank you for your reply. I am not sure how unusual my case is, however. A few minute's googling brought up others with the same situation: http://groups.google.com/group/mozilla.support.thunderbird/browse_thread/thread/89a1e821c2939845/c62868a8c0ced6b5?#c62868a8c0ced6b5 Furthermore, independent of my particular reason for caring about this, the web is full of pages -- including http://kb.mozillazine.org/Disappearing_mail -- that advise the mass deletion of msf files as a troubleshooting step, and reassure the reader that they will be rebuilt upon restart. Since they're not, it would be very useful for the users who follow this advice to have a way to mass-rebuild. As I noted in my message above, the only "unofficial" way to do this appears to have disappeared. Could you please reactivate this bug, even if nothing is going to be done about it in the near future.
The original issue of this bug it seems to me was to rebuild a specific folder, and as time went on it evolved to the broader issue of rebuilding many or all msf after failures in thunderbird which are now largely fixed in version 3. Hence, WFM is appropriate. unfortunately, mass deletion of msf is "easy advice" (too easy) to give out. It is mostly a bandaid over a broken bone, and simply defers solving the real problem. The disappearing mail issue for example has several fixes in version 3. It is certainly likely that not all those problems are gone, but the best route to address each issue is head on in a bug. As uncaring as it sounds, giving the average user a tool to wipe out all msf is not the preferred direction. Your issue is a specific use case and a worthy need, but as Andrew suggests could be solved via creating an extension for that use case. If you were to file such a bug, which you are welcome to do, using andrew's reasoning it would likely be closed WONTFIX.
Status: RESOLVED → VERIFIED
p.s. http://kb.mozillazine.org/Disappearing_mail addresses many possible causes, many of which don't involve any form of bug in a current version of thunderbird. And if after going through that process one concludes the fault is thunderbird, it unfortunately lacks any advice to the user about really resolving the issue - one first step of which might be to file a bug.
"As uncaring as it sounds, giving the average user a tool to wipe out all msf is not the preferred direction." Sorry -- I wasn't asking for that! Rather, we need a tool to trigger rebuilds of msf files in more than one folder. Since you do have such a tool on a per-folder basis, there can't be a moral objection to providing a way to do it for groups of folders....
"p.s. http://kb.mozillazine.org/Disappearing_mail addresses many possible causes, many of which don't involve any form of bug in a current version of thunderbird. And if after going through that process one concludes the fault is thunderbird, it unfortunately lacks any advice to the user about really resolving the issue - one first step of which might be to file a bug." Well it does say "It's possible that the ".msf" files (index files) are corrupted. To rebuild the index of a folder, right-click it, select Properties, and choose "Rebuild Index" from the General Information tab. You can also close Thunderbird and manually delete them from your profile folder; they will be rebuilt when Thunderbird starts." Similarly at http://kb.mozillazine.org/Compacting_folders: "If the corruption is mild you frequently can fix it by deleting the .msf files for the corrupt folders. [...] Delete all of the files ending in .msf in your profile. Thunderbird will recreate them when it starts." -- except they are NOT rebuilt when Thunderbird starts, no matter what all these pages say (and there are more of them, I think). And there's no way to trigger the rebuild except by acting on each folder individually.
Here we go again... My take on this bug. 1) Delete all of the IMAP comments. IMAP is a different animal. 2) Folders with corrupt MSF index files do not show new mail when a filter files messages to said folder until the index is rebuilt. aka a black hole problem. 3) I have bazillions of folders in many accounts. I ain't gunna right-click each folder and rebuild indexes EVER! 4) I found that doing a search for "zzzzzz" or some other nonsense string would infact rebuild indexes on-the-fly at the account level, so use that method from time to time to make sure all indexes are behaving properly. Since moving to Linux and the XFS filesystem I am having far less problems than when I was on Windows. Plain and simple, DO NOT TAKE AWAY the "search rebuilds indexes" functionality in Thunderbird without adding a global "index check" utility. DO NOT!!! A change in architecture that was a "bad decision" was made in FF3 - wish they had never done it. I see comments along the lines of, "too late to undo the change now, have to live with it." Do not make a similar mistake with TB, please.
Is there any update on this open issue which TB 3 has only made worse, not better? Obviously since TB 3 shipped shortly after my previous post, the developers indeed decided to sink the dagger though "using search to rebuild corrupted msf index files" and I am refusing to upgrade beyond TB 2.x series until some solution is provided to this.
Michael: you would be better off spending your time trying to help fix the causes of msf corruption (by figuring out when it happens, steps to repeat, etc) than trying to get implement, what is at best, a very bad workaround.
Mark, you miss the point. There needs to be a way to globally verify that all msf files are valid / correct. It is not a good idea to try to build something so good that you do not implement a verification function for that which has been created. Filesystem developers include utilities to verify the disk is in a consistent state. The same needs to exist in ThunderBird to verify msf files.
But you can't necessarily know that your problem lies with msf file corruption (or deletion) until you try rebuilding them! And you've taken away the ability to do that, except on a folder-by-folder basis. Unless there's some huge but secret programmer's obstacle to returning this function to Thunderbird, this conversation strikes me -- with all due respect -- as a bit nuts. It's like saying "yes we could give you lifeboats, dear passengers of the Titanic -- we could even give you access to hull-repairing technology. But we won't do that, because we feel your time is better spent figuring out whether the hole really came from the iceberg, what kinds of icebergs are most likely to sink your ship." Yes, people might want to know those things too, but first and foremost we want to be rescued, so we can get back to work.
(In reply to comment #35) > Is there any update on this open issue which TB 3 has only made worse, not > better? Why are you looking for an update on a closed bug?
(In reply to comment #39) > Why are you looking for an update on a closed bug? I do not see where this bug has been closed. Status is set to verified, not any of the closed statuses. Thus confused at your comment.
(In reply to comment #40) > I do not see where this bug has been closed. Status is set to verified, not any > of the closed statuses. Thus confused at your comment. In contrast to many other Bugzillas, Mozilla's bugs don't often get set to a "closed" state. Here, "resolved" essentially means what "closed" commonly means elsewhere. RTM: https://bugzilla.mozilla.org/page.cgi?id=fields.html#status
(In reply to comment #41) > Here, "resolved" essentially means what "closed" commonly means > elsewhere. But this one is set to "verified" not "resolved". > RTM: https://bugzilla.mozilla.org/page.cgi?id=fields.html#status I checked that link before asking comment #40. So, since there is no global way to verify msf files in TB 3, shall I open a new report and come back to this report to report the new number we are switching to... or can someone simply flip some status bit on this one that work still needs to be done to successfully resolve the situation?
Please do not open a new bug, it will be closed RESOLVED WONTFIX as per my note in comment 25 that this is the domain of an extension. Because there seems to be some semantic ambiguity with my original choice of WORKSFORME, I am changing the resolution of this bug to WONTFIX to help eliminate any ambiguity.
Resolution: WORKSFORME → WONTFIX
I find these negative responses surreal. The argument that "there is no justification for it in core given the reliability of msf files in Thunderbird 3.0" flies in the face of the fact that the msf files are demonstrably *not* reliable (http://www.google.com/search?client=safari&rls=en&q=%22thunderbird+3%22+msf+problem) -- plus the fact, documented in comment 33, that lots of Thunderbird help pages STILL tell users to solve problems by deleting them -- and reassure readers (wrongly) that "they will be rebuilt when Thunderbird starts". If you must, say that this request is too hard to take care of, or will take too much time. If I have to live with that, I will. But the claim that there can't *possibly* be a problem worth fixing because Thunderbird's MSF files are just too reliable to fail, and the idea that users with MSF file problems should spend their time tracking down the source of the problem rather than accessing a tool that might get them working again...those are very strange arguments for inaction.
The internet is wrong all the time. I have made my peace with it. If you have reproducible test cases that cause msf corruption using the current Thunderbird/Lanikai 3.1 nightlies, please file a new bug with the steps to reproduce so that we can fix the bug. All of our decisions are cost/benefit decisions. My impression of consensus from the rest of the (small, busy) development team is that we would rather fix any problems that make msf files unreliable (or replace them with something better) rather than pour all of our effort into the world's greatest chkdsk* implementation. * I am intentionally choosing a built-in disk checking program that many people abandoned in favor of third-party solutions**. I am also intentionally chose a disk-checking program for a file-system whose use is not a great idea versus something with journalling. ** I basically mean Norton here. Which is potentially ironic since the name is now mainly associated with anti-virus software, and anti-virus software causes more problems for Thunderbird and its users than Thunderbird does***. *** I'm not saying that actual viruses would not also be problematic for the users, but that's not really on Thunderbird.
(In reply to comment #45) > I am also intentionally chose a disk-checking program for a file-system whose use is not a great idea versus something with journalling. In my defense, I was eating soup and doing some other things when I wrote this. My soup is all gone now though :(
" My soup is all gone now though :(" So's my e-mail... (well, really)
I meant, *not* really. OK, bye.
(In reply to comment #45) > If you have reproducible test cases that cause msf corruption using the current > Thunderbird/Lanikai 3.1 nightlies, please file a new bug with the steps to > reproduce so that we can fix the bug. That is an absolutely ridiculous position to take! Thunderbird needs a seat belt rather than a dose of a Snapple bubble wrap commercial. It does not make good sense to develop a file format without a verification tool. Why is it so undesirable to develop a global msf verify function? "Ostrichism!!!" Perhaps stick it in the Advanced tab of Preferences. I am very disappointed in the position being suggested on this ticket. I think it is time to research the next cross-platform email client once again. :-(
just a quick note from a tbird 2 user (ok, my main reason was that from the 5 extensions i used 4 did not work with tbird 3...). argument that tbird 3 is great with msf files is bogus. you have to live with the legacy you have, and many, many users have been backing up their email by copying thunderbird directory over. you can not guarantee that all of those backups have perfect indexes. as such, when restoring from such a backup it is an extremely valuable ability to rebuild all msf files, just in case. restoring from old backups isn't something that will just go away in a few months either - personally, i have backups from my previous workplace that i do not keep in my main mailstore. if i ever need something from those backups (has happened a few times), i restore them to some folder & then search them. i have no idea in what state msf files for all the gazillion folders are in there, and i'd bet that at least a couple of them are actually corrupt.
Maybe I misunderstand, but I don't think anyone said TB3 is not without problems with msf files, or that it is even great. (although for me it is certainly better than TB2) I think Andrew's reasoning is quite sound - it is preferable to fix underlying problems that help everyone than give a bandaid that's only for people who know how to use it. Even though it's slightly more work, the enthusiasm shown in this bug might be used to kill some of the known bugs, and file new bugs for issues that arent' yet identified. Some are probably found in this list: https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=anywordssubstr&short_desc=index%20msf&field0-0-0=short_desc&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&resolution=---&classification=Client%20Software&classification=Components&chfieldto=2009-01-01&chfield=[Bug%20creation]&query_format=advanced&short_desc_type=anywordssubstr&type0-0-0=nowords&value0-0-0=count%20counts&component=Backend&component=Database&component=Folder%20and%20Message%20Lists&component=Mail%20Window%20Front%20End&field1-0-0=short_desc&product=MailNews%20Core&product=Thunderbird
Whiteboard: [add-on idea]
Yes of course it's preferable to fix underlying problems -- but frequently users have one-off problems that are not likely to get serious attention, because there aren't tons of people clamoring for a fix. The ability to expeditiously bulk-rebuild msf files can get such users back to work.
Hee hee hee... I just happened across this in the TB KB: http://kb.mozillazine.org/Compacting_folders#Quick_and_dirty_fix And I quote from the page: "Delete all of the files ending in .msf in your profile. Thunderbird will recreate them when it starts." Please, TB could greatly benefit from a "Global on-demand .msf verify/rebuild" button. That could be nicely tucked into Thunderbird \ Preferences \ Advanced tab \ Network and Diskspace tab. It is simply ghastly the list of "Oh by the way do not do xyz while compaction is underway... else you might end up with one/multiple corrupted .msf files and/or something else."
Is this really never going to get fixed? Once again, I have had need of this. I downloaded and tried a "remove duplicates" extension that did not work, and appears to have deleted about 2/3 of my .msf files. OK, a bad extension, not your fault. But what do I do now? The only option was to manually open every folder and wait while its msf file gets rebuilt. Is it really unreasonable to expect a more graceful way of recovering from such problems?
@pdpd #54, Which version of TB? 2.x or more recent? If more recent, I know of no work-around other than to right-click EACH folder, properties, rebuild index... Or do what I did which is to move from TB to SM and use the FF/SM combination as I was using FF/TB. Let me know directly if you need some assistance with the TB->SM migration.
(In reply to comment #53) > Hee hee hee... I just happened across this in the TB KB: > http://kb.mozillazine.org/Compacting_folders#Quick_and_dirty_fix > > And I quote from the page: > "Delete all of the files ending in .msf in your profile. Thunderbird will > recreate them when it starts." This is not quite accurate - thunderbird will recreate .msf when a folder gets touched. IOW not all folders get recreated when TB starts. To force the rebuild, you can do Edit > Search > Search Messages and select an account, then search all the folders of that account.
(In reply to comment #55) > @pdpd #54, Which version of TB? 2.x or more recent? If more recent, I know of > no work-around other than to right-click EACH folder, properties, rebuild > index... I'm using the latest official version, 3.1.7. (In reply to comment #56) No, unfortunately you're wrong about "Search Messages". Before Thunderbird 3, searching recursively through the folders of an account had the effect that you describe. In Thunderbird 3, they took that away. So as far as I know, in the current version there is no workaround whatsoever.
You need to log in before you can comment on or make changes to this bug.