Closed Bug 392704 Opened 18 years ago Closed 16 years ago

modify rebuild index to not lose tags

Categories

(Thunderbird :: Mail Window Front End, defect, P2)

Product:
Thunderbird
Component:
Mail Window Front End
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Thunderbird 3.0b1

People

(Reporter: wsmwk, Unassigned)

References

(Blocks 2 open bugs)

Details

(Keywords: dataloss)

rebuild index causes tags stored in .msf files to be lost. A big deal/heartache to people who use tags (especially in very big folders) to indicate action items or identify messages to facilitate virtual folders.
maybe avoided by fixing bug 378973?
Depends on: 378973
(In reply to comment #1) > maybe avoided by fixing bug 378973? No. Bug 378973 is for copied mail to local mail folder(same as mail in POP3 mail folder once copied to local). For Tag in IMAP folder, I think at least similar approach to X-Mozilla-Status & X-Mozilla-Status2: header is required for X-Mozilla-Keys: header.
This seriously sucks. I just lost 3 MONTHS' worth of tags, across ~2000 IMAP emails, because of this. I have a backup that's a week old, so I can install Thunderbird on another system using that backup; but then I'll need to manually re-tag all the messages on my active system by looking over at the screen on the backup system. That alone will take hours, and then it'll take at least a full day or two to go through the few hundred messages from the past week and try to remember how I tagged them so that I can re-tag them. This is a horrible, horrible bug.
I'll back Anthony up on this one - this bug seriously sucks. So serious that even tho Thunderbird is good, I'm considering moving to another mail client. I organize my whole working life using tags, and to lose those tags in an inbox with 10000 emails, organized with tags, is very painful.
... actually, just did a count 25000+ emails. Probably should do something about that.
(In reply to comment #4) >... > I organize my whole working life using tags, and to lose those tags in an inbox > with 10000 emails, organized with tags, is very painful. in what type of folder has this happened to you?
Flags: blocking-thunderbird3?
OS: Windows XP → All
Hardware: PC → All
Why are you rebuilding the index in the first place? What information should we keep when we rebuild the index? Obviously, not all of it, or else there wouldn't be any point in rebuilding the index.
> Why are you rebuilding the index in the first place? Because a problem in Thunderbird and/or the mail server caused the index to become corrupt so that it no longer accurately matches the actual mailbox data. Often, the rebuilding happens automatically -- we're not choosing to make it happen. > What information should we > keep when we rebuild the index? The question is, why in the world is the tag data being stored in the index in the first place? The index is built automatically and can be rebuilt on the fly from the mailbox data; the index does not (should not) contain any primary data; and certainly it does not (should not) hold the ONLY copy of any data. Tags are data that we (the end-users) have carefully built up over long periods of time. It should not be considered something temporary that you can throw out at any time, as the index is, because the index can be automatically rebuilt, while tags cannot.
We store tags on the imap server, but your imap server apparently doesn't support that.
> We store tags on the imap server, but your imap server apparently > doesn't support that. In that case, you must be reverting to storing them on the client, right? If so, then again, they need to be stored somewhere permanent and treated as actual user data that can't be thrown away on a whim. The index is not the right place for storing user data.
(In reply to comment #6) > (In reply to comment #4) > >... > > I organize my whole working life using tags, and to lose those tags in an inbox > > with 10000 emails, organized with tags, is very painful. > > in what type of folder has this happened to you? > ---------------- Wayne; the folder is an inbox of emails, of which most are copied from an IMAP service locally. Thunderbird locked up with an external plugin, and as a consequence when I started it I believe it rebuilt the index.
(In reply to comment #11) > (In reply to comment #6) > > (In reply to comment #4) > > >... > > > I organize my whole working life using tags, and to lose those tags in an inbox > > > with 10000 emails, organized with tags, is very painful. > > > > in what type of folder has this happened to you? > > > ---------------- > Wayne; the folder is an inbox of emails, of which most are copied from an IMAP > service locally. Thunderbird locked up with an external plugin, and as a > consequence when I started it I believe it rebuilt the index. local is ambiguous: * is folder on imap with a name something other than "Inbox"? * copied by an imap service on a machine other than PC running thunderbird?
Hi - it is a Local Folder, stored in "~/.mozilla-thunderbird/.../Mail/Local Folders", and indexed by the .msf file, which I can confirm is where the tags were being saved.
Flags: wanted-thunderbird3+
Flags: blocking-thunderbird3?
Flags: blocking-thunderbird3-
this would be great to have.
Priority: -- → P2
Target Milestone: --- → Thunderbird 3.0b2
As I understand this, it only affects IMAP servers that cannot store custom flags. Is that correct?
Status: NEW → ASSIGNED
(In reply to comment #15) > As I understand this, it only affects IMAP servers that cannot store custom > flags. Is that correct? i no longer recall whether my original report was about local, or only local folders certainly imap issue for some of the people who have since cc:
it mainly affects IMAP, but it can affect local message keywords when the x-mozilla-keywords header in the mail message has overflowed, and we haven't compacted, and thus written out the new keywords to the mail folder.
(In reply to comment #17) > it mainly affects IMAP, but it can affect local message keywords when the > x-mozilla-keywords header in the mail message has overflowed, and we haven't > compacted, and thus written out the new keywords to the mail folder. > So what would the logic be here? I have to decide when to trust the values coming from the reindex, and when to use the saved DB values. How about if I set the union of tags that come from the reindex, and those that come from the saved DB. The issue is that we don't really know why someone had to reindex. If we just restored everything from the database, then there was no need to reindex in the first place. The existing patch on bug 449768 works slightly differently. For any saved string, if there is any value coming from the reindex, it will use that instead of the saved value. So if a message had a few tags set in the header, it would not pickup the additional, overflowed tags.
rkent, for IMAP, you would always trust the values in the original message header, in the .msf file. If the servers supports user-defined keywords, we will automatically sync keywords to the .msf file, w/o rebuilding the index. I would say the same goes for local folders as well, though you could do your union thing there. But the local folder keywords are inherently less trustworthy than those in the .msf file. I think we have to assume the user is re-indexing for a limited set of reasons: 1. The message counts on the folder are wrong 2. The message threading or thread counts are horked. 3. The offline imap store is horked, or we've got incorrect offsets/sizes into the local store. In either case, rebuilding the index makes us forget that we had the message in the offline store. This isn't great; we should try to verify the offline store offsets are right, and keep them if they look OK. 4. The berkeley mailbox offsets have gotten confused. In this case, I think you could use the message-id relative uniqueness to maintain old attributes.
I have run into this problem twice over the past couple months. It happens when I run out of disk space. For some reason, when this happens, Thunderbird automatically rebuilds the index, causing me to lose all of my tags. Perhaps I should file a separate bug for this--it is certainly related to bug 339805. It seems to me to be a very critical bug--users are not even made aware that there is a possibility of data loss. I assumed tags were stored in the message header, but apparently my server (Courier IMAP 3.0.8) does not support this. It would probably also be useful to compile a list of IMAP servers that exhibit this behaviour.
Upgrading to Courier-IMAP 4.1 solved this problem for me, as it supports IMAP keywords as described in bug 392510. Since I don't believe there's a way to detect what IMAP server version is running, it seems a warning dialog should pop up the first time a user uses tags/flags/etc. on an IMAP account, informing them that they will be stored locally, and can be erased if the index ever needs to get rebuilt.
Although I do not seem to have access to a server that does not support user flags, I tested this in simulation by commenting out flag support - and confirming that by deleting the .msf file and seeing flags disappear on automatic rebuild. On manual reindex, the flags are preserved now using the code from bug 449768. So I believe that this bug is now fixed, at least for the case where reindexing is manually selected, or the folder is renamed. There are still some remaining issues that I am aware of involving message moves (bug 459680), and there may be others. So I propose that we close this bug as fixed - but if this bug was a blocker, so should bug 459680.
agreed that this should be marked fixed. The other bug won't block b1, but I'll mark it blocking b2 (this bug wouldn't block b1 at this point either)
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.