394098 - Compacting local folder causes tag loss

Status: RESOLVED DUPLICATE of bug 378973

(Thunderbird :: General, defect)

Description

[KK]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Build Identifier: 2007072817

User Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070728 Thunderbird/2.0.0.6 Mnenhy/0.7.5.0
Build ID: 2007072817

I rely upon assigning multiple tags to incoming e-mails in order to implement a GTD work flow. A minimum of 3 tags per message (that translate into next actions) are assigned in order to denote the priority of the action, context of the action, and to denote if the message contains an action (and not a project). After the tags are assigned, I move (archive) the message to a local folder. I have saved search folders operating on this local archive folder, which determine from the tags and flag (star) assigned to the messages, the current next actions, deferred next actions, current projects, future projects, etc.

If I set the option to automatically compact folders in Thunderbird, then accessing the saved search folder would cause Thunderbird to rebuild the mail summary file (depending if the amount of space saved is over the threshold), and this causes all the assigned tags to be lost.


Reproducible: Always

Steps to Reproduce:
1. Create 7 tags labeled "P1", "P2", "@work", "@home", "Actions", "Done" and "Projects"

2. Create a local folder named "Archive" and create a subfolder called "2007" under the "Archive" folder; this folder is referred to as the local archive folder in the following steps

3. Apply at least 3 tags to about 5-10 e-mail messages. Examples of tag assignments could be:

E-mail 1: "P1", "@home", "Actions"
E-mail 2: "P2", "@work", "Actions", "Done"
E-mail 3: "P1", "@work", "Project"
..
..

4. Move the tagged messages from the Inbox to the local archive folder 

5. Create saved search folders that operate on the local archive folder. Examples include:
- A saved search folder that shows all messages with the "Actions" tag and without the "Done" tag,
- A saved search folder that shows all messages with the "Actions" and "Done" tags,
- A saved search folder that shows all messages with the "Actions" tag and without the "Done" tag that are "P1" in priority and have the "@work" context, etc.

6. Access each of the saved search folders to see that the searches are working (it's useful if at least 2 messages are selected in each of the saved search folders)

7. Compact the local archive folder and select any of the saved search folders created above. The previously seen results should not be seen anymore. Sometimes I navigate from the Inbox to the saved search folder, and Thunderbird displays the status message that it is compacting and rebuilding the mail summary file. At the end of the process, the saved search folder does not contain any messages, i.e. the tags are removed.

Actual Results:  
I lost the tags I set up for the messages in the local archive folder; this significantly impacts my work flow as I cannot use the saved search folders to quickly see my next actions, projects, etc.

Expected Results:  
The tags should "stick" regardless of whether the local folders are compacted and regardless of whether the mail summary file is rebuilt.


Here are the list of extensions and themes I am using:

Enabled Extensions: [16]

    * Auto Copy 0.6.4.1
    * Contacts Sidebar 0.7
    * Copy Plain Text 0.3.3
    * GMailUI 0.6
    * Lightning 0.5
    * Mail Redirect 0.7.4
    * MessageID-Finder 2.0.0
    * Mnenhy 0.7.5
    * MR Tech Local Install 5.3.2.5
    * Nostalgy 0.2.13
    * Quicktext 0.9.9.9
    * Quote Colors 0.2.8
    * Stylish 0.5.2
    * Talkback 2.0.0.6
    * TB Header Tools Extension 0.6.6
    * Xpunge 0.2.3


Installed Themes: [2]

    * iV 0.1beta
    * Thunderbird (default)

Comment 1

[WADA:World Anti-bad-Duping Agency]

> Bug Summary : Compacting local folder causes tag loss

Tags are really lost when "Compact"? Is there any evidence of it?

Tag is held in X-Mozilla-Keys: header.
And, if no X-Mozilla-Keys: header, Tag data is held in ".msf", and they are written in X-Mozilla-Keys: header when "Compact Folder".
And, when old mails downloaded before Tb 2.0(Tag is newly supported by Tb 2.0), or when sent mails before Bug 383846 is fixed, or when IMAP mails copied/moved to local mail folder, X-Mozilla-Keys: header is not written until "Compact".
And, even when X-Mozilla-Keys: exists, if tag data overflow occurs for newly added Tags, overflowed tag data is not written in additional line of X-Mozilla-Keys: header until "Compact".

Thus, Tags are lost when "Rebuild Index", if tag data is not written to X-Mozilla-Keys: header yet.

See following bugs which is listed in search result for "open bugs" which has "X-Mozilla-Keys:" in bug summary.

> Bug 270068 NEW  X-Account-Key: & X-Mozilla-Keys: header is not removed from forwarded mail(message/rfc822 part) when foward a mail as attachment
> Bug 377986 DUPL X-Mozilla-Keys: header(for tag) is better to be added if not-exists, because old mails doesn't have X-Mozilla-Keys: header then tag information is lost when "delete .msf & restart mailer"
> Bug 380545 WONT X-Mozilla-Keys: header is limited to 96 bytes, then tags are not written in the header if total length exceeds 96 bytes
> Bug 383846 FIXE Tags could be lost when Sent folder index is rebuild (X-Mozilla-Keys: header is not written when copied to "Sent")
> Bug 380598 NEW  When tags are overflowed(due to X-Mozilla-Keys: size limit), compact folder duplicates overflowed tags in mail source, then tags are duplicated after rebuild index

DUP of Bug 392704 or Bug 392510?
> Bug 392704 modify rebuild index to not lose tags
> Bug 392510 "Rebuild Index" action effectively wipes tags from messages, should warn. 

Comment 2

[Wayne Mery (:wsmwk)]

Wada, are you saying currently tags should not be lost under any conditions using compact?

KK, can you still recreate this using trunk build?

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #2)
> Wada, are you saying currently tags should not be lost under any conditions using compact?

Yes.
And, if tags are not written in X-Mozilla-Keys: header yet, it'll be always lost by re-build index. This is concept & design of "rebuild-index", and problem of mismatch between design of rebuild-index and mechanism of Tag saving in X-Mozilla-Keys: is already reported to other bugs.

Although bug opener says "loss of tag by compact", problem possibly refresh problem of search folder after compact of search target folders.

Comment 4

[WADA:World Anti-bad-Duping Agency]

To Wayne:

Because bug opener says following(says "local archive" all the way),
> 4. Move the tagged messages from the Inbox to the local archive folder
it sounds that original "Inbox" is IMAP mail box.
When copy from IMAP mail box to local mail folder, X-Mozilla-Key: is not probably created by copy(created only when compact folder, I think). So tag data will be lost on "Rebuild-Index" usually(unless compact folder was executed when deleted mail exists in it before "rebuild-index".)

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

I agree with Wada, this sounds like the lack of the X-Mozilla-Keys header is the root problem and what eventually causes the tag loss here, so I am DUPing to bug 378973 which is more specific.  Arguably bug 392704 or its friend bug 392510 also apply, but the initial problem is the lack of the X-Mozilla-Keys.

KK, if the original Inbox is not an IMAP mailbox, please re-open with more details.