691770 - Tags are lost when an index file is rebuilt (mail copied from IMAP to local doesn't have X-Mozilla-Keys: until compacted)

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Laurent Bauvens]

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20100101 Firefox/8.0
Build ID: 20110928060149

Steps to reproduce:

I deleted a single character in an index file.


Actual results:

1 - all the tags associated with the message database had been lost when …
 A/ … I clicked on the repair button
or
 B/ … I restarted Thunderbird


Expected results:

No tag loss

Comment 1

[Laurent Bauvens]

I think index files are not the best place to store tags because these files are often subject to corruption.

Mozilla implements unlimited tags to help users to retrieve and classify easily their mails but they can't imagine all their tags could be lost after a thunderbird crash which would trouble an index file.

Why don't you integrate tags in the message header using "X-Mozilla" keys ?

Comment 2

[Ludovic Hirlimann [:Usul]]

Laurent is this a regression ?

Is this over imap or pop3 ?

Comment 3

[Laurent Bauvens]

It's not a regression. It's a problem in Thunderbird since the feature appeared long time ago.

Before the version 2.0, there are only 5 tags and one tag per mail. It was too poor to be useful for numbers of users.

As we hadn't deployed the version 2.*, our users have recently discovered tags with Thunderbird 3.1 and begun to use them.

The problem is clearly detected with local databases (ie local folders and pop accounts). I haven't tested with IMAP because in this case the tags are stored in our messaging system. But if we'd have a basic imap daemon which could not store tags I think the problem will be the same as with local message databases.

This bug is probably a sequel of this one: https://bugzilla.mozilla.org/show_bug.cgi?id=392510#c3

Comment 4

[Matthias Versen [:Matti]]

dupe of bug 392510 ?

Comment 5

[David :Bienvenu]

Are your users rebuilding the .msf files, and if so, how (through properties dialog, or deleting the .msf files by hand?)

Comment 6

[Laurent Bauvens]

Hi David

(In reply to David :Bienvenu from comment #5)
> Are your users rebuilding the .msf files,
Yes, that certainly happens every day.

> and if so, how (through properties
> dialog, or deleting the .msf files by hand?)

If you rebuild a healthy index file through properties dialog, nothing is lost.

If you manually delete an index file, you for sure lose everything but the messages. (Old school manner, but as our users are far from being geeks, they now prefer the new and friendly repair button)

If an index file is corrupted, Thunderbird will automatically detect the problem and rebuild it at restart, or you can manually start the rebuilding through the properties dialog. 

In these two latter cases, tags are lost.

Up to now, we had sometimes even excluded index files from backup tasks to speed them up. Because of the belief that index files are just generated data only useful for Thunderbird. We now learnt the lesson.

Once again, tags are real meta-data that users rely on. If any corruption of an index file leads to a data loss, so tags are just useless. But as I think tags are a very useful feature, they'd have to be stored in a safe place outside of the index files.

Another way is to make index files really secure.

Thanks

Comment 7

[David :Bienvenu]

Tags are stored in the actual mailbox, *and* in the .msf files. We allocate space for about 70 chars worth of tags, and expand that at compaction time if the user has used all the space. I'm getting the impression that your users aren't getting their folders compacted regularly, or not as regularly as they're blowing away the index files. This is a weakness of the berkeley mailbox store, and we're trying to move to a different message store that doesn't require compaction, and allows for easier growing of the tag space in the message store.

Are you really seeing corrupted index files? And if so, how are they corrupted? Are you seeing out of date timestamps because the index files are stored on a networked file system?

Comment 8

[:aceman]

David, what does TB do when the tags reach those 70 chars and the user tries to add more?
Still, when the index is removed those 70 char tags stored in the mailbox should survive.

Laurent, can you check if the tags are really stored in your mailbox files. See source of some message (Ctrl+U), then tag it and see the source again. Watch changes in X-Mozilla-Keys: header.
Got the tag stored there?
Also see if the header X-MozillaStatus2 exists and changes in any way.

Comment 9

[David :Bienvenu]

(In reply to aceman from comment #8)
> David, what does TB do when the tags reach those 70 chars and the user tries
> to add more?

It remembers the extra tags in the index, and sets a flag so that when the mailbox is compacted, space for new tags is allocated in the compacted folder, and the extra tags are written to the compacted folder.

Comment 10

[Laurent Bauvens]

I spent some times to test in depth. Here are the results:

Local message databases (ie POP & Local Folders): Thunderbird adds in the message header a key named X-Mozilla-Keys to store the tags, if they exist and, as David wrote in comment 7, tags are also stored in index files. Fine. :)

IMAP messages: Thunderbird store the tags on the server if it offers the feature. Nevertheless, TB also stores tags in local IMAP index files.

But, if you move a mail from an IMAP server to a local database, the X-Mozilla-Keys is never added to its header and only the index file is modified to add tags. 
(fyi: My main account is IMAP and most of my local emails come from it, that's why I never found a tag key in the header of my messages). In this case, a broken index file makes you to lose tags after a rebuild. Is there a reason to not add the key when a mail come from an IMAP account ? Is there a way to restore/create headers keys from an index file ?

David, I confirm compaction is a real problem for users who have big databases sometimes hosted on Network Drives. It takes too time and users often ask the admin to definitively cancel it.

Comment 11

[David :Bienvenu]

(In reply to Laurent Bauvens from comment #10)
> But, if you move a mail from an IMAP server to a local database, the
> X-Mozilla-Keys is never added to its header and only the index file is
> modified to add tags. 

That's a bug, and I should fix that.
> (fyi: My main account is IMAP and most of my local emails come from it,
> that's why I never found a tag key in the header of my messages). In this
> case, a broken index file makes you to lose tags after a rebuild. Is there a
> reason to not add the key when a mail come from an IMAP account ? Is there a
> way to restore/create headers keys from an index file ?
 adding a tag to all messages should cause us to notice that there's not room in the x-mozilla-keys header (because there isn't one), and then the next time you compact, we should add all the tags from the index file. Then you could remove that tag. I haven't tried this, but it should work.
> 
> David, I confirm compaction is a real problem for users who have big
> databases sometimes hosted on Network Drives. It takes too time and users
> often ask the admin to definitively cancel it.

Yeah, compaction is a real problem, which is why we're trying to move to maildir type message store, though that will have different issues on network drives.

Comment 12

[David :Bienvenu]

Attachment: fix for imap to local move/copy

simple fix for copying imap mesage to local folder - add x-mozilla-keys when we add x-mozilla-status(2)

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to David :Bienvenu from comment #12)
> simple fix for copying imap mesage to local folder - add x-mozilla-keys when
> we add x-mozilla-status(2)

If it would have been done with next your comment of bug 378973 comment #1 on 2007-04-26, I would have beeen very happy :-)
> But yes, we should add the x-mozilla-keys header when the imap message is added to the local folder...
David, can I close bug 378973 as dup of this bug?

Mail&News has next problem. 
> Bug 426651 X-Mozilla-Keys: header is not removed when copy/move a mail from local mail folder to IMAP mail folder
If user upload mail from local to IMAP then copy to local again, will multiple X-Mozilla-Keys:'s be generated after your fix?
If yes, it's a new variant of bug 561703, X-Mozilla-Keys: version of that bug.
Will such duplicated headers produce unwanted problem?

Comment 15

[David :Bienvenu]

(In reply to WADA from comment #13)

> David, can I close bug 378973 as dup of this bug?
thx, I've duped that.
> 
> If user upload mail from local to IMAP then copy to local again, will
> multiple X-Mozilla-Keys:'s be generated after your fix?
> If yes, it's a new variant of bug 561703, X-Mozilla-Keys: version of that
> bug.
> Will such duplicated headers produce unwanted problem?
We should fix this on upload to the imap server - it may cause issues in the case of db regeneration, similar to but maybe not quite the same as duplicate x-mozilla-status headers.

Comment 16

[[e-mail]]

Comment on attachment 565028 [details] [diff] [review]
fix for imap to local move/copy

[f+, will mark r+ when I get my tree rebuilt]

>-    mCopyState->m_fromLineSeen = PR_TRUE;
>+   mCopyState->m_fromLineSeen = PR_TRUE;
[Butterfingers?]

>+    result = X_MOZILLA_KEYWORDS;
MXR confused me by not showing all the spaces in this macro ;-)

Comment 17

[David :Bienvenu]

fixed on trunk - http://hg.mozilla.org/comm-central/rev/3fa232956b09

thx, Ludo.

Comment 18

[Laurent Bauvens]

Thanks David & others.

Have an idea of the thunderbird version which will embed this fix ?

Comment 19

[David :Bienvenu]

sorry, setting TB fix version to TB 10.

Comment 20

[David Hobbs]

I'm using TB 15.0.1 and still have this problem. Tagged messages moved from IMAP to local folder have empty "X-Mozilla-Keys". And compaction, which used to add the tags to "X-Mozilla-Keys", no longer does it. The only thing that works for me is to remove the tag in the local copy and then add it back.

Comment 21

[Kent James (:rkent)]

David, please open a new bug if you are having an issue.

Comment 22

[WADA:World Anti-bad-Duping Agency]

Checked with Tb 10.0.4esr(ja build)
> Mozilla/5.0 (Windows NT 5.1; rv:10.0.4) Gecko/20120421 Thunderbird/10.0.4

(1) At an IMAP folder, Repair Folder, to force clean mail folder data.
(2) Add tag(call Tag-1) to mail in the folder
    (Gmail IMAP, so user keyword is fully supported).
(3) At Inbox of Local Folders, delete all mails, Compact, in order to get clean mail folder. Copy some mails for future mail delete in order to execute Compact.
(4) Copy the mail in IMAP folder to Inbox of Local Folders.
(5) Mail data of copied mail in Local olders.
    Tag(Tag-1) is inherited at Inbox of Local Folders.
    (=> Tag-1 is held in .msf file as xpected)
    X-Mozilla-Keys: header is written since initial of copy from IMAP to local,
    (=> a part of this bug was fixed by the patch)
    but nothing is written in X-Mozill-Keys: header(data spaces).
    (=> main issue was not resolved by patch)
(6) Add other tags(call Tag-2 to Tag-N) to the mail.
    Newly added Tag-2 to Tag-N is wrtten in X-Mozilla-Keys: header,
    but Tag-1 is not written to X-Mozilla-Keys: header.
(7) Delete a mail at lowest offset in order to force Compact to do compaction,
    then execute Compact of Inbox of Local Folders,
    Tag-1 is not written in X-Mozilla-Keys: header.
(8) Repeat step (7) several times.
    Tag-1 is stil not written in X-Mozilla-Keys: header.
    (=> worse than before patch)
(9) Repair Folder of Inbox of Local Folders.
    Tag-1 is kept.
    i.e. tag data itself is held in .msf and is inherited by Repair Folder.  

Re-opening, because "this bug was not fixed by patch" was found by fix verification test with Tb 10esr.

Comment 23

[WADA:World Anti-bad-Duping Agency]

Additional check result with Tb 15.0.1.

Even when Tag-1(added while held in IMAP) was not written in X-Mozilla-Keys: header and only newly added tags are written in single line X-Mozilla-Keys: header, when many tags were additionaly added and overflow of X-Mozilla-Keys: header occurred, Tag-1(added while held in IMAP) and "nonjunk"(not defined as user tag. mailnews.tags.nonjunk is not defined) were added to multi-line X-Mozilla-Keys: header by Compact.

Problem in "tag & X-Mozilla-Key: header processing in Compact" was exposed by "X-Mozilla-Keys: header with space only" which was newly added by patch for this bug since initial copy from IMAP to local?
(i) If no X-Mozilla-Keys: header, Compact adds X-Mozilla-Keys: header and writes all tags in the header.
(ii) If status of "overflow of tag" is known, Compact expands X-Mozilla-Keys: header and writes all tags in the header.
(iii) When copy from IMAP to local, tag(added while held in IMAP) is held in .msf, and blank X-Mozilla-Keys: is written by patch. However, status of "overflow of tag" is not set. So, Compact does do nothing on tag & X-Mozilla-Keys: header.

Comment 24

[WADA:World Anti-bad-Duping Agency]

If X-Mozilla-Keys: header is manually deleted from mail data and Repair Folder is executed, Compact inserts X-Mozzila-Keys: header and writes all of Tag-1(added while held in IMAP), Tag-2 to Tag-N(added after copy from IMAP to local), and "nonjunk"(not defined as user defined tag) in X-Mozilla-Keys: header.
This is seen in both Tb 10 and Tb 15.0.1.
"Writing nonjunk which is held in .msf to X-Mozilla-Keys:" looks design.

Comment 25

[WADA:World Anti-bad-Duping Agency]

David :Bienvenu, your patch's fault? Or flaw in Compact logic?

Comment 26

[David :Bienvenu]

(In reply to WADA from comment #25)
> David :Bienvenu, your patch's fault? Or flaw in Compact logic?

Some other code needs to actually write the keywords from the imap message into the local message x-mozilla-keys header. All my patch did was make space for subsequent tags to get written to the actual mailbox.

Comment 27

[WADA:World Anti-bad-Duping Agency]

(In reply to David :Bienvenu from comment #26)
> All my patch did was make space for subsequent tags to get written to the actual mailbox.

I see. 
By the patch, "newly added tags after copy from IMAP to local" won't be lost by deletion of .msf, unless "overflow of tags" occur. So, major problem of this bug was resolved by your patch.
I'll open separate bug for outstanding case.
- Tags added while mail was held in IMAP folder is lost if .msf file is deleted,
  unless "overflow of tags" happens after mail copy and Compact writes all tags
  to expanded X-Mozilla-Keys: header.

Cloing again.

Comment 28

[WADA:World Anti-bad-Duping Agency]

I've opeed bug 791925 for problem of comment #20 / comment #22.