380598 - When tags are overflowed(due to X-Mozilla-Keys: size limit), compact folder duplicates overflowed tags in mail source, then tags are duplicated after rebuild index

Status: NEW

(MailNews Core :: Database, defect)

Description

[WADA:World Anti-bad-Duping Agency]

Attachment: screen shot of mail windows (before and after rebuild index)

This is spi-off of Bug 380545 Comment #3.

When total length of X-Mozilla-Keys: header exceeds size limit(96 bytes now), overflowed tags are held in .msf and are written in additional lines of X-Mozilla-keys: when compact folder.
But when additional header line is written, some overflowed tags were duplicated in mail source.
And these duplicates were treaded as independent tags after Re-build Index.

Test Scenario (latest-trunk 2007/5/09 build, MS Win-XP SP2)
 1. A mail has tags of tag-001 to tag-009 (within 96 bytes, single line)
 2. Add tag-00000010, tag-10, tag-00000011, tag-11 (in this order)
    => tag-10 was written because space was available
    => tag-00000010, tag-00000011, tag-11 were not written due to overflow
 3. Compact folder
    (X-Mozilla-Keys: header in message source)
    First line : tag-001 to tag-009, tag-00000010, tag-10, tag-00000011, tag-11
    Second line:  tag-00000011, tag-11 (<=duplicated)
    Third line : Null line 
    (Mail Display after compact - upper mail window in screen shot)
    tag-001 to tag-009, tag-00000010, tag-10, tag-00000011, tag-11 (correct)
 4. Rebuild Index
    (Mail Display after Re-build Index - lower mail window in screen shot)
    tag-001 to tag-009, tag-00000010, tag-10, tag-00000011, tag-11
    and duplicated tag-00000011 and tag-11 were displayed.

Comment 1

[WADA:World Anti-bad-Duping Agency]

Correction: (sorry for spam)
At step 4, duplicated tag was tag-00000011 only in screen shot. I can't know whether tag-11 is not duplicated in .msf or duplicated tag-11 is not displayed because of display area space limit.

Comment 2

[David :Bienvenu]

I'm going to try to reproduce this with a trunk build

Comment 3

[WADA:World Anti-bad-Duping Agency]

Following tags made my problem re-creation test very easy.
(no need of many tags, top in tag list, short display of tag name in UI)
> user_pref("mailnews.tags.$_1.tag", "X-1");
> user_pref("mailnews.tags.$_1_1234567890_1234567890_1234567890_1234567890_1234567890_1234567890_1234567890.tag", "X-1-L");
> user_pref("mailnews.tags.$_2.tag", "X-2");
> user_pref("mailnews.tags.$_2_1234567890_1234567890_1234567890_1234567890_1234567890_1234567890_1234567890.tag", "X-2-L");
> user_pref("mailnews.tags.$_3.tag", "X-3");
> user_pref("mailnews.tags.$_3_1234567890_1234567890_1234567890_1234567890_1234567890_1234567890_1234567890.tag", "X-3-L");
> user_pref("mailnews.tags.$_4.tag", "X-4");
> user_pref("mailnews.tags.$_4_1234567890_1234567890_1234567890_1234567890_1234567890_1234567890_1234567890.tag", "X-4-L");

I tried above tags yesterday with trunk build, and rough result was as follows. 
1. Copy a mail to a folder, compact
2. Add tags of X-1, X-1-L, X-2, X-2-L X-3, X-3-L X-4, X-4-L
   => X-1-L, X-2-L, X-3-L, X-4-L are not written in X-Mozilla-Keys:
3. Copy a mail to folder, and delete it, compact
   => All tags are written in 3 line X-Mozilla-Keys:
      (dup'ed short tag on second line may be observed at this step)
4. Remove tag of X-1-L, X-2-L, X-3-L, X-4-L
   => Line-1 X-Mozilla-Keys: X-1 X-2 X-3 X-4
      Line-2  Remained tags (X-3, X-3-L etc.) on second line
      Line-3  Line of all spaces
5. Copy a mail to folder, and delete it, compact
   => Garbage(including dup'ed short tags) still remain
6. Re-build Index will produce revived long tags.
Note: If mail at step 4 is copied to IMAP folder, Line-3(all space line) will produces protocol violation.

Comment 4

[Wayne Woods]

Sorry if I'm misunderstanding, but is this bug to do with the following? I have some very large mailbox folders. Occasionally when I switch to one, the messages won't appear immediately. Instead Tbird takes several minutes - presumably it's decided to rebuild the folder. After the threadpane is finally drawn, a considerable number of the messages are now marked as unread and appear as duplicate pairs.

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to Wayne Woods from comment #4)
> Sorry if I'm misunderstanding, but is this bug to do with the following? I
> have some very large mailbox folders. Occasionally when I switch to one, the
> messages won't appear immediately. Instead Tbird takes several minutes -
> presumably it's decided to rebuild the folder. After the threadpane is
> finally drawn, a considerable number of the messages are now marked as
> unread and appear as duplicate pairs.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #5)
> Flags: needinfo?([e-mail])

What kind of Info do you want from me for this bug?
I believae that "[Steps to reproduce] what I presented in comment #0 of this bug" is one of simplet/easiest procedures on the Earth for problem of this bug.
If you or your friends can't reproduce problem of this bug by the "[Steps to reproduce] what I presented in comment #0 of this bug", let me know about "what you exactly did", please. And, please describe about detail of "what hapened by your duplication test", please.
Sorry but I didn't do duplication test of this bug with newer Thunderbird releases, I don't know status of recent Thunderbird releass.

Comment 7

[Wayne Mery (:wsmwk)]

I am asking you about comment 4 (not comment 0) which I quoted, which was posed as a question to you.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #7)
> I am asking you about comment 4 (not comment 0) which I quoted, which was　posed as a question to you.
Why shoud I answer to comment 4? What kind of answer should I do, even though essentially irrelevant comment to actual problem of this bug?

Comment 9

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #8)
>  essentially irrelevant comment to actual problem of this bug?

That's a perfectly fine response. Again, it wasn't my question. Please Don't shoot he messenger :)

Comment 10

[:aceman]

Yeah, comment 4 seems not related to this bug. Messages suddenly appearing unread could be related to bug 840418.

WADA, do you still see this problem in TB38?

Comment 11

[WADA:World Anti-bad-Duping Agency]

I guess this bug was caused by problem like "error in duplication check of tag" in the past, and I hope already resolved.
But STR is pretty simple, and any one can do duplication test... read comment #6, please...
And, this pretty old bug is never relevant to relatively new bug 840418...