730250 - When having large number of Tags (20+) and trying to apply one of the newly created they disapear on refresh for imap account

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[Dragan Popovic]

User Agent: Opera/9.80 (Windows NT 6.1; U; en) Presto/2.10.229 Version/11.61

Steps to reproduce:

On one computer that i use i have 20+ Tags i created that i use.
This morning i created one more Tag to use.

I was able to reproduce the problem on other computer by creating 20+ Tags. If i use one of the first 15 Tags created they work OK. If i use one of the last created they disapear. (Both computers have Thunderbird 10.0.2)


Actual results:

When i put the newly created Tag on incoming emails i noticed that it disapears after refreshing Message List in folder. Tags that i have from before are working OK and stay after refreshing.


Expected results:

When i Tag the message, Tag should stay.

Comment 1

[Dragan Popovic]

Similar bug #689726 that is resolved by changing setting in dovecot is not in question here. Since i am the server admin and Dovecot is being used, i changed the setting in question but the problem with tags still exist.

Comment 2

[:aceman]

I can't reproduce this. Can you find your prefs.js file and paste here all lines starting with 'user_pref("mailnews.tags' ?

Comment 3

[Dragan Popovic]

user_pref("mailnews.tags.$label1.color", "#FF0000");
user_pref("mailnews.tags.$label1.tag", "Important");
user_pref("mailnews.tags.$label2.color", "#FF9900");
user_pref("mailnews.tags.$label2.tag", "Work");
user_pref("mailnews.tags.$label3.color", "#009900");
user_pref("mailnews.tags.$label3.tag", "Personal");
user_pref("mailnews.tags.$label4.color", "#3333FF");
user_pref("mailnews.tags.$label4.tag", "To Do");
user_pref("mailnews.tags.$label5.color", "#993399");
user_pref("mailnews.tags.$label5.tag", "Later");
user_pref("mailnews.tags.1.color", "#FFCCCC");
user_pref("mailnews.tags.1.tag", "1");
user_pref("mailnews.tags.10.color", "#FF9966");
user_pref("mailnews.tags.10.tag", "10");
user_pref("mailnews.tags.11.tag", "11");
user_pref("mailnews.tags.12.tag", "12");
user_pref("mailnews.tags.13.tag", "13");
user_pref("mailnews.tags.14.tag", "14");
user_pref("mailnews.tags.15.tag", "15");
user_pref("mailnews.tags.16.tag", "16");
user_pref("mailnews.tags.17.tag", "17");
user_pref("mailnews.tags.18.tag", "18");
user_pref("mailnews.tags.19.tag", "19");
user_pref("mailnews.tags.2.color", "#FFCC99");
user_pref("mailnews.tags.2.tag", "2");
user_pref("mailnews.tags.20.tag", "20");
user_pref("mailnews.tags.21.tag", "21");
user_pref("mailnews.tags.22.tag", "22");
user_pref("mailnews.tags.23.tag", "23");
user_pref("mailnews.tags.24.tag", "24");
user_pref("mailnews.tags.25.tag", "25");
user_pref("mailnews.tags.26.tag", "26");
user_pref("mailnews.tags.3.color", "#FFFF99");
user_pref("mailnews.tags.3.tag", "3");
user_pref("mailnews.tags.4.color", "#99FF99");
user_pref("mailnews.tags.4.tag", "4");
user_pref("mailnews.tags.5.color", "#99FFFF");
user_pref("mailnews.tags.5.tag", "5");
user_pref("mailnews.tags.6.color", "#CCCCFF");
user_pref("mailnews.tags.6.tag", "6");
user_pref("mailnews.tags.7.color", "#FFCCFF");
user_pref("mailnews.tags.7.tag", "7");
user_pref("mailnews.tags.8.color", "#CCCCCC");
user_pref("mailnews.tags.8.tag", "8");
user_pref("mailnews.tags.9.color", "#FF6666");
user_pref("mailnews.tags.9.tag", "9");
user_pref("mailnews.tags.version", 2);

Comment 4

[:aceman]

OK, that looks fine (I got mostly the same lines).

Can you choose a message, view its source (Ctrl-U) and note contents of "X-Mozilla-Keys:" header? Then apply the tag to this message and see the source again. Has "X-Mozilla-Keys:" changed? Can you find any pattern in the behaviour?

Comment 5

[Dragan Popovic]

(In reply to :aceman from comment #2)
> I can't reproduce this. Can you find your prefs.js file and paste here all
> lines starting with 'user_pref("mailnews.tags' ?

Can you try putting bunch of tags on same email. like putting 15 tags than refreshing the view!?

Comment 6

[Dragan Popovic]

(In reply to :aceman from comment #4)
> OK, that looks fine (I got mostly the same lines).
> 
> Can you choose a message, view its source (Ctrl-U) and note contents of
> "X-Mozilla-Keys:" header? Then apply the tag to this message and see the
> source again. Has "X-Mozilla-Keys:" changed? Can you find any pattern in the
> behaviour?

i tried viewing source of 10 messages. none of them had "X-Mozilla-Keys:"

Comment 7

[:aceman]

Even after applying the tags?
What about "X-Mozilla-Status2:" and "X-Mozilla-Status:"? Do those change after applying tags?

Comment 8

[:aceman]

OK, when I put about 25 tags on a single message then I see it. Some of the tags did not appear in "X-Mozilla-Keys:". But they are still visible and show OK when going back and forth between messages/folders.
But how do you refresh the view?

Comment 9

[Dragan Popovic]

I choose to view different subfolder on same email account or to view different email account, than go back to folder containing the message i was tagging.

Comment 10

[Dragan Popovic]

(In reply to :aceman from comment #7)
> Even after applying the tags?
> What about "X-Mozilla-Status2:" and "X-Mozilla-Status:"? Do those change
> after applying tags?

No line in the source that has X-Mozzila-

Comment 11

[:aceman]

That is very strange. What type of server is that? POP or IMAP?

From memory I remember from David Bienvenu's comment that some tags are not flushed to the mbox file when there is too many of them and they hit some limit. However they are stored in the corresponding .msf file and will be written to the mbox file when the folder compacted (or Repair folder is initiated from the folder Properties). I can confirm all the 25 tags were there in the .msf file. However, after Repair folder, they were cut down to 20 in the .msf file too and the message view also shows only 20 (while it was 25 before the repair).

Comment 12

[Dragan Popovic]

Linux server. IMAP access.
As i said similar bug that was caused by Dovecot for some other people is not in question here. I changed the config in dovecot as suggested but tags keep disapearing.

Comment 13

[David :Bienvenu]

This sounds exactly like the server is dropping the tags. an IMAP protocol log would show you if the server is keeping the keywords or not - https://wiki.mozilla.org/MailNews:Logging

Aceman, the x-mozilla-status and x-mozilla-keywords stuff is just for local messages.

Comment 14

[:aceman]

Thanks for the explanation David. What about the dropping tags from 25 to 20 that I saw on local POP3? I couldn't find a separate bug for that.

Comment 15

[Dragan Popovic]

(In reply to David :Bienvenu from comment #13)
> This sounds exactly like the server is dropping the tags. an IMAP protocol
> log would show you if the server is keeping the keywords or not -
> https://wiki.mozilla.org/MailNews:Logging
> 
> Aceman, the x-mozilla-status and x-mozilla-keywords stuff is just for local
> messages.

Here is the link to log file. 
https://rapidshare.com/files/4088341950/imap.rar

Actions performed were: Open Thunderbird, set the tag on two messages, refresh the view that showed tags not being applied. Closed Thunderbird.

Comment 16

[David :Bienvenu]

I suspect this has more to do with an issue parsing tags that are substrings of existing tags, e.g., we confuse 1 and 11.

Comment 17

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #8)
> OK, when I put about 25 tags on a single message then I see it.
> Some of the tags did not appear in "X-Mozilla-Keys:".
> But they are still visible and show OK when going back and forth between messages/folders.
(In reply to :aceman from comment #14)
> What about the dropping tags from 25 to 20 that I saw on local POP3?

FYI.
This is current retriction due to length limitation of pre-defined X-Mozilla-Keys: header when local mail folder. This is current design.
"Over flowed tag" is written to additional line of X-Mozilla-Keys: header when Compact Folder is actually invoked(deleted mail exists). This is current implementation.
Because of this, if ".msf" is intensionally deleted before Compact is actually invoked after over flow, over flowed tags are lost, because they can't be revoverd from X-Mozilla-Keys: header data.

As David says, X-Mozilla-Keys: is not relevant to IMAP folder.
Tb's tag == flag(user defined keyword) in IMAP.
If IMAP server fully supports user defined keyword, all Tb's tags are saved at server.
If IMAP server never supports user defined keyword for Tb's tag, they are saved in .msf file locally. So if user intentionally deletes .msf or synchronization with server is lost, user defined keyword(Tag in Tb) is lot.
Some IMAP server lies on "support of user defined keyword", and some IMAP server has limitation of number of "user defined keyword". In such case, servr silently ignores requested "user defined keyword"(==Tb's tag).

In any case, Tb shows tags defined prefs.js only.
So, if mail of X-Mozilla-Keys: with unknown tag is imported, or if unknown tag for Tb(user defined keyword) is added by other IMAP client, Tb won't show it.

Why no IMAP log is provided yet in this bug even though requested in Feburuary?

Comment 18

[WADA:World Anti-bad-Duping Agency]

Oh sorry, IMAP log was provided by comment #15. 

Following an example of "select" (Inbox of an IMAP server).
Because "\*" in select response. This server fully supports flag(user keyword).
>  6 select "INBOX"
>  * FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $MDNSent)
>  * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $MDNSent \*)] Flags permitted.
>  * 2 EXISTS
>  * 0 RECENT
>  * OK [UNSEEN 1] First unseen.
>  * OK [UIDVALIDITY 1314394155] UIDs valid
>  * OK [UIDNEXT 178] Predicted next UID
>  6 OK [READ-WRITE] Select completed.
>
>  7 UID fetch 1:* (FLAGS)
>  * 1 FETCH (FLAGS (NonJunk) UID 176)
>  * 2 FETCH (FLAGS (NonJunk) UID 177)
>  7 OK Fetch completed.

Following is an example of "uid store flag" at your same IMAP server(different from above Inbox because different UID).
>  14 uid store 1655 +FLAGS (1)
>  * 1285 FETCH (UID 1655 FLAGS (\Seen NonJunk $label2 $label1 $label3 $label4 $label5 1 10 11 12 13 14 15 16 17))
>  14 OK Store completed.
Server normally stores user defined keyword(tag in Tb).

Which tag(keyword) of which mail(UID) is lost at UI when you did what operation, even though you added many tags and they were once shown by Tb?

(In reply to David :Bienvenu from comment #16)
> I suspect this has more to do with an issue parsing tags that are substrings
> of existing tags, e.g., we confuse 1 and 11.

I checked with tag of "1, 10, 11, 2, 20, 21", with Gmail IMAP who supports \*, with Tb 17.0.1.
In my test, tag's definition is same as bug opener's one. tag name is digits only without preceding 0, tag's label is same as tag name.
> user_pref("mailnews.tags.NN.tag", "NN");
I don't think simple "confusion on 1 and 10/11/1... by Tb".

(1) one mail only in Mbox. UID=1. No tag is added to this mail.
(2) Add tag 11, 10, 1 => all are shown at UI. all keyword are stored.
(3) Add tag 21 => 21 are shown at UI. all keywords are stored.
(4) Add tag 20 => 1 disappears from UI. 20 is shown. all keywords are stored.
    Gmail IMAP's untagged response to "uid store flags".
      * 1 FETCH (FLAGS (\Seen 21 20 10 1 NonJunk 11) UID 1) 
(5) Add tag  2 => 1 still not shown. 2 is shown. all keywords are stored.
(6) Remove tag 10, 11, 2, 20, 21
    => no tag is shown at UI, although server returns "1" correctly.
    59 uid store 1 -FLAGS (21) 
    * 1 FETCH (FLAGS (\Seen 1 NonJunk) UID 1) 
(7) Repair Folders => "1" comes back.

Array indexing related phenomenon? Should start from 0, but starts from 1?
Or confusing with UID?
Confusion in parsing of server response? Confusion in searching tag definition for tag display?

Comment 19

[WADA:World Anti-bad-Duping Agency]

Additional funny phenomenon.
After removal of all tags, I added tag 2, 20, 21, 22, 11, 10, 1 in this order.
After it, when I removed tag 11, 11 and 1 disappered from UI at once.
  24 uid store 1 -FLAGS (11) 
  * 1 FETCH (FLAGS (\Seen 21 2 20 10 1 22 NonJunk) UID 1) 
At UI(header pane), tags are shown as "21 2 20 10 22". It looks order in server response.

Comment 20

[:aceman]

(In reply to WADA from comment #17)
> (In reply to :aceman from comment #14)
> > What about the dropping tags from 25 to 20 that I saw on local POP3?
> FYI.
> This is current retriction due to length limitation of pre-defined
> X-Mozilla-Keys: header when local mail folder. This is current design.
> "Over flowed tag" is written to additional line of X-Mozilla-Keys: header
> when Compact Folder is actually invoked(deleted mail exists). This is
> current implementation.
> Because of this, if ".msf" is intensionally deleted before Compact is
> actually invoked after over flow, over flowed tags are lost, because they
> can't be revoverd from X-Mozilla-Keys: header data.
Notice in my comment 11 that I did compaction and that is when the tags got lost even from the msf file. So this is some other problem you describe.
But yes, this bug is about IMAP so I stop now with the POP3 issue.

Comment 21

[chrizilla]

very similar to bug 687784