366503 - Custom tags only work properly for certain e-mails

Status: RESOLVED FIXED

(MailNews Core :: Backend, defect)

Description

[Andy Boze]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070108 SeaMonkey/1.1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070108 SeaMonkey/1.1

I have added a custom tag called Test with a brown color. When I try to tag a message with Test, some will get the tag AND the color brown, others will only get the tag but not be colored brown.

Reproducible: Always

Steps to Reproduce:
1.Create a custom tag called Test with a brown color
2.Right click on a message and tag it as Test
3.
Actual Results:  
The message will be tagged with the text Test (add the Tags column in the mailbox summary to see this), but the message may or may not be colored brown.

Expected Results:  
The color of the message should match the text of the tag (at least when a message has only one tag attached to it).

At first I thought this behavior was random, then I realized that messages from particular individuals would always behave correctly when tagged. If the sending e-mail client is SeaMonkey (doesn't seem to matter which version), the tag behavior is always correct. If the sending client is anything else, the behavior is always incorrect, even if the e-mail is sent from Thunderbird or Netscape 7. The only exception I've noticed is that if a message sent from SeaMonkey is passed through a list server, LISTSERV at least, then the behavior of a tagged message is incorrect.

This tagging behavior does not seem to affect the 5 default tags.

I noticed this on email in my IMAP inbox, but I found it applies to Local Folders as well.

Also of note is that if I do a search based on a custom tag, and save the search as a Search Folder (I've also selected the Search Online option, but that may not matter), the contents of the search folder will only be those messages that are colored correctly. That is, if the message is tagged Test but not colored brown, the message will not appear in the search folder.

I noticed that there are other tag-related bugs noted in bug 350446, but my observations didn't seem to be quite the same.

Comment 1

[Karsten Düsterloh]

This is WFM for me with Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.2pre) Gecko/20070108 SeaMonkey/1.1. Mind that tagging IMAP messages may be unimplemented or prohibited by your IMAP server.

Comment 2

[Andy Boze]

Attachment: Showing custom tags that are colored incorrectly

Comment 3

[Andy Boze]

Attachment: Showing built-in tags in correct color

Comment 4

[Andy Boze]

I've attached two images that may help clarify this case. In the first image, notice that there are several messages tagged as "Test Tag". Two of them, on lines 4 and 9, are in the correct color, as I have configured for that custom tag. But on lines 1, 3, 5, 6 and 7, even though the messages are tagged correctly, the color is wrong. This wouldn't be a big problem, but those messages that are colored incorrectly also do not show up in a saved search folder based upon that tag. No matter how many times I deselect and reselect the custom tag for a message, it will either always appear in the correct color or never appear in the correct color.

In the second image, notice that I have changed the tag on the message on line 7 from "Test Tag" to "Important". Using this built-in tag, it will always show up in the correct color.

I would not expect this behavior from an IMAP server that does not implement or that prohibits tagging.

Comment 5

[Karsten Düsterloh]

Strange. Can you post your tag definitions?
You'll find them in your profile in the text(!) file prefs.js (<http://www.mozilla.org/start/1.5/faq/profile.html#location> may be helpful); you may open it by dragging into a browser window.
Only lines starting with 
  user_pref("mailnews.tags.
are of interest.

Comment 6

[Andy Boze]

Here's the relevant section.

user_pref("mailnews.tags.$label1.color", "#FF0000");
user_pref("mailnews.tags.$label1.tag", "Important");
user_pref("mailnews.tags.$label2.color", "#FF9900");
user_pref("mailnews.tags.$label2.tag", "Work");
user_pref("mailnews.tags.$label3.color", "#009900");
user_pref("mailnews.tags.$label3.tag", "Personal");
user_pref("mailnews.tags.$label4.color", "#3333FF");
user_pref("mailnews.tags.$label4.tag", "To Do");
user_pref("mailnews.tags.$label5.color", "#993399");
user_pref("mailnews.tags.$label5.tag", "Later");
user_pref("mailnews.tags.untitled_tag.color", "#33CCFF");
user_pref("mailnews.tags.untitled_tag.tag", "Test Tag");
user_pref("mailnews.tags.version", 2);

This is from a new profile I created in SM 1.1b just to make sure there wasn't some problem with an existing profile. I find the problematic behavior in existing and new profiles in both SM 1.1b and 1.5a.

Comment 7

[Andy Boze]

Out of curiosity, I edited the prefs.js file and changed 

user_pref("mailnews.tags.untitled_tag.color", "#33CCFF");
user_pref("mailnews.tags.untitled_tag.tag", "Test Tag");

to

user_pref("mailnews.tags.$label6.color", "#33CCFF");
user_pref("mailnews.tags.$label6.tag", "Test Tag");

After doing so, the tags began behaving as I expected.

Comment 8

[Karsten Düsterloh]

This is really strange; and I still can't reproduce this.

Comment 9

[Andy Boze]

I've got it figured out. I still don't understand why some messages get colored correctly and others don't. At first I thought it had to do with the sending e-mail client because I could pretty well predict which messages would get colored and which wouldn't. After a LOT of tagging, I found I really couldn't predict. But, here's what I did find.

Notice that the preference names for the built-in tags all have the $ sign. For example

user_pref("mailnews.tags.$label1.tag", "Important");

I found that whenever a custom tag is created, it's always created *without* the $, for example

user_pref("mailnews.tags.ignore.tag", "Ignore");

A tag created this way will cause tagged messages to behave unpredictably. If I edit the prefs.js file to change the preference to

user_pref("mailnews.tags.$ignore.tag", "Ignore");

the tagged messages behave exactly the way the built-in tags behave, that is, exactly as one would expect.

In fact, if I create a custom tag using the GUI and give it a name beginning with a $, such as $Ignore, the tagged messages behave correctly. Once the custom tag is created, I can remove the $ from the tag name using the GUI and the tag will continue to work properly because the preference name will stay the same and retain the $.

So, it appears that a patch is necessary to make sure that the preference name always includes the $.

Comment 10

[Karsten Düsterloh]

> Notice that the preference names for the built-in tags all have the $ sign.

That was chosen to avoid easy name clashes, not for some other special reason.

> I found that whenever a custom tag is created, it's always created *without*
> the $

Yes, the $ does not matter, only certain "bad" characters are stripped out when the internal tag name is created from the user defined name in the UI.

> A tag created this way will cause tagged messages to behave unpredictably.

What still amazes me is that you say that you see this behaviour even for messages under Local Folders. I would understand problems for IMAP in this regard, but for Windows XP (which comment #0 says you're using), this simply does not make any sense. (But I'll doublecheck with a Win XP box maybe tomorrow to make sure.)

> In fact, if I create a custom tag using the GUI and give it a name beginning
> with a $, such as $Ignore, the tagged messages behave correctly. Once the
> custom tag is created, I can remove the $ from the tag name using the GUI and
> the tag will continue to work properly because the preference name will stay
> the same and retain the $.

The names such as $ignore are only used to group data belonging to one tag, they could even just be numbers...

> So, it appears that a patch is necessary to make sure that the preference name
> always includes the $.

We'd first need to understand what's happening here and especially why you seem to be the only one seeing this, before changing anything.

Adding David, who wrote the tag naming scheme. 
David, do you see any reason why/how such a behaviour could happen?

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I cannot reproduce it with Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061121 Thunderbird/2.0b1pre ID:2007011504

Comment 12

[Andy Boze]

(In reply to comment #10)
> What still amazes me is that you say that you see this behaviour even for
> messages under Local Folders. I would understand problems for IMAP in this
> regard, but for Windows XP (which comment #0 says you're using), this simply
> does not make any sense. (But I'll doublecheck with a Win XP box maybe tomorrow
> to make sure.)

OK, I've done some additional testing with local folders. Initially I looked at a set of messages that I had copied from IMAP to a local folder and found that the behavior of these copied messages exactly replicated the behavior of tagging the same messages in the IMAP folder.

Today, using the latest SM 1.5a nightly (1.1 and 1.5a seem to behave identically for me with tagging), I tried tagging messages that are in local folders by way of having been placed there by filters. They all tag perfectly.

However, messages that are copied by being dragged from IMAP to a local folder do exactly replicate the tagging behavior of the original messages in the IMAP folder. I copied three new, read messages from IMAP to local. After copying, I tagged the IMAP messages with a custom tag. Two of the tagged IMAP messages did not color, but one did. Then I tagged the copied messages in a local folder. The messages colored in the same way as the originals in IMAP.

> We'd first need to understand what's happening here and especially why you seem
> to be the only one seeing this, before changing anything.

Is there anything I can do to help figure this out? Would a copy of the local .msf file be of any help?

Comment 13

[Andy Boze]

More experimenting. I'm tellin' ya, there's something magical about that $. :-) I created another tag, this one with a $. It didn't work. In order of importance, it was below the earlier custom tag that wasn't working. I increased its importance so that it was anywhere above the custom tag without the $, and it started working. Here's the relevant section of the prefs.js file.

user_pref("mailnews.tags.$doodle.color", "#FFCC00");
user_pref("mailnews.tags.$doodle.ordinal", "nothing_speciam");
user_pref("mailnews.tags.$doodle.tag", "Doodle");
user_pref("mailnews.tags.$label1.color", "#FF0000");
user_pref("mailnews.tags.$label1.tag", "Important");
user_pref("mailnews.tags.$label2.color", "#FF9900");
user_pref("mailnews.tags.$label2.tag", "Work");
user_pref("mailnews.tags.$label3.color", "#009900");
user_pref("mailnews.tags.$label3.tag", "Personal");
user_pref("mailnews.tags.$label4.color", "#3333FF");
user_pref("mailnews.tags.$label4.tag", "To Do");
user_pref("mailnews.tags.$label5.color", "#993399");
user_pref("mailnews.tags.$label5.tag", "Later");
user_pref("mailnews.tags.nothing_special.color", "#996633");
user_pref("mailnews.tags.nothing_special.tag", "Nothing Special");
user_pref("mailnews.tags.version", 2);

In the above prefs, the tag "Doodle" is ordered below "Nothing Special", so the pref

user_pref("mailnews.tags.$doodle.ordinal", "nothing_speciam");

lacks the $. Whenever "Doodle" is raised in importance to number 2, the pref becomes

user_pref("mailnews.tags.$doodle.ordinal", "$label1a");

with a $.

This even takes care of the issue with the copied messages in the local folders.

Hope this helps, or maybe it's just more baffling.

Comment 14

[Karsten Düsterloh]

If you do View->Message Source for both messages with and without working labels, does it contain the header X-Mozilla-Keys? And does the value of this header correlate with the tags set for this message?

Comment 15

[Andy Boze]

X-Mozilla-Keys only appears in message headers in my POP account folder. X-Mozilla-Keys never appears in message headers in my IMAP or Local Folders, regardless of whether the tag works right or not. In the POP folder, the value does correlate with the tag set for the message.

Comment 16

[Andy Boze]

BTW, in Local Folders, the tag name appears to be stored in the .msf file, not in the messages header. Here's a section of the file, where the tag is "nothing_special".

@$${6{@
< <(a=c)> // (f=iso-8859-1)
  (C7=notAPhishMessage)(C8=growKeywords)>
<(2ED=nothing_special)>[-0:^80(^88=1)(^89=4)(^8A=0)(^8B^83)(^82^84)
    (^85^85)(^81^86)(^83^87)(^86^88)(^95^89)(^87^8A)(^8C=a4)(^8E=0)
    (^B7^2ED)(^C8=1)]
@$$}6}@

Comment 17

[Andy Boze]

I checked with our mail server admin today to find out what we use. If it makes a difference, we use Sendmail with IMAP access via a proxy from the same company.

If you're interested, I may be able to get an account set up that you can use to test.

Comment 18

[Karsten Düsterloh]

Perhaps you could attach a mailbox file with just a "taggable" and a "non-taggable" message? Just to make sure it isn't something with those particular messages.

> If you're interested, I may be able to get an account set up that you can use
> to test.

That might be helpful if the above test messages "don't not work" for me either.

Comment 19

[Andy Boze]

Attachment: Mailbox file containing two messages

This mailbox file contains one message that tags properly and one messages that does not tag properly.

Comment 20

[Andy Boze]

Attachment: msf to accompany mailbox file

Comment 21

[Andy Boze]

Attachment: Same mailbox file, compacted

Comment 22

[Andy Boze]

Attachment: msf to accompany compacted mailbox file

Comment 23

[Andy Boze]

OK, I've attached a mailbox file. It contains two messages. The message with the subject "Where's Sandy?" tags properly; that is, it gets the color I've assigned to the tag. The second message with the subject "Editor's Pick: OpenSUSE 10.2 and Ubuntu 6.10" does not get the correct tag color. Notice that neither one includes X-Mozilla-Keys in the header. That's why I included the .msf file, because that's where the tag name is stored.

Now I've noticed something interesting. I added a third message to the mailbox and then I deleted it. Then I compacted the mailbox. Doing that resulted in the mailbox file adding X-Mozilla-Keys to the message header.

Notice that in the compacted mailbox, the message that doesn't display the tag in the correct color includes 

X-Mozilla-Keys: nonjunk nothing_special

in the header. Out of curiosity, I manually edited the file and removed "nonjunk" so that it read

X-Mozilla-Keys: nothing_special

I reopened SeaMonkey and looked in the mailbox. Interestingly, both tagged messages were now displayed in the correct color.

Comment 24

[Karsten Düsterloh]

> Notice that in the compacted mailbox, the message that doesn't display the tag
> in the correct color includes 
> 
> X-Mozilla-Keys: nonjunk nothing_special
> 
> in the header. Out of curiosity, I manually edited the file and removed
> "nonjunk" so that it read
> 
> X-Mozilla-Keys: nothing_special
> 
> I reopened SeaMonkey and looked in the mailbox. Interestingly, both tagged
> messages were now displayed in the correct color.

Nice catch!
Like tags, the junk status is stored as keywords (junk/nonjunk) and usually gets filtered out (which is quite messy, imo). Obviously something goes wrong here when trying to get a color...
Taking.

Comment 25

[Andy Boze]

Karsten, more info for you, although it may already be obvious to you.

I believe you're familiar with the extension Mnenhy. ;-) I hadn't reinstalled it with the profile I've been using recently, but I just did today to check out one of its features. (Thanks for a nice extension, BTW.) After doing so, looking at the messages in my IMAP Inbox, the messages that consistently get the correct tag color are those that also have a ? in the Junk status column. If I mark a message with a ? Junk status as "Not Junk", the next time I return to the Inbox, that message no longer has the correct Tag color. Same thing happens if I mark a ? messages as Junk; it loses the correct tag color.

Interesting thing is that the junk status also affects another tagging problem. If I tag a messages in my IMAP Inbox with three tags, say Important, Work, and Personal, which all should get the color assigned to Important, the ones with a ? Junk status retain the color of the Important tag. The ones without the ? Junk status turn green and change the label in the Tag column to Personal. The also lose the check mark in the tag selection menu. If I uncheck the Personal tag for one of these messages, the color becomes orange, and the Work tag becomes checked and displayed. If I uncheck Work, then the message becomes red and the Important tag becomes checked and displayed.

So, it seems as though the junk or nonjunk keyword is clobbering the tags.

Comment 26

[Tristan Harward]

Just some extra support for this -- I see the same problem in TB 2.0b2 20070116.

Posted here before being directed: https://bugzilla.mozilla.org/show_bug.cgi?id=355547

Looks like you're narrowing in on a solution; let me know if you need testing help in either TB or SM.

Comment 27

[Reid D Rivenburgh]

Another hopefully helpful data point.  I, too, am seeing this problem.  Messages with a custom tag aren't displayed in the tag color in the summary.  Changing the tag name from "xemacs" to "$xemacs" in prefs.js as someone noted above fixed it for me as well.  Now all xemacs tags display in the xemacs color.  Using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070207 Thunderbird/2.0pre ID:2007020704 with an IMAP server.

Comment 28

[Karsten Düsterloh]

Attachment: ignore non-tag keywords

When calculating the current most important tag for colouring we need to ignore all non-tag keywords, else a message explicitly set as (non)junk would not show tag colours for custom tags like 'Questions' etc.
Furthermore, we should not create internal keys matching the predefined ones (fortunately, these are only 'junk' and 'nonjunk' atm).

Comment 29

[David :Bienvenu]

Comment on attachment 255450 [details] [diff] [review]
ignore non-tag keywords

Thx, Karsten. I buy the first part completely, but I don't really like the junk/nonjunk part. I think we should just let the user define those tags - suppose they want to color junk messages in a particular way - adding a tag makes it easy to do that.

There are an arbitrary set of possible keywords set on imap messages, possibly by the server, possibly by other clients. We can't prevent the user from creating tags that match keywords set on some messages (we can't know about all the keys set on all the folders on a user's imap account) so I think we just have to call this a feature - adding a tag that corresponds to some existing keyword makes the keyword show up in the tag system, and I don't really buy special-casing the junk/nonjunk keywords. Yes, we set those keywords, but *so* do other imap clients.

Comment 30

[Karsten Düsterloh]

Attachment: allow junk and nonjunk as tags

Okay, junk as a tag key may lead to inconsistencies between the .msf and the "real" data, but this could happen by others tinkering with our tags as well...
This patch is almost the same as attachment 255450 [details] [diff] [review], just without the special treatment for the junk keyword.
Asking for Thunderbird 2 approval to get this onto the branch.

Comment 31

[Karsten Düsterloh]

Landed on trunk.

Comment 32

[Karsten Düsterloh]

Landed on MOZILLA_1_8_BRANCH.

Comment 33

[Carsten Book [:Tomcat]]

verified fixed 1.8.1.3 Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.1.3) Gecko/20070326 Thunderbird/2.0.0.0 Mnenhy/0.7.5.0 ID:2007032620 using the steps to reproduce in this bug here and tested custom tagging also.