678148 - Tags sometimes get lost from the UI for messages tagged in the current session on a condstore server

Status: RESOLVED FIXED

(Thunderbird :: Message Reader UI, defect)

Description

[Mark Banner (:standard8)]

STR:

1) Have a folder where you've got tags pretty much on every message, and unread (I have a bugmail folder where things get tagged by component).

2) Be archiving those messages as you read them (using the Conversations Archive tool).

3) Occasionally pause for about 10-30 seconds between archiving messages.

Actual Results

On the pause, after around 10 seconds the message in the thread pane will turn black and it has "lost" its tag.

However, if I look in my zimbra mail online, the tags are still there.

It doesn't always happen, but it does frequently.

I've definitely seen it on Earlybird & Shredder, not sure about 6 beta.


I have logs of this happening so I'll send them to David.

This could be related to bug 675868.

Comment 1

[Mark Banner (:standard8)]

I am seeing this in 6 beta... 

David, can you try and get an anaylsis of what's happening here?

Comment 2

[David :Bienvenu]

Is the tag build_config? That's the only tag I see on the message with uid 58479. Because zimbra does modseq, I can't tell what the initial flags might be, but when we store the seen flag on a bunch of messages, including 58479, it returns \SEEN and build_config for that message. There's nothing in the log that would make me think we're throwing away the keywords. If I'm understanding you correctly, the message that loses its tag is a message that was not moved?

If I had to guess, I would guess that this is CONDSTORE related.

Comment 3

[David :Bienvenu]

(In reply to David :Bienvenu from comment #2)

> If I had to guess, I would guess that this is CONDSTORE related.

And nothing has changed in that area for quite a while.

Comment 4

[David :Bienvenu]

I haven't seen this happen - I tagged all my bug mail, archived/deleted a bunch of mail, and the tag has persisted on all the messages. This is using the same zimbra server...

Comment 5

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

just did a 1 hour scan of Get Satisfaction:

1. As is typical nobody has tagged anything with tb6 or Thunderbird 6 or related tags, so nothing in the 194 topics here:
https://gist.github.com/1140547

2. However from the Get Satisfaction management interface:
http://getsatisfaction.com/mozilla_messaging/admin/topics?commit=Apply&created_at_end=08%2F11%2F2011&created_at_start=07%2F15%2F2011&emotion=&participating_users=&product=87&raw_query=tags&sort=recently_active&status=&style=&user_defined_code=

 I found two topics (with unfortunately no useful troubleshooting details) which I have tagged "disappearing tags"
http://getsatisfaction.com/mozilla_messaging/tags/disappearing_tags

I will follow up and ask for more detailed troubleshooting info in GS and try it myself using Mozilla's Zimbra

Comment 6

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

(In reply to Mark Banner (:standard8) from comment #0)
> STR:
> 
> 1) Have a folder where you've got tags pretty much on every message, and
> unread (I have a bugmail folder where things get tagged by component).
> 
> 2) Be archiving those messages as you read them (using the Conversations
> Archive tool).
> 
> 3) Occasionally pause for about 10-30 seconds between archiving messages.
> 
> Actual Results
> 
> On the pause, after around 10 seconds the message in the thread pane will
> turn black and it has "lost" its tag.
> 
> However, if I look in my zimbra mail online, the tags are still there.
> 
> It doesn't always happen, but it does frequently.
> 

I just tried the above steps and I was able to reproduce the problem in TB 6 Beta 3 Windows 7 64 bit, so it's a real problem. Luckily most folks don't use tags :-)

Comment 7

[David :Bienvenu]

presumably repairing the folder fixes the tags. I'm not seeing this, so there's probably some detail I'm not getting (server side filters, client side filters, etc)

Comment 8

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

(In reply to David :Bienvenu from comment #7)
> presumably repairing the folder fixes the tags. I'm not seeing this, so
> there's probably some detail I'm not getting (server side filters, client
> side filters, etc)

yes, repairing the folder via:
 right click on the folder (control click on mac) | properties | repair folder fixed the problem, yay for workarounds!

i am using zimbra i.e. my mozilla.com email account and have no server side or client side filters for my mozilla.com account, anything else to check david?

Comment 9

[David :Bienvenu]

Can you give the complete steps? Did you tag the messages in your inbox, and then move them to the other folder? or did you tag them in the other folder after moving the messages? Or did this all happen in your inbox?

Comment 10

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

(In reply to David :Bienvenu from comment #9)
> Can you give the complete steps? Did you tag the messages in your inbox, and
> then move them to the other folder? or did you tag them in the other folder
> after moving the messages? Or did this all happen in your inbox?

nope, just followed a variant of :standard8's steps i.e.:

STR
1. in my zimbra folder called bugmail using Thunderbird, I tagged the first 10 bugs "bugzillaTestTag11August2011" (and their conversations including replies; yes i am using :protz's conversation add-on from yesterday's build and lightning and google calendar provider but no other add-on) 

2. archived one of the replies to one of the bugs using the Conversations Archive Tool in Thunderbird

Expected Results: bugzillaTestTag11August2011 tag retained for all 10 bugs and their replies

Actual Results:

1. After a second or so, the tags were "lost" in Thunderbird i.e. email turned black and if you look at it in Thunderbird there was no bugzillaTestTag11August2011 tag

2.  However, if I look in my zimbra mail online, the tags are still there.

3. Fixed by Repairing the folder in Thunderbird

Comment 11

[David :Bienvenu]

OK, I've found some steps for reproducing this. They're similar to Roland's. In particular, it's only messages that you tagged in the current session of the current folder with a condstore server that seem to be at risk of losing their tags. I had somehow got the impression that all tags on all messages were lost, which doesn't seem to be the case. It seems that messages that are tagged in the current session get added to fFlagState, but w/o their keywords set, so that when we sync flags later, we don't think those messages have any tags.

Comment 12

[Mark Banner (:standard8)]

My tags get applied by server side filters, not sure if that's current session, but I guess it could be as they'll have been downloaded in the same session.

Comment 13

[Mitra Ardron]

Re Comment #11: is that something that could be the same mechanism causing problems in Bug #675868

Comment 14

[David :Bienvenu]

Attachment: possible fix

I'll generate a try server build with this patch. I don't have reliable steps to reproduce this, but I did see this happening in the debugger. For a connection, we keep a hash table that maps IMAP UID's to keywords. We build up this information when we fetch the flags for the mailbox from the server, and when we get unsolicited change responses from the server (i.e., the server tells us new info about a message UID, including keyword info). For a condstore folder, this information tends to only contain info about new or recently changed messages.

When sending unsolicited change responses, some servers send the flag info before the UID. This means that while parsing the flag response, we have to remember the flags until we see the UID, and store the flag info in the uid hash table after we've finished parsing the line. We were doing this by checking that the current UID we knew about was -1, but I saw in the debugger cases where it was 0, which meant we didn't save the information correctly. This patch *should* fix that, though, as I say, I couldn't reliably reproduce the issue, so I couldn't verify the fix.

I'll let folks know when there's a try server build up with the patch.

Mitra, I doubt this is what you were seeing since your servers don't seem to support CONDSTORE. W/o condstore, when you shutdown and restart, we do a full sync of the flag/keyword info.

Comment 15

[David :Bienvenu]

try server builds finally finished - http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/[e-mail]-8f7fe249332e/

Comment 16

[David :Bienvenu]

Comment on attachment 552692 [details] [diff] [review]
possible fix

I don't have steps to reproduce this, but I think this should fix it. The issue has to do getting unsolicited responses with UIDs after the flags, which, prior to this patch, would get added as UID 0 to flag state, instead of getting remembered in fCustomFlags for the current line, and getting played back when finished parsing the line.

I do have a more involved fix where we use nsMsgKey_None consistently instead of mostly, and sometimes 0. But this is much smaller.

Comment 17

[Mark Banner (:standard8)]

Unfortunately I'm still seeing this in the try server build.

This is for email which the headers (and probably the bodies) had already been downloaded in a previous session.

Note that at the moment I'm also using conversations. I'll try without.

Comment 18

[Mark Banner (:standard8)]

Nope, it just happened without conversations as well.

Comment 19

[Mark Banner (:standard8)]

Urgh, I just had the case where I read a whole load at once, didn't archive, but came back a while later and all the ones I had just read showed as no tags.

I think the folder would have got new mail since I'd read it, but the mail that I had read was old mail.

Comment 20

[David :Bienvenu]

I'm pretty sure the patch does fix at least some of the problems. Also, be aware that if you run a bad build, and then run a good build, you might still see problems caused by the bad build...

The thing I was able to reproduce w/o the patch is the following:

1. Open a folder with old, un-tagged messages.
2. Tag a collapsed thread.
3. select a different folder.
4. Go back to the original folder

The thread will have lost its tags.

W/ the patch, I can't make that happen.

Comment 21

[David :Bienvenu]

an other try server build here - http://ftp.mozilla.org/pub/mozilla.org/thunderbird/try-builds/[e-mail]-5f2279508b5f/

Comment 22

[Mitra Ardron]

Hi David - what is the difference between the macosx and macosx64 builds, mac users (like me) aren't used to having to pick by word size? (I'm on a MacBook Pro)

Comment 23

[David :Bienvenu]

Mitra, your server doesn't do CONDSTORE, from what I can tell from your logs, so you would not be helped by this patch. I think you could run either osx or osx64 builds.

Comment 24

[Mark Banner (:standard8)]

Comment on attachment 552692 [details] [diff] [review]
possible fix

Ok, this definitely fixes the case you cite. I'm not sure it fixes my case yet.

However, lets get this out on all branches for some testing as I know I was seeing it in the 6 betas, and it definitely fixes something :-)

Comment 25

[Mark Banner (:standard8)]

Oh, please use hg transplant to get this onto the branches, I can tell you how if necessary - but I believe it makes the merge points easier.

Comment 26

[David :Bienvenu]

fixed on trunk http://hg.mozilla.org/comm-central/rev/323fac961903

Comment 27

[David :Bienvenu]

ugh, finally got hg transplant working on my mac - still working on Windows. Fix landed/transplanted to comm-aurora and comm-beta.

Comment 28

[Justin Wood (:Callek)]

This Landed in comm-beta as on the COMM70_20110831_RELBRANCH rather than on |default|

Please re-land to default.

Comment 29

[Justin Wood (:Callek)]

(Unsetting fixed flag, per c#28)

Comment 30

[David :Bienvenu]

(In reply to Justin Wood (:Callek) from comment #28)
> This Landed in comm-beta as on the COMM70_20110831_RELBRANCH rather than on
> |default|
> 
> Please re-land to default.

When I pull a brand new hg.mozilla.org/releases/comm-beta, I see my fix. So I'm not sure what else I can do. Perhaps the hg transplant confused things.

Comment 31

[Mark Banner (:standard8)]

This did land on default:

http://hg.mozilla.org/releases/comm-beta/rev/fa73eb44ddd4

I then landed it on the relbranch so we could take it for build 2 of 7.0b2:

http://hg.mozilla.org/releases/comm-beta/rev/900579ee49e0

Therefore this is fixed for 7.

Comment 32

[Wayne Mery (:wsmwk)]

thanks for fixing this

(In reply to Roland Tanglao :rolandtanglao from comment #6)
> Luckily most folks don't use tags :-)

Thomasd and I were just discussing that. more to follow


(In reply to Roland Tanglao :rolandtanglao from comment #5)
>  I found two topics (with unfortunately no useful troubleshooting details) which I have tagged "disappearing tags"
> http://getsatisfaction.com/mozilla_messaging/tags/disappearing_tags

I also found only those two, plus the one you already marked http://getsatisfaction.com/mozilla_messaging/tags/bug_678148 and have posted this is fixed in v7.