523939 - Open as List view does not show all messages shown in faceted search

Status: NEW

(Thunderbird :: Search, defect)

Description

[Jens Müller (:tessarakt)]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.9.1.3) Gecko/20090824 Firefox/3.1b4pre
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; de; rv:1.9.1.4pre) Gecko/20090923 Thunderbird/3.0b4

I entered a search term which resulted in 9 hits in the faceted search view. I clicked on "view all as list" in order to work with them. The resulting list view had only two entries ...

Reproducible: Always

Comment 1

[Jens Müller (:tessarakt)]

OK, works after deleting and rebuilding global-messages.sqlite ...

Comment 2

[Ludovic Hirlimann [:Usul]]

Did you keep a copy of the initial global-messages.sqlite ?

Comment 3

[Jens Müller (:tessarakt)]

Sorry, no :-( It was 400M, so I wanted to get rid of it before I forget ...

It's probably one of these bugs that will become obsolete when the indexer itself works fine, I hope ...

Comment 4

[Ludovic Hirlimann [:Usul]]

(In reply to comment #3)
> Sorry, no :-( It was 400M, so I wanted to get rid of it before I forget ...
> 
> It's probably one of these bugs that will become obsolete when the indexer
> itself works fine, I hope ...

I think so too. And this should land in the not so distant future ! Next time can you keep a copy so we can if needed analyse it :-)

Comment 5

[Jens Müller (:tessarakt)]

(In reply to comment #4)
> I think so too. And this should land in the not so distant future ! Next time
> can you keep a copy so we can if needed analyse it :-)

Well, I certainly won't give you access to my private mails, so it would probably be me analysing it with you guidance - quite time-consuming ...

Comment 6

[Jens Müller (:tessarakt)]

Again have it with one message in RC1.

Comment 7

[Ludovic Hirlimann [:Usul]]

can you share the message publicly ?

Comment 8

[Jens Müller (:tessarakt)]

(In reply to comment #7)
> can you share the message publicly ?

No, sorry.

But I can mail you when I'm back home on Sunday.

Comment 9

[Jens Müller (:tessarakt)]

And I have a gloda dump:

Gloda dump:

2009-11-14 23:08:32     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
* FROM contacts WHERE (id IN (SELECT id FROM contacts WHERE name LIKE ?
ESCAPE '/')) ARGS: %boor%
2009-11-14 23:08:32     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
* FROM identities WHERE (id IN (SELECT id FROM identities WHERE (kind IN
('email')) AND value LIKE ? ESCAPE '/')) ARGS: %boor%
2009-11-14 23:08:32     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
* FROM identities WHERE (id IN (SELECT id FROM identities WHERE
(contactID IN (477)))) ARGS:
2009-11-14 23:08:32     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
* FROM contacts WHERE (id IN (477,3873,3874,3875)) ARGS:
2009-11-14 23:08:32     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
* FROM contacts WHERE (id IN (477)) ARGS:
2009-11-14 23:08:32     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
* FROM identities WHERE (id IN (SELECT id FROM identities WHERE
(contactID IN (477,3873,3874,3875)))) ARGS:
2009-11-14 23:08:33     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
* FROM contacts WHERE (id IN (SELECT id FROM contacts WHERE name LIKE ?
ESCAPE '/')) ARGS: %boorberg%
2009-11-14 23:08:33     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
* FROM identities WHERE (id IN (SELECT id FROM identities WHERE (kind IN
('email')) AND value LIKE ? ESCAPE '/')) ARGS: %boorberg%
2009-11-14 23:08:33     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
*, ((((((length(osets) + 6) / 10) * 1) + notability) * 604800000000) +
date) AS dascore FROM (SELECT messages.*, messagesText.*,
offsets(messagesText) AS osets FROM messages, messagesText WHERE
messagesText MATCH ? AND messages.id == messagesText.docid AND
+messages.deleted = 0 AND +messages.folderID IS NOT NULL AND
+messages.messageKey IS NOT NULL) ORDER BY dascore DESC LIMIT ? ARGS:
"boorberg",400
2009-11-14 23:08:34     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
* FROM conversations WHERE (id IN (1889,7635)) ARGS:
2009-11-14 23:09:18     gloda.datastore DEBUG   QUERY FROM QUERY: SELECT
* FROM messages INNER JOIN messagesText ON messages.id =
messagesText.rowid WHERE (id IN (SELECT id FROM messages WHERE
(conversationID IN (7635))) AND +deleted = 0 AND +folderID IS NOT NULL
AND +messageKey IS NOT NULL) ORDER BY date ASC ARGS:

Comment 10

[Ludovic Hirlimann [:Usul]]

(In reply to comment #8)
> (In reply to comment #7)
> > can you share the message publicly ?
> 
> No, sorry.
> 
> But I can mail you when I'm back home on Sunday.

don't forget to mention the bug number in your email ..

Comment 11

[Andrew Sutherland [:asuth] (he/him)]

Jens, does this still happen for you?  I would hope the gloda correctness patch and follow-ons resolved this.

Comment 12

[Wayne Mery (:wsmwk)]

I'm hoping jens still sees the problem :) and it's the same issue as mine (an old problem actually) ... unless it's a different bug

searched on a bug# 521849 which yields 2 hits in an imap folder (dated thursday) and 2 in a local folder (dated Tuesday). But open as list or click any messages shows only the 2 imap messages.

per glodazilla, both messages in local folder are indexed, not dirty, and on disk.

further, consulting the imap folder, there are *3* messages not 2. one has a time of 4 hours ago still has dirty bit =1.  And now that I have dabbled for several minutes in that folder, the message is now indexed.  But interestingly, the imap message is *not* on disk.

In the case of the imap folder, the messages were dragged there from another imap folder. IN the case of the local folder, I moved a hundred+ with context menu move>local folders>folder>subfolder. In the case of both folders, there are no fancy messages. It's all bugmail.

Comment 13

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #12)
> per glodazilla, both messages in local folder are indexed, not dirty, and on
> disk.

There is no glodazilla, yet... but maybe that's a good idea...

The local messages can be lying to you.  The most likely explanation for the situation you're seeing is that those messages got moved and gloda did not get a notification.  Because the mailnews code is clever, it propagates all the attributes on the message, so the message would look indexed even though gloda is badly confused.

It should be reasonably short work to write an extension that walks your folders and verifies that:
1) gloda has indexed everything it should index
2) the attributes are correct, meaning that the gloda message they describe knows the correct folder and offset for the message.

You would likely need to take Thunderbird offline when you run the tool to avoid the introduction of false positives.

Comment 14

[Wayne Mery (:wsmwk)]

(In reply to comment #13)
> (In reply to comment #12)
> > per glodazilla, both messages in local folder are indexed, not dirty, and on
> > disk.
> 
> There is no glodazilla, yet... but maybe that's a good idea...

hehe. gloda on the brain. glodaquilla

> The local messages can be lying to you.  The most likely explanation for the
> situation you're seeing is that those messages got moved and gloda did not get
> a notification.  Because the mailnews code is clever, it propagates all the
> attributes on the message, so the message would look indexed even though gloda
> is badly confused.

too clever for it's own good. :)  
Is there a bug#? [1]
Does that also explain the one imap message not being indexed after 4 hours?

[1] imperfect query https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=nowordssubstr;field0-0-0=longdesc;bug_severity=critical;bug_severity=major;bug_severity=normal;type0-0-1=substring;field0-0-1=longdesc;resolution=FIXED;resolution=DUPLICATE;resolution=---;value0-0-2=show;chfieldto=2010-01-22;query_format=advanced;chfieldfrom=90d;value1-0-0=keyboard%20draft%20tags%20Killed%20manager;field0-0-2=short_desc;value0-0-1=open%20list;type0-0-0=substring;value0-0-0=open%20as%20list;component=Backend;component=Search;field1-0-0=short_desc;product=MailNews%20Core;product=Thunderbird;product=Toolkit;type0-0-2=substring

> It should be reasonably short work to write an extension that walks your
> folders and verifies that:
> 1) gloda has indexed everything it should index
> 2) the attributes are correct, meaning that the gloda message they describe
> knows the correct folder and offset for the message.
> 
> You would likely need to take Thunderbird offline when you run the tool to
> avoid the introduction of false positives.

I'm game. But more interested of course in the prevention :)

Note http://gsfn.us/t/p1gb reports the same issue. but we don't know yet  whether it or my comments here are the same as Jens'

Comment 15

[Jens Müller (:tessarakt)]

Sorry, I have not been able to reproduce this with the search string I used in comment 9 ...

Comment 16

[Kent James (:rkent)]

asuth's point in comment 14 made me look again at the code to propogate message attributes. We are not propogating gloda-id and gloda-dirty when messages are copied, but we *are* preserving them when we move them to new messages. This is set in the preferences mailnews.database.summary.dontPreserveOnCopy and mailnews.database.summary.dontPreserveOnMove.

Perhaps it is an error to leave gloda-dirty intact when the message is moved. It is easy enough to test since this is set in a preference.

Comment 17

[Wayne Mery (:wsmwk)]

I've added gloda-dirty to .dontPreserveOnMove to see what happens

Comment 18

[Thomas D. (:thomas8)]

We're not clear about the cause, but there seems to be enough evidence that this bug actually occurs. Further evidence from bug 541350.

Comment 19

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #16)
> Perhaps it is an error to leave gloda-dirty intact when the message is moved.
> It is easy enough to test since this is set in a preference.

The current behaviour is fine, the bug is most likely gloda not receiving or not being able to correctly process the move notification.  Not propagating gloda-dirty is likely to just make things worse, as if a non-zero dirty status gets propagated the folder sweep has a better chance of patching things up.

Comment 20

[Wayne Mery (:wsmwk)]

FWIW .dontPreserveOnMove gloda-dirty seems to be working fine for me.
but in my case dirty of the moved messages was zero.

Comment 21

[Wayne Mery (:wsmwk)]

asuth, rkent, Is this dependent on some other bug?

Also, I've seen this in other bug reports, and would be surprised if it's not mentioned on gsfn ... bug I haven't time to research and reference them in this comment.

Comment 22

[Andrew Sutherland [:asuth] (he/him)]

Bug 534449 or bug 562115 might be related.  protz is going to be looking at those in the near future, I think, so they are certainly the most likely to have a positive impact on the behaviour described by this bug.

Comment 23

[Wayne Mery (:wsmwk)]

Seen comment 12 again, as mentioned on IRC with asuth (sorry for the ping bug I couldn't find this bug # at the time, which I now rectify by munging the summary)

I'm not sure mine is the same as jens' issue.

searching my messages for "524315", facet results lists all the messages I would expect. but open in list yields
3 in moz-archive from 2009 - which is correct (local folder)
4 in moz-later from 2010   - which is correct (imap folder)
3 in moz-archive are missing from the list, 1 is 2/8/2011 and 2 are 2/10/2011, all have a subject which is changed from the prior 7 (the bug summary was changed, but still contains 524315)

all messages were moved from "moz" (which gets messages only via filter) to moz-archive and moz-later. all messages have gloda-id and no dirty bit.

** the "missing" messages are listed in the facet results (I know because they are dated 2011 date and have the changed subject), but clicking on them yields only the 7 message in the message list, not 10

Comment 24

[Wayne Mery (:wsmwk)]

from the IRC conversation:

asuth: is this on a profile that does regression finding with older versions of thunderbird?
wsmwk: (paraphrased) I've been using trunk and I would not have used a 3.0 (probably not even a 3.1 build) on the day I moved those 3 messages. 

asuth: 
this sounds like either the nebulous known issue where this exact set of stuff happens, or a new regression related to the msgKeyChanged stuff
(new IMAP stuff on trunk relating to IMAP offline operations has changed how gloda handles various events, and certainly could result in a regression.)
(and also the events that are generated, so it could also be a non-gloda-fault thing)
I would suggest/request that you see if you can reproduce it intentionally. a move could very likely be involved.
the fact that it doesn't show up in the message list means gloda knows about the message but its mapping is out of date.

Comment 25

[Wayne Mery (:wsmwk)]

needs QA. 
especially if bug 534449 isn't enough. (note "sent folder" can be any folder name including inbox)

Comment 26

[Mark Banner (:standard8)]

Not blocking on this. We need more QA and possible STR to understand what's going on here if it is still happening.

Comment 27

[Ludovic Hirlimann [:Usul]]

(In reply to comment #26)
> Not blocking on this. We need more QA and possible STR to understand what's
> going on here if it is still happening.

I'm not seeing anything like that on the few test I've just conducted. Jens do you still have the issue ?

Comment 28

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #27)
> I'm not seeing anything like that on the few test I've just conducted. Jens
> do you still have the issue ?

I rather doubt we've totally fixed the underlying issues to this problem.  However, since we've turned on auto-compacting/auto-ask-about-compacting, gloda is much more likely to fix things up in its compaction pass...

Comment 29

[Wayne Mery (:wsmwk)]

I still see this, and I've had auto compact on for several months.
But I also don't have STR.

Comment 30

[Jens Müller (:tessarakt)]

(In reply to comment #27)
> Jens
> do you still have the issue ?

No. (at least not with the search term I used last time to reproduce).

Comment 32

[jnnk]

I am facing this issue in betterbird 140.2.1esr-bb10. If I search for an email address, faceted view shows 4 emails, only 2 of them show up on list view.