528608 - Activity manager shows gloda activity with ever-rising total message count

Status: RESOLVED FIXED

(Thunderbird :: Search, defect)

Description

[Jens Müller (:tessarakt)]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.9.1.5) Gecko/20091102 Firefox/3.1b4pre
Build Identifier: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.9.1.6pre) Gecko/20091111 Lightning/1.0pre Shredder/3.0pre

I have a Gloda indexing activity in the activity bar/manager.

It goes like

Indexing x of y messages
with x,y
619 640 
625 640
633 640
641 672

It's very annoying to not know when it will stop.

Besides, I have no idea what gloda indexing is doing there. Normally, it will show a folder.

Eventually, the process will finish, but I have no idea what it is doing ...

Reproducible: Sometimes

Steps to Reproduce:
Happens after starting TB, no idea by what it is triggered ...

Comment 1

[[e-mail]]

The same happens with me on Debian Squeeze with Thunderbird 3.0 (official release).  x and y may reach numbers as high as 8000 and often it's running for several hours, hogging 100% CPU.

Comment 2

[[e-mail]]

Attachment: Screenshots of the issue

Comment 3

[Wayne Mery (:wsmwk)]

it seems scary when you see it.  3.0.1

Isn't there another bug on this? Or was it resolved?

Comment 4

[Jens Müller (:tessarakt)]

No, it's definitely not resolved for me.

Comment 5

[David :Bienvenu]

I believe asuth knows what's going on here. I'm tentatively marking it needed simply because it really freaks people out, and gives the impression that gloda will never finish. But feel free to correct me if I've over-estimated the need for this fix. It didn't sound terribly hard to fix.

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

Yeah, we can just issue a COUNT() SQL query at the start of our batch to get more accurate numbers.  There will be an efficiency cost but given that deletion currently is very inefficient, when we fix that problem it will still be a net win.

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

Attachment: v1 count the number of deleted messages at the start of the job and when the count clearly grew

This is somewhat tricky to unit test because it would require us to treat the worker's actions as non-atomic and the gloda indexer listener notification makes no guarantees about notifications other than telling you when indexing is completed.  Even if we dealt with that, we can only differentiate pre-patch and post-patch behavior when the number of messages to delete is greater than 32.  As such, I'm going to cop-out and say visual inspection along the following lines should be sufficient verification...

The easiest way to verify this change is to look for the following debug that the patch adds assuming you are running with gloda dump debug active:
  There are currently ### messages awaiting deletion processing.

Comment 9

[Jens Müller (:tessarakt)]

IMO, you should use another string for deletion, like "Removing ... of ... deleted messages from index."

Comment 10

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #9)
> IMO, you should use another string for deletion, like "Removing ... of ...
> deleted messages from index."

Indeed.  Sadly that is not an option at this point because of the string freeze.

Comment 11

[David :Bienvenu]

Comment on attachment 443865 [details] [diff] [review]
v1 count the number of deleted messages at the start of the job and when the count clearly grew

blank line between these two lines?

+}
+SingletonResultValueHandler.prototype = {

Comment 12

[David :Bienvenu]

One form of this bug still happens w/ this patch. If I have a large folder that both gloda and autosync are working on at the same time, gloda keeps discovering new messages it needs to index, and eventually it starts saying things like indexing 411 of 410 messages...421 of 420, etc, where both numbers keep going up.

Comment 13

[Andrew Sutherland [:asuth] (he/him)]

Hm, if gloda's event-driven processing is chasing after autosync or if folder-based indexing is basically doing the same thing, I'm not really sure there's anything we can do about that without compromising the efficiency of our indexing.

I think the 'solution' to that one is likely bug 493848 where we just stop telling people what gloda is up to on the status bar.  I'm fine with that being a blocker :)

Comment 14

[Jens Müller (:tessarakt)]

(In reply to comment #13)
> Hm, if gloda's event-driven processing is chasing after autosync or if
> folder-based indexing is basically doing the same thing, I'm not really sure
> there's anything we can do about that without compromising the efficiency of
> our indexing.

Combine the status messages of both activities ...

"Downloading and indexing messages. x of y messages downloaded, z of x indexed."

Comment 15

[David :Bienvenu]

(In reply to comment #13)
> Hm, if gloda's event-driven processing is chasing after autosync or if
> folder-based indexing is basically doing the same thing, I'm not really sure
> there's anything we can do about that without compromising the efficiency of
> our indexing.
It's awkward. We could at least sanity check the counts when displaying the message, so that we're not indexing y of x where y > x.
> 
> I think the 'solution' to that one is likely bug 493848 where we just stop
> telling people what gloda is up to on the status bar.  I'm fine with that being
> a blocker :)

I agree that "looking for messages to index" or whatever it is and this message aren't useful for the status bar, and merely cause confusion, and I'd be all for fixing that.

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

pushed to comm-1.9.2:
http://hg.mozilla.org/releases/comm-1.9.2/rev/2f3980bdd48e

pushed to comm-central:
http://hg.mozilla.org/comm-central/rev/d9a66920d33b

I am going to follow-up on bug 493848 in the hopes of avoiding continuing the gloda status bar blamefest.

Comment 17

[Jens Müller (:tessarakt)]

What about comment 9 and comment 14? Should I file  a new bug?

Comment 18

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #17)
> What about comment 9 and comment 14? Should I file  a new bug?

Yes, please.

Comment 19

[Jens Müller (:tessarakt)]

for comment 9: bug 564999