Closed Bug 566332 Opened 14 years ago Closed 14 years ago

Thunderbird keeps reindexing mail with no upper bound or indication of what is being indexed (With MS Exchange Server)

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Version:
1.9.2 Branch
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: hrunting, Unassigned)

Details

(Whiteboard: [closeme 2011-01-10][gs]) 

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 ( .NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4

http://getsatisfaction.com/mozilla_messaging/topics/repeated_download_sync_indexing_of_mails_again_and_again_in_thunderbird_3_using_imap_in_exchange

My comments:
All I want to know is why Thunderbird constantly displays 'Indexing ### of ### messages (9x% complete). The second number goes up, the first number is always less. It seems to get into a state where it's always doing this, and there's no upper bound on the second number. It slows things down. This is different from other indexing, where it says it's indexing ### of ### in X folder. This is not associated with a folder, and it seems to have no true high bound.

To clarify, I have four accounts configured in my Thunderbird. The "indexing" messaging doesn't give any indication which account is triggering the indexing problem, but given the volume of e-mail being indexed, it has to be one or both of the exchange accounts.

Other notes:
This indexing will go on for hours.  If I close Thunderbird and reopen, it just restarts the process.  If I walk away and let it run, it will eventually finish, but I don't know now much it does.  I cannot force it to happen.  It just frequently does.  My two IMAP Exchange accounts have over 50 folders, with some having one additional layer of subfolders.  Each folder is a subfolder off of Inbox, and at most, there is one more subfolder depth beneath that (max depth of 2 below Inbox).  My mailboxes and folders combined are probably around 150,000 messages total and receive about 5-6000 messages per day across all of them.  I've been running this setup for a couple of years, but the indexing issue obviously happened when I upgraded to Thunderbird 3.  I let it index *everything* initially.

I will reiterate, I know that it will index messages as they come in.  It does that.  I see it frequently say, "Indexing XX of XX in folder XX".  This error is different.  No new mail comes in and it starts just "Index XX of XX (XX%)".  No folder identification.  The high bound will start at something low, like 16, and then rise by a unit value (usually either 8, 12, or 16) continuously as it indexes more messages.

Reproducible: Sometimes

Steps to Reproduce:
1. Don't know how to reproduce, but it happens frequently.  If I had to guess, I'd say deleting messages causes it the most frequently.
Indexing(by Gloda, Global Search and Indexer) of all mails is required, so indexing is periodicaly executed until all mails is indexed, and indexing is executed for all new mails.
Please show evidence(message, log, ...) of (a) instead of (b).
(a) Indexing of already indexed mail is executed again ang again.
(b) Indexing of all mails is not completed yet, so indexing is periodically executed until all mails are indexed.
If problem like (a), show evidence that it occues you delete mails. 

Activity Manager window via Tools/Activity Manager may help your checking.
Gloda debbuging data can be used for cheking of above.
> https://wiki.mozilla.org/Thunderbird:Debugging_Gloda
Addon of GlodaQuilla help checking, by new "on Disk", "glodaid", "gloda dirty" column of thread pane.  
> https://addons.mozilla.org/en-US/thunderbird/addon/9873/
Note-1:
Many issues around Gloda are already resolved by Tb 3.1xpre. But checking of Gloda with Tb 3.2xpre is still troublesome due to problems in Tb 3.2xpre.
Please use Tb 3.1xpre for checking of Gloda related issues.
Note-2:
AFAIK, schema is changed and re-indexing occurs when upgrading to Tb 3.1 or downgrading to Tb 3.0. So, if you use same profile with Tb 3.1 and Tb 3.0 alternatively, re-indexing occurs upon each Tb version switch.
Note-3:
Indexing is done on mails in real mail folders. So, checking with "Smart Folders" view may produces confusions sometimes, because a "Smart Folder" is internaly a saved search folder(Virtual Folder). Please check with "All Folders" view of folder pane.
Note-4:
As it's issue with IMAP Exchange, problem of auto-sync is also susected. 
IMAP log may help your checking.
> https://wiki.mozilla.org/MailNews:Logging
> IMAP command and response : http://tools.ietf.org/html/rfc3501
It'd be helpful if someone could point out what Gloda log lines or IMAP log lines would indicate the various problems are happening.  Those logs dump out tons of information, some of it personal in nature, and I'm happy to look for specific things, but I have no idea what they are.  The Gloda log is constantly logging stuff related to new messages coming in and background tasks running.  The IMAP log is ridiculously large, but during a period when I saw the 'Index XX of XX (XX% complete)', it was inactive.
Check Gloda log with new profile, a dummy pop3 account first, to know what kind of data is logged. .eml file can be imported by Drag&Drop to thread pane. So, if you save some mails in .eml before test, you can see indexing by Gloda by Drag&Drop of .eml. Check IMAP log with the new profle with minimum access to server first, to know whay kind of data is logged - no new mail check, no auto-sync, single mail folder access only.

(In reply to comment #5)
> It'd be helpful if someone could point out what Gloda log lines or IMAP log
> lines would indicate the various problems are happening.

No log line indicates problems are happening, unless problem is "error response from server" like one. Both logging records log for normal activities. You say there is bug of Tb(you opened bug report at B.M.O), so, you need to clearly state what phenomenon is wrong and is caused by Tb's bug. Log helps to know what kind of actions are taken by Tb when the phenomenon happens.
(In reply to comment #5)
> It'd be helpful if someone could point out what Gloda log lines or IMAP log
> lines would indicate the various problems are happening.

If you are talking about bug of Exchange server(RFC violation), see Bug 92111(Exchange case), Bug 390795, Bug 403032, Bug 403039, Bug 564755, and read my comments in Bug 564755, please. RFC violation by Exchange server caused Tb's invalidation of data in offline-store file. So, it produced re-download of mails from server, then re-index by Gloda happens upon each re-download.
Bug 518702 was fixed by Tb 3.0, but Bug 534835 will be fixed by Tb 3.1.
If wrong RFC822.SIZE is cause, see comments pointed in Bug 92111 Comment #87 for workaround of it. Will it work?
Summary: Thunderbird keeps reindexing mail with no upper bound or indication of what is being indexed → Thunderbird keeps reindexing mail with no upper bound or indication of what is being indexed (With MS Exchange Server)
I don't know how to tell if it's the size issue.  There are no messages in either the gloda or imap logs that say anything about size.  During the "indexing" message I'm describing, the gloda log is inactive.  The imap log records lots of imap related commands, but because multiple threads are running and checking folders, etc. I can't tell if this is just normal Thunderbird operation or a result of it doing work for the indexing message.

I'd love to produce a simple test case, but it seems to require an active, 7-20k message mailbox to reproduce.  Anytime I try something simpler, I can't seem to get the behavior.  If anyone has any suggestions for simplifying the log hunt down, I'm all ears.
And by the way, I can reproduce this with my mailbox across Windows XP SP3, Windows 7, and MacOS X, all running the latest stable version of Thunderbird.
(In reply to comment #11)
> And by the way, I can reproduce *this* with my mailbox across ...(snip)

What is the "*this*"? What phenomenon? What is evidence that the phenomenon you saw is problem caused by flaw in code of Tb?

"Downloading(or synching) XX of YY where XX>YY" in Activity Manager for auto-sync was observed while I tested "interfere of downloading" for problem of other bug.
1. Put a small mail-1, two big mails(mail-2,mail-3, 12MB or 20MB attachments in my test), ad some mails of medium size(say mail-X) in IMAP folder, and rebuild-index.
2. Wait for start of downloading by auto-sync.
3. When downloading is started, click mail-2.
   auto-sync is interfered and paused.
   download of mail-2 takes long.
4  When subject/mail-body of mail-2 is shown, click mail-3.
   downloading of mail-2 is interfered and stopped.
   download of mail-3 takes long.
   When subject/mail-body of mail-3 is shown, click mail-2.
5. Repeat 3/4 Download of mail-2/mail-3 never completes.
6. Click a mail-X and wait for downloa of the mail-X.  
7. Wait for re-schedule of downloading by auto-sync.
8. When downloading by auto-sync is started, go to step 3.

If step 3 to step 8 is repeated, "Downloading(or synching) XX of YY where XX>YY" can be observed, because YY is not changed since initial of scheduling of the auto-sync but XX includes failed download count which was interfared by step 3.
"Moving of mails just while indexing" also can produce such external phenomenon.

Please state detail of phenomenon you thinks Tb's bug with your detailed observation of the phenomenon and other conditions when the phenomenon is observed.
WADA, read the initial comments at the beginning of this ticket.  It has nothing to do with downloading or syncing.  Read the text in the very first comment on this post and tell me what is unclear about it.  I never see this "Downloading" message you're referring to, and I never once mentioned a problem with downloading mail.

The evidence I have that it's a Tb problem is that it's happening in Tb.  Are you telling me that Thunderbird *should* just start indexing messages, giving a high bound to its indexing that repeatedly increases as it indexes, so the end-user has no idea how much longer the indexing really will end up lasting?
hrunting, much has changed in version 3.1. 
Do you still see the problem when using 3.1.7?
Whiteboard: [closeme 2011-01-10]
Version: unspecified → 1.9.2 Branch
xref these fixed bugs from the getsatisfaction article
Bug 534835 - offline folder size constantly increases using Exchange IMAP (Exchange returns wrong rfc822 size) - messages redownload
Bug 548584 - IMAP Condstore server results in wrong folder size
Whiteboard: [closeme 2011-01-10] → [closeme 2011-01-10][gs]
Wayne, it's not something I've noticed recently with Thunderbird 3.1.  I'll have to pay a little more attention to know for sure.
RESOLVED WFM per comment 16 and passage of time.
Status: UNCONFIRMED → RESOLVED
Closed: 14 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.