578918 - Indexing will not work, Gloda database journal is not created, warning in error console

Status: VERIFIED FIXED

(MailNews Core :: Database, defect)

Description

[Karel de Grootte]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-us) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.4) Gecko/20100608 Lightning/1.0b2 Thunderbird/3.1

Indexing and therefore searching through my emails does not work.
Also the drop down arrow on the left of the global search bar has disappeared.
Searching gives no results in the global search bar.

I've tried the following to enable the indexer, without any results.
- Checkbox to enable/disable GLODA Search and Indexer, and restart TB3.1 after change
- removed global-messages-db.sqlite file. After restart this file is created again.
- disabled all add-ons, like lightning 1.0b2, Provider for Google calendar, Enigmail 1.1.2., GlodaQuila 0.3.2
- reinstalled TB 3.1

I get a warning message in the error console:
2010-07-15 09:29:24	gloda.indexer	WARN	Problem during [job:folderSweep id:null items:0 offset:0 goal:null], bailing.  Problem was at undefined:1905: [Exception... "Component returned failure code: 0x80550007 [nsIMsgFolder.getStringProperty]"  nsresult: "0x80550007 (<unknown>)"  location: "JS frame :: file:///Applications/Thunderbird.app/Contents/MacOS/modules/gloda/datastore.js :: gloda_ds_mapFolderURI :: line 1905"  data: no]

Also I have an ERROR message in the Error Console, which might be, or isn't related at all.
Error: uncaught exception: [Exception... "Component returned failure code: 0x80550007 [nsIMsgFolder.getStringProperty]"  nsresult: "0x80550007 (<unknown>)"  location: "JS frame :: chrome://messenger/content/folderPane.js :: ftvItem_getProperties :: line 1963"  data: no]

I've posted two messages in the getsatisfaction forums.
http://getsatisfaction.com/mozilla_messaging/topics/thunderbird_3_search_not_working
http://getsatisfaction.com/mozilla_messaging/topics/search_lost_all_its_nice_features_like_giving_a_few_name_with_the_same_letters


Reproducible: Always

Steps to Reproduce:
1. No idea, I just don't get the indexing to work.
2.
3.
Actual Results:  
Indexing doesn't work
global-messages-db.sqlite.journal is not created
Warning message in Error console
Error message in Error Console, which might be, or isn't at all related:


Expected Results:  
Working index and therefore being able to search globally through all my emails and accounts.

Very easy simple installation of TB3.1, no themes, just a few add ons like Lightning and Enigmail
I do have 13 e-mail (POP) accounts and one feed account with about 20-25 RSS feeds.

Comment 1

[Ludovic Hirlimann [:Usul]]

Asuth what do you need to be able to analyze what's going on ?

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Probably something is wrong with the profile; gloda is dying on the exact same thing that is making the folder pane angry.  Without further investigation I'd hazard a guess that the preferences have a glitch in them, but it might also be a bad mork store.

It would be nice to handle this case better but unless we are seeing a ton of these problems, I'd suggest the reporter create a new profile or what not and just keep track of how often this comes up.

Comment 3

[Karel de Grootte]

Hi Andrew,

Ok, I understand. I tried to look for help on how to then move my existing mail and mail accounts to my new profile, but there is nothing there on this specific topic. Do you know how I can import my existing mail into my new profile. I would rather not copy the directories, as that might cause the same problem.

Karel

Comment 4

[Karel de Grootte]

So I created a new profile and added one account, still indexing and search didn't work.

So I downgraded to 3.05 again, and voila, indexing and global search works like a charm again. The .journal file is created as well.
So I guess it has something to do with the 3.1 version on my system.

I do get some error/warning messages in 3.05 but I don't know if that would interfere with an upgrade to 3.1.
I'va added them anyway, just in case.

Warning: Cannot specify value for internal property.  Error in parsing value for '-x-system-font'.  Declaration dropped.
Source File: chrome://messenger/content/glodaFacetView.xhtml
Line: 0

Error: [Exception... "Component returned failure code: 0x80004001 (NS_ERROR_NOT_IMPLEMENTED) [nsIRequest.name]"  nsresult: "0x80004001 (NS_ERROR_NOT_IMPLEMENTED)"  location: "JS frame :: file:///Applications/Thunderbird305.app/Contents/MacOS/components/nsLoginManager.js :: anonymous :: line 328"  data: no]
Source File: file:///Applications/Thunderbird305.app/Contents/MacOS/components/nsLoginManager.js
Line: 328

Can I provide any information to help solve this issue?

Kind regards,

Karel

Comment 5

[Ludovic Hirlimann [:Usul]]

You provide most of what we could easily get. I would redownload 3.1 and retry updating (when this happens open Tools -> Error console). SOmething might have gone wrong while you first updated.

Comment 6

[Karel de Grootte]

I did that once, but I will again, and now as if I was starting from scratch.

Removed TB3.1, as far as I know, I did that completely.

1. Redownload TB3.1
2. Installed TB 3.1
3. Created a new profile "Test"
4. Added my default account
5. Looked at the error console. The following errors were there:
Error: Gloda.myContact is null
Source File: file:///Volumes/Thunderbird/Thunderbird.app/Contents/MacOS/components/glautocomp.js
Line: 283

2010-07-16 14:48:48	gloda.ds.qfq	ERROR	Exception: TypeError: Gloda.myContact is null

Error: account.outgoing.hostname.toUpperCase is not a function
Source File: chrome://messenger/content/accountcreation/accountConfig.js
Line: 259
Warning: Unknown descriptor 'panose-1' in @font-face rule.  Skipped to next declaration.

Source File: mailbox:///Users/Username/Library/Thunderbird/Profiles/cwvvpfhl.test/Mail/mailaccount1/Inbox?number=128268
Line: 20

Warning: Expected end of value but found 'none'.  Error in parsing value for 'text-decoration'.  Declaration dropped.
Source File: mailbox:///Users/Username/Library/Thunderbird/Profiles/cwvvpfhl.test/Mail/mailaccount1/Inbox?number=128268
Line: 38

Warning: Cannot specify value for internal property.  Error in parsing value for '-x-system-font'.  Declaration dropped.
Source File: mailbox:///Users/Username/Library/Thunderbird/Profiles/cwvvpfhl.test/Mail/mailaccount1/Inbox?number=133433
Line: 0

Comment 7

[Karel de Grootte]

BTW, indexing still doesn't work.

Comment 8

[Ludovic Hirlimann [:Usul]]

When you start TB are there any messages in /Applications/utilities/Console.app ?

Comment 9

[Karel de Grootte]

Yes, a whole bunch related to thunderbird. A lot of INFO and DEBUG messages. I'm not sure what you would be looking for, so therefore, as it is not too long, I pasted the lines in this message.

After the startup messages, it ends with the indexing of the a couple of specific accounts and folders. The moment it "calls" folderSweep, there is a warning message, which is the same as in the Error Console of Thunderbird.
And after that, there is no message anymore. 

16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Logging Initialized
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	Beginning datastore initialization.
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	Initializing folder mappings.
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	Populating managed id counters.
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	Completed datastore initialization.
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: bool
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: number
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: string
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: date
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: fulltext
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: folder
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: conversation
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: message
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: contact
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: identity
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: parameterized-identity
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	INFO	loading all attribute defs
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:subjectMatches primary: 32
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:folder primary: 33
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:messageKey primary: 34
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:_deleted primary: 35
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:fulltextMatches primary: 36
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:bodyMatches primary: 37
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:attachmentNamesMatch primary: 38
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:authorMatches primary: 39
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:recipientsMatch primary: 40
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:conversation primary: 41
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:from primary: 42
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:to primary: 43
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:cc primary: 44
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:bcc primary: 45
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:date primary: 46
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:headerMessageID primary: 47
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:attachmentTypes primary: 48
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:involves primary: 49
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:recipients primary: 50
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:fromMe primary: 51
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:toMe primary: 52
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:mailing-list primary: 53
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:tag primary: 54
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:star primary: 55
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:read primary: 56
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:repliedTo primary: 57
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:forwarded primary: 58
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:identities primary: 59
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:name primary: 60
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:popularity primary: 61
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:frecency primary: 62
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:contact primary: 63
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:kind primary: 64
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:value primary: 65
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	built-in:freetag primary: 66
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	INFO	done loading all attribute defs
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	... loading resource:///modules/gloda/fundattr.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: mime-type
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.noun.mimetype	DEBUG	loading MIME types
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.datastore	DEBUG	QUERY FROM QUERY: SELECT * FROM ext_mimeTypes ARGS: 
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	+++ loaded resource:///modules/gloda/fundattr.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	+++ inited resource:///modules/gloda/fundattr.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	... loading resource:///modules/gloda/explattr.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: tag
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	+++ loaded resource:///modules/gloda/explattr.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	+++ inited resource:///modules/gloda/explattr.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	... loading resource:///modules/gloda/noun_tag.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	+++ loaded resource:///modules/gloda/noun_tag.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	... loading resource:///modules/gloda/noun_freetag.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.NS	INFO	Defining noun: freetag
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	+++ loaded resource:///modules/gloda/noun_freetag.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	... loading resource:///modules/gloda/noun_mimetype.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	+++ loaded resource:///modules/gloda/noun_mimetype.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	... loading resource:///modules/gloda/index_msg.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.indexer	INFO	Registering indexer: index_msg
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.index_msg	INFO	Event-Driven Indexing is now true
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	+++ loaded resource:///modules/gloda/index_msg.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	... loading resource:///modules/gloda/index_ab.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.indexer	INFO	Registering indexer: index_ab
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	+++ loaded resource:///modules/gloda/index_ab.js
16-07-10 (wk28) 15:49:40	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:40	gloda.everybody	INFO	+++ inited resource:///modules/gloda/index_ab.js
16-07-10 (wk28) 15:49:41	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:41	gloda.datastore	DEBUG	QUERY FROM QUERY: SELECT * FROM contacts ORDER BY popularity DESC LIMIT ? ARGS: 200

16-07-10 (wk28) 15:49:48	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:48	gloda.indexer	INFO	+++ Indexing Queue Processing Commencing
16-07-10 (wk28) 15:49:48	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:48	gloda.index_msg	DEBUG	Entering folder: mailbox://mailboxaccount-1/Drafts
16-07-10 (wk28) 15:49:48	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:48	gloda.indexer	INFO	--- Done indexing, disabling timer renewal.

16-07-10 (wk28) 15:49:50	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:50	gloda.indexer	INFO	+++ Indexing Queue Processing Commencing
16-07-10 (wk28) 15:49:50	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:50	gloda.datastore	DEBUG	!! mapped 136 from mailbox://mailboxaccount-2/Sent
16-07-10 (wk28) 15:49:50	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:50	gloda.index_msg	DEBUG	Entering folder: mailbox://mailboxaccount-1/Inbox/Tweakers

16-07-10 (wk28) 15:49:50	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:50	gloda.indexer	INFO	Queue-ing job for indexing: folderSweep
16-07-10 (wk28) 15:49:50	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:50	gloda.indexer	WARN	Problem during [job:folderSweep id:null items:0 offset:0 goal:null], bailing.  Problem was at undefined:1905: [Exception... "Component returned failure code: 0x80550007 [nsIMsgFolder.getStringProperty]"  nsresult: "0x80550007 (<unknown>)"  location: "JS frame :: file:///Applications/Thunderbird3-1.app/Contents/MacOS/modules/gloda/datastore.js :: gloda_ds_mapFolderURI :: line 1905"  data: no]
16-07-10 (wk28) 15:49:50	[0x0-0x99d99d].org.mozilla.thunderbird	2010-07-16 15:49:50	gloda.indexer	INFO	--- Done indexing, disabling timer renewal.

Comment 10

[Ludovic Hirlimann [:Usul]]

There is something wrong  mailbox://mailboxaccount-1/Inbox/Tweakers which makes gloda bail. can you zip that and send it to me ?

Comment 11

[Karel de Grootte]

The folder is empty. I deleted it, purge the trash and deleted an old associated filter as well.
Restarted TB.
Now It does not enter folders any more, now the last couple of lines read:

16-07-10 (wk28) 17:59:31	[0x0-0x9ca9ca].org.mozilla.thunderbird	2010-07-16 17:59:31	gloda.datastore	DEBUG	QUERY FROM QUERY: SELECT * FROM contacts ORDER BY popularity DESC LIMIT ? ARGS: 200
16-07-10 (wk28) 17:59:31	[0x0-0x9ca9ca].org.mozilla.thunderbird	2010-07-16 17:59:31	gloda.datastore	DEBUG	QUERY FROM QUERY: SELECT * FROM identities WHERE (id IN (SELECT id FROM identities WHERE (contactID IN (2)))) ARGS: 
16-07-10 (wk28) 17:59:32	[0x0-0x9ca9ca].org.mozilla.thunderbird	2010-07-16 17:59:31	gloda.datastore	DEBUG	QUERY FROM QUERY: SELECT * FROM contacts WHERE (id IN (2)) ARGS: 
16-07-10 (wk28) 17:59:40	[0x0-0x9ca9ca].org.mozilla.thunderbird	2010-07-16 17:59:40	gloda.indexer	INFO	Queue-ing job for indexing: folderSweep
16-07-10 (wk28) 17:59:40	[0x0-0x9ca9ca].org.mozilla.thunderbird	2010-07-16 17:59:40	gloda.indexer	INFO	+++ Indexing Queue Processing Commencing
16-07-10 (wk28) 17:59:41	[0x0-0x9ca9ca].org.mozilla.thunderbird	2010-07-16 17:59:41	gloda.indexer	WARN	Problem during [job:folderSweep id:null items:0 offset:0 goal:null], bailing.  Problem was at undefined:1905: [Exception... "Component returned failure code: 0x80550007 [nsIMsgFolder.getStringProperty]"  nsresult: "0x80550007 (<unknown>)"  location: "JS frame :: file:///Applications/Thunderbird3-1.app/Contents/MacOS/modules/gloda/datastore.js :: gloda_ds_mapFolderURI :: line 1905"  data: no]
16-07-10 (wk28) 17:59:41	[0x0-0x9ca9ca].org.mozilla.thunderbird	2010-07-16 17:59:41	gloda.indexer	INFO	--- Done indexing, disabling timer renewal.

Comment 12

[mark.brandon]

I have Thunderbird 3.1 running on OSX Snow Leopard. I have had the same exact problem as described by Karel de Grootte. The error console shows the same error message as well. I downgraded to Thunderbird 3.0.6 and the global search now works. I can also verify this by looking for the two files: global-messages-db.sqlite, and global-messages-db.sqlite-journal. These two files are growing in size now under v. 3.0.6, whereas in 3.1, the journal file would disappear and the other file would remain at about 45 kb. 

2010-07-23 02:44:42	gloda.indexer	warn	problem during [job:foldersweep id:null items:0 offset:0 goal:null], bailing.  problem was at undefined:1905: [exception... "component returned failure code: 0x80550007 [nsimsgfolder.getstringproperty]"  nsresult: "0x80550007 (<unknown>)"  location: "js frame :: file:///applications/thunderbird.app/contents/macos/modules/gloda/datastore.js :: gloda_ds_mapfolderuri :: line 1905"  data: no]

Comment 13

[Ludovic Hirlimann [:Usul]]

Makoto San any Idea what might be happening here ?

Comment 14

[Andrew Sutherland [:asuth] (he/him)]

This is the new "gloda, no, don't index that!" functionality...

http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/modules/datastore.js#1905
let stringPrio = aFolder.getStringProperty("indexingPriority");

It is troubling that the explosion is happening.

Comment 15

[David :Bienvenu]

This will happen if the folder doesn't really exist. We have a similar problem with the folder pane and unified folders. Perhaps getStringProperty shouldn't throw an error in this case, though it's debatable.

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

So you're saying that this will happen on virtual folders, or at least the unified virtual folders?  Gloda should not be trying to map those at all because of their virtual bits... did we ever have a bug where we didn't set them during development?  (No need to do research for that q, just wondering; it's possible there's an obvious glitch in gloda about mapping and this is the first time it's mattered.)

Comment 17

[Paul-Henri Lampe]

I just had a very similar problem to Karel's: after upgrading from TB 2 to TB 3.1 on Mac OS 10.6.4, the global search didn't return anything at all (just like your screenshot demonstrates). I described my workaround over here as a comment to Karel's post: http://getsatisfaction.com/mozilla_messaging/topics/thunderbird_3_search_not_working?utm_content=topic_link&utm_medium=email&utm_source=reply_notification

Here's a description of my setup and what I did:
. I store my messages on an external drive, that's mounted just like another USB drive. It's actually an encrypted file mounted by TrueCrypt, path "/Volumes/Crypto" (or something similar, I'm citing from memory as this is on my office computer).
. I use one main account in TB, that has a specific local directory to store its messages: in the "Server Parameters" of that account, I have "/Volumes/Crypto/Data/Thunderbird/PHL BMS" in the local directory field
. in the Local Folders section of my parameters, I have "/Volumes/Crypto/Data/Thunderbird/Local" in the local directory field
. I noticed I had a 934 Mb Inbox file in the "PHL BMS" folder, but that all the message and msf files for my archives and TB folders were in the "Local" directory
. I stopped TB, removed that big Inbox file, restarted TB and everything is fine again: the search function works perfectly as before
. I just suppose that when my email is fetched from my POP account, it sorts of goes through the Inbox file of the "PHL BMS" account and then gets moved to the Inbox of the Local folder. What makes me think of that, it's that when I moved the 934 Mb file out of the "PHL BMS" folder, I compressed it and it became very small: a sign for me of an empty TB message file that should have been compacted, as if messages had gone through it and then been moved (but that's just a supposition)
. my profiles are in the default location
. I suppose that the msf file or the inbox file was corrupted, and that removing it forced TB to reset it properly.

I hope this helps,
Paul-Henri

Comment 18

[David :Bienvenu]

(In reply to comment #16)
> So you're saying that this will happen on virtual folders, or at least the
> unified virtual folders?

No, sorry, I meant when a virtual folder had a non-existent folder in its search scope, as specified by the list of folder uri's in virtualfolders.dat. So it's not that the virtual folder doesn't exist, but one of the real folders doesn't exists.

Comment 19

[David Ascher (:davida)]

FYI -- a try-catch around the getStringProperty in datastore.js allows gloda to keep going on.  In my case it was complaining about the mailbox://nobody@Local%20Folders/mailbox%3A/Drafts folderURI.

Comment 20

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #18)
> No, sorry, I meant when a virtual folder had a non-existent folder in its
> search scope, as specified by the list of folder uri's in virtualfolders.dat.
> So it's not that the virtual folder doesn't exist, but one of the real folders
> doesn't exists.

Ah.  While I think we do want to add the try/except here just because folderSweep death is so troublesome, I probably also want to add an explicit check to the "should gloda index this folder?" logic that checks if the folder actually exists.

What is the suggested path to ask an (nsIMsgFolder) folder if it 'exists'?

Comment 21

[David :Bienvenu]

existence is complicated :-) generally we check the parent folder to see if the folder is rooted; if not, it doesn't exist. But I doubt that you could get a folder that wasn't in the hierarchy, since I think you iterate over the folders top down. I think what doesn't exist in this case is the .msf file

http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#1998

Though the actual folder didn't exist either in davida's local folders directory. Any check you write would probably be invalidated by the pluggable storage work anyway.

I'd need to poke around a bit and try to recreate what davida saw before I have a better idea of what it looks like internally. Which means firing up the mac...my guess right now is that having a file called mailbox: in your local directory confuses some uri parsing which creates a strange hierarchy.

Comment 22

[J Kay]

I have exactly the same problem as Karel de Grootte with 3.1 on Mac OS X 10.6.4.

Appearance a per his screen-shot, and no journal file either.  I deleted the global-messages-db.sqlite file, and watched as Thunderbird re-created it and its associated journal file, which then immediately disappeared!  The remaining db file is the 41Kb file.  I only use one pop3 account in Thunderbird.

have gone back to v2 till this is sorted.  I did use 3.0.5 on another Mac, with the global indexing working perfectly.

Comment 23

[Andrew Sutherland [:asuth] (he/him)]

Right, so line 1998 is the fallout of dbPath->Exists().  Unfortunately, GetStringProperty is probably the most benign way to call for that check.  So solutions sound like:

1) Have GlodaMsgIndexer.shouldIndexFolder() call GetStringProperty on the folder with a well-known string that should already be in the cache with the intent of provoking an exception if the folder is gibberish and a cache hit if it is fine.  Just adding a try/except to _mapFolder is a bad idea.

2) Have the mailnews logic that builds the nsIMsgFolders ignore things that are gibberish.

Fallout-wise, #1 is certainly safer, but if you want to fix up #2, that also works for me.  (Gloda uses ListDescendents on the root folder to get the folders to index, if that helps.)

Comment 24

[David :Bienvenu]

(In reply to comment #23)
If the folder got added to the tree at ListDescendents time, then it existed, and we would try to create an empty database for it, if none existed. If the folder got created later, we should also create an empty database for it. From davida's screen shot, the Drafts folder did not exist, nor did mailbox:.sbd, i.e., the parent directory.  So either they existed at discovery time, and then got removed, or the nsIMsgFolder object was added later, and a notification sent around, but we failed to create the directory and .msf file on disk.

Since the issue seems to survive restart, and the folder in question was the Drafts folder, I suspect the issue was post-discovery. But I can't recreate the issue.

Re GetStringProperty, it doesn't have to be a well-known property - GetStringProperty doesn't return an error on properties that aren't set, it just returns the empty string.

> Just adding a try/except to _mapFolder is a bad idea.

Can you elaborate? It seems like an error examining one folder shouldn't prevent indexing/searching of all folders. Having a bad folder object is, well, bad, but it could be ignored.

Comment 25

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #24)
> > Just adding a try/except to _mapFolder is a bad idea.
> 
> Can you elaborate? It seems like an error examining one folder shouldn't
> prevent indexing/searching of all folders. Having a bad folder object is, well,
> bad, but it could be ignored.

Before gloda ever tries to index a folder, it sees if shouldIndexFolder thinks the folder is the kind of folder we want to index.  I assert that we want to detect the problem in there, rather than subsequently in _mapFolder whose semantics are "returns a GlodaFolder".  If _mapFolder silently eats the exception and goes on to return a GlodaFolder, the implication is that we are then going to move on to try and enumerate the headers in the folder or what not.  (I presume the reason the addition of the try/except handler worked out was either because the explosion was deferred into a folder-indexing job whose death does not gum up the entire indexing sweep or we luck out and the subsequent calls don't throw exceptions.)  The other option for handling in _mapFolder is to change all the callers to assume that it can now return null, which is also not cool.

Comment 26

[Andrew Sutherland [:asuth] (he/him)]

(In reply to comment #24)
> Re GetStringProperty, it doesn't have to be a well-known property -
> GetStringProperty doesn't return an error on properties that aren't set, it
> just returns the empty string.

My concern is triggering a folder cache miss that results in opening the msf.

Comment 27

[David :Bienvenu]

(In reply to comment #25)

> Before gloda ever tries to index a folder, it sees if shouldIndexFolder thinks
> the folder is the kind of folder we want to index.  I assert that we want to
> detect the problem in there, rather than subsequently in _mapFolder whose
> semantics are "returns a GlodaFolder".
Ah, so move the try catch up a level, and that prevents bad folders from ever getting in the map. Sounds good.

"flags" is a well-known property that should always be in the folder cache.

Comment 28

[Andrew Sutherland [:asuth] (he/him)]

Attachment: v1 avoid exploding folders

I tried to write a unit test that created a JS nsIMsgFolder fake that would explode the right way and then make sure that shouldIndexFolder and indexing worked out satisfactorily.  Unfortunately, I was unable to outwit XPConnect (maybe because things never crossed through XPConnect?  but I think the instanceof hook should have been cool?) to a sufficient extent that the deadly folder would actually make it through gloda's other checks.

Comment 29

[David :Bienvenu]

Comment on attachment 460786 [details] [diff] [review]
v1 avoid exploding folders

This could be simulated by having an xpcshell test remove the local mail folder between the time listdescendents is called, and shouldIndexFolder is called, but that can't be what's happening in the wild.

I'm a bit worried that somehow the local folder is getting removed on disk between the time listdescendents and shouldIndexFolder are called, and we try to map the folder. It would be good if Davida could try running with this patch and see if the bug is still fixed for him, perhaps after moving back the "mailbox:" file he removed from his profile.

Comment 30

[Andrew Sutherland [:asuth] (he/him)]

The indexing sweep logic converts all nsIMsgFolders to GlodaFolder references in a single pass without yielding control:
http://mxr.mozilla.org/comm-1.9.2/source/mailnews/db/gloda/modules/index_msg.js#971

(It then saves off the list of GlodaFolders and slowly whittles them down.)

When it comes time to pick a new folder to index, it checks if the GlodaFolder has been deleted (which we would know about from a notification about it) and ignores such folders.

Obviously if some external process is deleting stuff from the profile directory, that is very bad but sort of unavoidable.  If an internal process is deleting stuff but not sending the appropriate notifications, it should really be sending the notifications.

I agree with your original premise that something related to URL parsing or perceived illegal file characters is creating an 'impossible folder' in the sense that we refuse to/are unable to create storage for it which allows ListDescendents to both tell us about it and accesses to it to immediately explode.

Comment 31

[Karel de Grootte]

Following your discussion is quite difficult for me, as I'm not a programmer. But although I'm sure you are aware, nevertheless I have to say that please bear in mind that the Global Indexing does work in 3.05 and is broken in 3.1. This with the exact same profile.
That makes me wonder what has changed in the (Global) indexing between the 3.0 and 3.1.

Anyway, thanks for all the work guys!

Cheers,

/Karel

Comment 32

[David Ascher (:davida)]

I recreated a failure my profile by creating a mailbox: directory in my Local Folders with an empty Drafts file, and a) reverting my patch again caused gloda to fail early on, b) applying the current patch allowed gloda to keep on going on.  So, from my POV, this patch works.

Comment 33

[Mark Banner (:standard8)]

Comment on attachment 460786 [details] [diff] [review]
v1 avoid exploding folders

I'm assuming with David's testing there's nothing else more we need/can do before we can release this. Assuming that is correct, then a=Standard8 for landing on comm-1.9.2.

Please land on trunk, default on comm-1.9.2 and COMM1927_20100730_RELBRANCH on comm-1.9.2.

Comment 34

[Scott]

(In reply to comment #0)

Same issue here, I'll try the patch now.

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.7) Gecko/20100713 Thunderbird/3.1.1

2010-07-31 00:55:58	gloda.indexer	WARN	Problem during [job:folderSweep id:null items:0 offset:0 goal:null], bailing.  Problem was at undefined:1905: [Exception... "Component returned failure code: 0x80550007 [nsIMsgFolder.getStringProperty]"  nsresult: "0x80550007 (<unknown>)"  location: "JS frame :: file:///Applications/Thunderbird.app/Contents/MacOS/modules/gloda/datastore.js :: gloda_ds_mapFolderURI :: line 1905"  data: no]

Comment 35

[Scott]

Patch worked.
Is there a simple way I can find out what folder(s) are causing the problem? I suspect one or more did not get recovered when I had a crash a couple months back and if I know which ones I can bring back an older version of each.

Comment 36

[Mark Banner (:standard8)]

Checked in to trunk: http://hg.mozilla.org/comm-central/rev/0b74f98d5116

Comment 37

[Mark Banner (:standard8)]

Landed on comm-1.9.2 and relbranch for 3.1.2:

http://hg.mozilla.org/releases/comm-1.9.2/rev/d9ed6ef6f3c5
http://hg.mozilla.org/releases/comm-1.9.2/rev/3e8b8dcafe94

Comment 38

[Karel de Grootte]

Fixed! Thanks a lot guys! Appreciate all the effort!