554215 - gloda fts3 tokenizer probably violates newly enforced constraint by OFFSETS command in SQLite 3.6.23, results in gloda test_query_core.js test failure [database disk image is malformed]

Status: RESOLVED INVALID

(MailNews Core :: Database, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

sqlite3Fts3Offsets returns SQLITE_CORRUPT when it believes the index does not agree with the table that is actually storing the text.  This is probably due to some behaviour of our tokenizer that deviates from its expectations.

This results in failures like so (on the comm-central tree built against mozilla-central where 3.6.23 has landed):
http://tinderbox.mozilla.org/showlog.cgi?log=Thunderbird/1269284373.1269285673.22003.gz

2010-03-22 12:32:39	gloda.datastore	DEBUG	QUERY FROM QUERY: SELECT *, ((((((length(osets) + 1) / 8) * 1) + notabilityCol) * 604800000000) + dateCol) AS dascore FROM (SELECT ext_widget.*, offsets(ext_widgetText) AS osets FROM ext_widget, ext_widgetText WHERE ext_widgetText MATCH ? AND ext_widget.id == ext_widgetText.docid) ORDER BY dascore DESC ARGS: foo
TEST-INFO | (xpcshell/head.js) | test 2 finished
2010-03-22 12:32:39	gloda.datastore	ERROR	Async queryFromQuery error: 11: database disk image is malformed

I am not going to be able to get to this until tomorrow.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

the good/bad news is that if I change the code to use "porter" (not our tokenizer) instead of "mozporter" (our tokenizer) the problem still happens.

Comment 2

[Nomis101]

Is this Bug the reason why the Faceted Global Search isn't working anymore in current Trunk? If I try to search something the results page is blank.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

It turns out this is not a bug in our tokenizer and was a SQLite bug.  We will be landing the fixed SQLite 3.6.23.1 or what not this week.

If 'current trunk' means a build denominated 3.2.x based off of mozilla-central, then yes.  No one should be using 3.2.x nightlies if they want something reliably usable though; the builds are just made to help keep us aware of changes to mozilla-central.