1252937 - Define SQLITE_ENABLE_FTS3_TOKENIZER for Thunderbird/SeaMonkey only for the gloda global database that uses a custom FTS3 tokenizer

Status: RESOLVED FIXED

(Toolkit :: Storage, defect)

Description

[Richard Marti (:Paenglab)]

The errors I get:

Wed Mar 02 2016 19:08:28
Error: 2016-03-02 19:08:28	gloda.datastore	ERROR	Unexpected error when trying to open the database: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIFts3Tokenizer.registerTokenizer]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource:///modules/gloda/datastore.js :: gloda_ds_init :: line 1062"  data: no]

Source file: resource://app/modules/gloda/log4moz.js
Line: 691
 ----------
2016-03-02 19:08:28	gloda.datastore	ERROR	Unexpected error when trying to open the database: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIFts3Tokenizer.registerTokenizer]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource:///modules/gloda/datastore.js :: gloda_ds_init :: line 1062"  data: no]

 ----------
Wed Mar 02 2016 19:08:28
Error: 2016-03-02 19:08:28	gloda.everybody	ERROR	!!! error loading resource:///modules/gloda/fundattr.js

Source file: resource://app/modules/gloda/log4moz.js
Line: 691
 ----------
2016-03-02 19:08:28	gloda.everybody	ERROR	!!! error loading resource:///modules/gloda/fundattr.js

 ----------
Wed Mar 02 2016 19:08:28
Error: 2016-03-02 19:08:28	gloda.explattr	ERROR	Error in init: TypeError: this.asyncConnection is null

Source file: resource://app/modules/gloda/log4moz.js
Line: 691
 ----------
2016-03-02 19:08:28	gloda.explattr	ERROR	Error in init: TypeError: this.asyncConnection is null

 ----------
Wed Mar 02 2016 19:08:28
Error: 2016-03-02 19:08:28	gloda.everybody	ERROR	!!! error initializing resource:///modules/gloda/explattr.js

Source file: resource://app/modules/gloda/log4moz.js
Line: 691
 ----------
2016-03-02 19:08:28	gloda.everybody	ERROR	!!! error initializing resource:///modules/gloda/explattr.js

 ----------
Wed Mar 02 2016 19:08:28
Error: 2016-03-02 19:08:28	gloda.everybody	ERROR	!!! error loading resource:///modules/gloda/noun_mimetype.js

Source file: resource://app/modules/gloda/log4moz.js
Line: 691
 ----------
2016-03-02 19:08:28	gloda.everybody	ERROR	!!! error loading resource:///modules/gloda/noun_mimetype.js

 ----------
Wed Mar 02 2016 19:08:28
Error: 2016-03-02 19:08:28	gloda.everybody	ERROR	(resource:///modules/gloda/datastore.js:1418) TypeError: this.asyncConnection is null

Source file: resource://app/modules/gloda/log4moz.js
Line: 691
 ----------
2016-03-02 19:08:28	gloda.everybody	ERROR	(resource:///modules/gloda/datastore.js:1418) TypeError: this.asyncConnection is null

 ----------
Wed Mar 02 2016 19:08:28
Error: 2016-03-02 19:08:28	gloda.abattrs	ERROR	Error in init: TypeError: this.asyncConnection is null

Source file: resource://app/modules/gloda/log4moz.js
Line: 691
 ----------
2016-03-02 19:08:28	gloda.abattrs	ERROR	Error in init: TypeError: this.asyncConnection is null

 ----------
Wed Mar 02 2016 19:08:28
Error: 2016-03-02 19:08:28	gloda.everybody	ERROR	!!! error initializing resource:///modules/gloda/index_ab.js

Source file: resource://app/modules/gloda/log4moz.js
Line: 691
 ----------
2016-03-02 19:08:28	gloda.everybody	ERROR	!!! error initializing resource:///modules/gloda/index_ab.js

 ----------
Wed Mar 02 2016 19:08:28
Error: 2016-03-02 19:08:28	gloda.everybody	ERROR	(resource:///modules/gloda/datastore.js:1550) TypeError: this.asyncConnection is null

Source file: resource://app/modules/gloda/log4moz.js
Line: 691
 ----------
2016-03-02 19:08:28	gloda.everybody	ERROR	(resource:///modules/gloda/datastore.js:1550) TypeError: this.asyncConnection is null

 ----------
Wed Mar 02 2016 19:08:29
Error: nounDef is undefined
Source file: resource://app/modules/gloda/gloda.js
Line: 1901
 ----------
Wed Mar 02 2016 19:08:30
Error: TypeError: this.asyncConnection is null
Source file: resource://app/modules/gloda/datastore.js
Line: 1418
 ----------
Wed Mar 02 2016 19:08:39
Error: TypeError: this.asyncConnection is null
Source file: resource://app/modules/gloda/datastore.js
Line: 1550


I don't know from which change in m-c this comes.

Comment 2

[Jorg K (CEST = GMT+2)]

Last good push on C-C:
Mon Feb 29, 23:19:03
Nightly Requested: Tue Mar 1, 13:45:56 is bad.
So your window is pretty small.

Must be one of these two:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=8ef94be995a4 - Tue Mar 1, 11:59:06
https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=8ef94be995a4

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=5cafa6f3019b - Tue Mar 1, 12:02:11
https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=5cafa6f3019b

Comment 3

[Jorg K (CEST = GMT+2)]

Typical error seems to be:
TypeError: msgNounDef is undefined at resources/glodaTestHelper.js:208
Source:
206  let msgNounDef = GlodaMessage.prototype.NOUN_DEF; <===
207  let orig_insertMessage = GlodaDatastore.insertMessage;
208  GlodaDatastore.insertMessage = msgNounDef.objInsert = function() {

So something has changed in JS.

Comment 4

[Wayne Mery (:wsmwk)]

agree we know the window, but let's let the keyword stand please until we have identified the checkin at fault.

Comment 5

[aleth [:aleth]]

Sqlite got updated in Bug 1248472, and the changelog says:
"Because of continuing security concerns, the two-argument version of of the seldom-used and little-known fts3_tokenizer() function is disabled unless SQLite is compiled with the SQLITE_ENABLE_FTS3_TOKENIZER."
https://www.sqlite.org/releaselog/3_11_0.html

Comment 6

[:aceman]

Attachment: patch

This patch according to bug 1248472 comment 9 works for me locally.

Comment 7

[Jorg K (CEST = GMT+2)]

Should this bug go to Toolkit / Storage like bug 1248472?

Comment 8

[:aceman]

Attachment: patch v2

In case you do not like to use that option for toolkit and Firefox, this enables it only when building Thunderbird and Seamonkey.

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 8726327 [details] [diff] [review]
patch v2

Since the tokenizer needs to be native and thread-safe and Firefox does not support binary extensions anymore, it does seem appropriate to only enable this for Thunderbird and SeaMonkey.

I'm a storage peer so I can r+ the semantic change, but I think https://wiki.mozilla.org/Build_config (which seems slightly out of date, but I think the intent still stands) strongly suggests that a build peer review this change too.  I got to that page via searching for "mozilla-central build changes peer"; it's possible there is a more up-to-date appropriate page that that page should redirect to or something.

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8726327 [details] [diff] [review]
patch v2

Review of attachment 8726327 [details] [diff] [review]:
-----------------------------------------------------------------

You don't need a build peer review on this since this has very little to do with the build system and more about sqlite3.

Comment 11

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8c7227510049

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/8c7227510049