549594 - GlodaMsgSearcher needs to avoid generating clauses that the tokenizer will eat

Status: NEW

(MailNews Core :: Database, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

As discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=523183#c5 any search query where we generate an ANDed clause that the tokenizer completely eats will result in no results being returned.

We should simply gobble that which the tokenizer will ignore.

Unfortunately, the tokenizer is not currently exposed via XPCOM so we either need to approximate its behaviour or expose it via XPCOM.  Happily, approximating the behavior for most cases should be easy by using a regex to eat punctuation and then making sure what we are left with is long enough.