Ah, so I looked into doing this in mozilla/application-services (e.g. rust-places), and there's a larger issue than tokenizers: FTS queries don't support non-prefix matches. That is to say, it only supports MATCH_BEGINNING, e.g. if you're searching for 'ample', there's no way to use the FTS index such that it would find 'example'. (`exam` would find it though, since it has good support for prefix matches). Because of this, it seems like you would need to always have the current logic present as a fallback. It's possible this would still be good enough, if we only use the fallback when the FTS query doesn't return enough results. Still, this would be a degradation in terms of search quality. It's worth noting that we'd need a custom tokenizer still to implement our notion of 'boundary' also.
Bug 1340487 Comment 9 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Ah, so I looked into doing this in mozilla/application-services (e.g. rust-places), and there's a larger issue than tokenizers: FTS queries don't support non-prefix matches. That is to say, it only supports MATCH_BEGINNING, e.g. if you're searching for 'ample', there's no way to use the FTS index such that it would find 'example'. (`exam` would find it though, since it has good support for prefix matches, although they can have some perf issues unless you add a prefix index). Because of this, it seems like you would need to always have the current logic present as a fallback. It's possible this would still be good enough, if we only use the fallback when the FTS query doesn't return enough results. Still, this would be a degradation in terms of search quality. It's worth noting that we'd need a custom tokenizer still to implement our notion of 'boundary' also.