937874 - Support non-UTF-8 encodings of source files

Status: RESOLVED WONTFIX

(Webtools Graveyard :: DXR, defect)

Description

[Erik Rose [:erik][:erikrose]]

Some codebases include non-UTF-8-encoded files, and we'd like to be able to index and search them sanely.

Had a conversation in #static on 2013-10-24 about trilite and char encodings. Due to the lucky nature of UTF-8, where any >127 byte I were to search for could not, once encoded into UTF-8, possibly match starting on a non-0th octet of a multibyte sequence (because those all have the high-order bit set), we're safe from false positives as long as we...
  * UTF-8 encode all search queries
  * Store only UTF-8 in trilite indices (so we might have to transcode sometimes)

However, the extents that come out of trilite mean bytes rather than chars. So either we have to change that (and teach trilite how to count UTF-8 chars) or do the mapping after the fact.

There's also the fun of clang, which emits its offsets in bytes rather than chars, I believe.

A course-grained start on this is at https://github.com/mozilla/dxr/pull/165.

Comment 1

[Sylvestre Ledru [:Sylvestre]]

DXR is no longer available. Searchfox is now replacing it.
See meta bug 1669906 & https://groups.google.com/g/mozilla.dev.platform/c/jDRjrq3l-CY for more details.