1886331 - Syntax highlighting can break syntax highlighting on kotlin backtick identifier strings in test files because the tokenizer doesn't understand the backtick syntax

Status: NEW

(Webtools :: Searchfox, defect)

Description

[jackyzy823]

https://searchfox.org/mozilla-central/rev/09b59a7de6848639f27297888889a06e97561074/mobile/android/android-components/components/feature/customtabs/src/test/java/mozilla/components/feature/customtabs/CustomTabsToolbarFeatureTest.kt#246

It looks like ' (quote) between ` (backtick) prevent rendering correctly for kotlin files.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

The kotlin tokenizer spec doesn't seem to set backtick_strings to true, and it seems like perhaps the c-like tokenizer doesn't really understand backtick strings if that's not enabled.

It seems that kotlin's use of backticks is for escaping sketchy identifiers so the additional state machine that backtick_strings enables which exists for JS backtick strings might not be desirable to turn on. Probably the most pragmatic thing would be to do something like the rust_tweaks mechanism.

That said, there's an open enhancement bug to move to just using tree-sitter as our tokenizer and https://github.com/mozsearch/mozsearch/blob/master/tools/src/tree_sitter_support/cst_tokenizer.rs from the hyperblame enhancement work has a building block of that in place.