1478738 - Searchfox doesn't understand C++14 digit separators yet

Status: RESOLVED FIXED

(Webtools :: Searchfox, enhancement)

Description

[Jeff Walden [:Waldo]]

Searchfox prettyprints C++ text that contains C++14 digit separators -- that is, single quotes embedded within numbers -- as if the separators were really some sort of string delimiter, resulting in something like

  uint8_t a = 0b1100'1111;
  uint8_t b = 0b0111'1111;

having all text from the first ' to the second ' be colored (currently) green.  It instead should understand digit separators as merely another part of the surrounding numeric literal.

Filing in the Searchfox component for now 'cause that's where I see this, but I could imagine it really residing in whatever sub-component Searchfox uses to determine the structure of C++ text, but I don't know what that component might actually be.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

Do you have an example of this in m-c?

Comment 2

[Andrew McCreight [:mccr8]]

The URL field has an example from mfbt/Utf8.h

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Doh. Thanks!

Comment 4

[Kartikaya Gupta (email:[e-mail])]

From a glance through the code it looks like the is_ident loop at [1] gets used to parse numbers too? In which case we might need to special case this c++14 syntax there.

[1] https://github.com/mozsearch/mozsearch/blob/e014d6831c80b3d77b9647195a3e71f200b57374/tools/src/tokenize.rs#L255

Comment 5

[Botond Ballo [:botond]]

Why does a clang-based tool reimplement C++ lexing?

Comment 6

[Kartikaya Gupta (email:[e-mail])]

The syntax highlighting code is independent of the c++ code indexer. One of the advantages of this is it can do syntax highlighting of C++ code that doesn't get built.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

https://github.com/mozsearch/mozsearch/pull/121

Comment 8

[Kartikaya Gupta (email:[e-mail])]

This is deployed now.