1188991 - Rewrite nsCharSeparatedTokenizer on top of Tokenizer

Status: NEW

(Core :: XPCOM, defect)

Description

[Honza Bambas (:mayhemer)]

...or remove nsCharSeparatedTokenizer altogether and migrate usage to Tokenizer.

Needs a careful check, however based on quick look into MXR less work will probably be to wrap Tokenizer in nsCharSeparatedTokenizer for the starter.

Comment 1

[anjali.jha1903]

(In reply to Honza Bambas (:mayhemer) from comment #0)

...or remove nsCharSeparatedTokenizer altogether and migrate usage to
Tokenizer.

Needs a careful check, however based on quick look into MXR less work will
probably be to wrap Tokenizer in nsCharSeparatedTokenizer for the starter.

Hi I would like to work on this issue. Could you please guide me bit as to how to begin with this?
Thanks :)

Comment 2

[Honza Bambas (:mayhemer)]

An overview doc is here:
https://www.janbambas.cz/string-parsing-made-simple-with-mozillatokenizer/

Then read the header of the TokenizerBase class, it's a base template for char and wchar

Mainly understand the Next and Check methods usage. There are variants to Check* that will help you build a simple code.

Few good examples how you can use mozilla::Tokenizer are here:

• CacheControlParser
• proxy filter parsing

Comment 3

[anjali.jha1903]

Attachment: Bug 1188991 Rewrite nsCharSeparatedTokenizer on top of Tokenizer

Comment 4

[anjali.jha1903]

(In reply to Honza Bambas (:mayhemer) from comment #2)

An overview doc is here:
https://www.janbambas.cz/string-parsing-made-simple-with-mozillatokenizer/

Then read the header of the TokenizerBase class, it's a base template for char and wchar

Mainly understand the Next and Check methods usage. There are variants to Check* that will help you build a simple code.

Few good examples how you can use mozilla::Tokenizer are here:

• CacheControlParser
• proxy filter parsing

Hi! I am trying to apply the approach of declaring aSource in the header so that it is available for all the functions. Something like this

template <typename DependentSubstringType, bool IsWhitespace(char16_t)>
class nsTCharSeparatedTokenizer {
  typedef typename DependentSubstringType::char_type CharType;
  typedef typename DependentSubstringType::substring_type SubstringType;
  const SubstringType& aSource;

 public:
  // Flags -- only one for now. If we need more, they should be defined to
  // be 1 << 1, 1 << 2, etc. (They're masks, and aFlags is a bitfield.)
  enum { SEPARATOR_OPTIONAL = 1 };

  nsTCharSeparatedTokenizer(aSource,
                            CharType aSeparatorChar, uint32_t aFlags = 0)
      : mIter(aSource.Data(), aSource.Length()),
        mEnd(aSource.Data() + aSource.Length(), aSource.Data(),
             aSource.Length()),
        mSeparatorChar(aSeparatorChar),
        mWhitespaceBeforeFirstToken(false),
        mWhitespaceAfterCurrentToken(false),
        mSeparatorAfterCurrentToken(false),
        mSeparatorOptional(aFlags & SEPARATOR_OPTIONAL) {
    mozilla::TTokenizer<CharType> t(aSource);
    // Skip initial whitespace
    while (t.CheckWhite()) {
      mWhitespaceBeforeFirstToken = true;
    }
  }

Does this sound okay?

Comment 5

[anjali.jha1903]

I'll remove mIter and mEnd after I refactor the entire file. I just wanted to know if I am heading in the right direction

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months, so the assignee is being reset.

Comment 7

[Christian]

I would be interested in working on this. I have a long train ride coming up and am looking to make some open source contributions.