1199775 - mozilla::Tokenizer improvements, vol 2

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Comment 1

[Honza Bambas (:mayhemer)]

Attachment: wip1

Comment 2

[Honza Bambas (:mayhemer)]

Attachment: v1

- tokens returned by Next() and Check() now expose "fragment" which is a substring (reference, no copy!) from the input buffer representing the token
- introduced ReadChar(classifier_function) that works the same way as CheckChar(classifier_function) but also returns the value of the character read
- introduced Claim() overload that works with nsDependentCSubstring to store the result -> no memcopy


https://treeherder.mozilla.org/#/jobs?repo=try&revision=490f1b4c09f8

Comment 3

[Honza Bambas (:mayhemer)]

one more:
- internal cursors and state flags made protected: only to allow derived classes do more job more conveniently

Comment 4

[Honza Bambas (:mayhemer)]

Attachment: v1.1

slightly improved:
- HasFailed() was not in two cases reverted to false after a successful parse operation
- test enhanced according that

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4079a4002f27

Comment 5

[Nathan Froyd [:froydnj]]

Comment on attachment 8655972 [details] [diff] [review]
v1.1

Review of attachment 8655972 [details] [diff] [review]:
-----------------------------------------------------------------

::: xpcom/ds/Tokenizer.h
@@ +79,5 @@
>      char AsChar() const;
>      nsDependentCSubstring AsString() const;
>      uint64_t AsInteger() const;
> +
> +    nsDependentCSubstring Fragment() const { return mFragment; }

I wonder if AsString and Fragment should return |const nsDependentCSubstring|...if such values are even feasible to use, depending on how const-correct ns*String's API is.  Not something to fix in this bug, but something to think about for later.  File a bug, please?

Comment 6

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3f38aafec98d

Comment 7

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/3f38aafec98d