1679987 - Allow nsTCharSeparatedTokenizer to be used in a range-based for loop

Status: RESOLVED FIXED

(Core :: XPCOM, task)

Description

[Simon Giesecke [:sg] [he/him]]

At the moment, using nsTCharSeparatedTokenizer (and its specializations) requires custom while loops like

      nsCCharSeparatedTokenizer tokenizer(headerValue, ';');
      while (tokenizer.hasMoreTokens()) {
        const nsDependentCSubstring& token = tokenizer.nextToken();

        // do something with token
      }

With an appropriate adapter, C++11 range-based for loops could be used instead, which would provide a simpler syntax, reduce scope of the tokenizer and increase regularity:

      for (const nsDependentCSubstring& token :
           nsTokenizedRange{nsCCharSeparatedTokenizer(headerValue, ';')}) {
        // do something with token
      }

Also, several uses of strtok could be transformed to use the tokenizers in a range-based way.

Comment 1

[Simon Giesecke [:sg] [he/him]]

Attachment: Bug 1679987 - Convert tokenizer specializations from classes to type aliases. r=#xpcom-reviewers

Comment 2

[Simon Giesecke [:sg] [he/him]]

Attachment: Bug 1679987 - Add nsTokenizedRange adapter to enable range-based for with tokenizers. r=#xpcom-reviewers

Depends on D98306

Comment 3

[Simon Giesecke [:sg] [he/him]]

Attachment: Bug 1679987 - Use nsTokenizedRange where easily possible. r=#xpcom-reviewers

Depends on D98307

Comment 4

[Simon Giesecke [:sg] [he/him]]

Attachment: Bug 1679987 - Replace strtok for Vary header by nsCCharSeparatedTokenizer and range-based for. r=#necko-reviewers,#dom-workers-and-storage

Depends on D98308

Comment 5

[Andrew McCreight (out until 2026) [:mccr8]]

Please provide an explanation of what you are doing and why. The two major parts of the review process are determining whether the goals of the author are good and whether they are being achieved, and without knowing what you are trying to do, that makes review much more difficult. Thanks.

Comment 6

[Simon Giesecke [:sg] [he/him]]

(In reply to Andrew McCreight [:mccr8] from comment #5)

Please provide an explanation of what you are doing and why. The two major parts of the review process are determining whether the goals of the author are good and whether they are being achieved, and without knowing what you are trying to do, that makes review much more difficult. Thanks.

Sorry, I added a description now.

Comment 7

[Simon Giesecke [:sg] [he/him]]

Attachment: Bug 1679987 - Implement nsTSubstring::Split using nsTokenizedRange. r=#xpcom-reviewers

Depends on D98307

Comment 8

[Simon Giesecke [:sg] [he/him]]

Attachment: Bug 1679987 - Make nsTokenizerFlags a scoped enum and make them a template argument to nsTCharSeparatedTokenizer. r=#xpcom-reviewers

There are no uses of nsTCharSeparatedTokenizer that require run-time
configuration of the flags, so having them a compile-time template
argument allows for generation of more efficient code.

This might not matter that much now, but a subsequent patch will add another
flag to allow merging the implementation of nsTSubstring::Split with
nsTCharSeparatedTokenizer, through which the compile-time evaluation will
become more relevant.

Depends on D98307

Comment 9

[Simon Giesecke [:sg] [he/him]]

Attachment: Bug 1679987 - Removed unused includes of nsCharSeparatedTokenizer.h. r=nika

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6568490004a6
Removed unused includes of nsCharSeparatedTokenizer.h. r=nika
https://hg.mozilla.org/integration/autoland/rev/6cfaa0523589
Convert tokenizer specializations from classes to type aliases. r=xpcom-reviewers,nika,necko-reviewers
https://hg.mozilla.org/integration/autoland/rev/e94505c67802
Add nsTokenizedRange adapter to enable range-based for with tokenizers. r=xpcom-reviewers,nika
https://hg.mozilla.org/integration/autoland/rev/ad59c7dccefb
Make nsTokenizerFlags a scoped enum and make them a template argument to nsTCharSeparatedTokenizer. r=xpcom-reviewers,necko-reviewers,nika
https://hg.mozilla.org/integration/autoland/rev/3fb5e713945b
Use nsTokenizedRange where easily possible. r=xpcom-reviewers,nika
https://hg.mozilla.org/integration/autoland/rev/f84ee933797d
Replace strtok for Vary header by nsCCharSeparatedTokenizer and range-based for. r=necko-reviewers,dom-workers-and-storage-reviewers,asuth,valentin

Comment 11

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/6568490004a6
https://hg.mozilla.org/mozilla-central/rev/6cfaa0523589
https://hg.mozilla.org/mozilla-central/rev/e94505c67802
https://hg.mozilla.org/mozilla-central/rev/ad59c7dccefb
https://hg.mozilla.org/mozilla-central/rev/3fb5e713945b
https://hg.mozilla.org/mozilla-central/rev/f84ee933797d

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/aa95b6225a74
Implement nsTSubstring::Split using nsTokenizedRange. r=xpcom-reviewers,necko-reviewers,nika

Comment 13

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/aa95b6225a74