369066 - can't distinguish between for([x,y] in o) and for each(x in o) in o.__iterator__

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine, enhancement)

Description

[Yuh-Ruey Chen]

Attachment: example

Each of the following do something different:

for (var k in o) print(k);             // prints keys
for (var [k,v] in o) print(k+','+v);   // prints key-value pairs
for each (var t in o) print(t);        // prints values

However, when overriding o's __iterator__, there's no way to distinguish between the last two cases. __iterator__ accepts a single boolean parameter. This parameter is true in the 1st case, and false in the last two cases.

For example, suppose o was constructed from foo, and:

foo.prototype.__iterator__ = function(flag) {
	print(flag);
	return new Iterator(this, flag);
};

This is what happens to each case:

for (var k in o) print(k);             // prints true, then keys
for (var [k,v] in o) print(k+','+v);   // prints false, then key-value pairs
for each (var t in o) print(t);        // prints false, then key-value pairs

As you can see, originally the 2nd and 3rd cases do different things, but after overriding __iterator__, they are forced to do the same thing.

I'm asking for some mechanism to distinguish between for 2nd and 3rd cases.

Comment 1

[Brendan Eich [:brendan]]

The problem with the boolean flag argument is that it is impossible to usefully type the signature of iterator::get (nee __iterator__), or of the next method of the iterator object it returns.

So for ES4 there is no flag parameter, and the current proposal always either enumerates values for 'for each (v in o)' if o has the default iterator::get (from Object.prototype; the Pythonic alternative of not having any such method was not favored by TG1 members who wanted philosophical and practical unity of iteration protocol, meaning universal use of it -- even when enumerating); or else iterators over values returned by the next method of the iterator object returend from the non-default iterator::get method.

This means for (v in iter) and for each (v in iter) are equivalent when iter is an iterator object. More generally, for (v in o) and for each (v in o) where o does override iterator::get both iterate values returned by the iterator found or created for o. And any destructuring form simply destructures that value.

The alternative to a flag, which could be usefully typed, is to have iterator::get for for-in, and different getters for the other forms (for ([k,v] in o), for each (v in o)). But when iterating, as opposed to enumerating, there is no reason to make for ([k,v] in o) a special case -- any value could be returned, and any destructuring pattern should be allowed. (See bug 366941 for a glitch resulting from the for ([k,v] in o) special case excluding a useful value-destructuring inductive case.) And anyway, having multiple getters for different flavors of syntax multiplies implementation costs and user-facing complexity.

So I am in favor of making iteration the general case, and enumeration the backward-compatible default case that is also special in that it has for each (v in o) support per E4X (ECMA-357).

Up for debate is whether the for ([k,v] in o) special case for enumeration, where o has default iterator::get, is worth it. One problem with this special case in light of general iteration support (again see bug 366941) is that it can't be checked statically.

The full decision tree looks like this:

1. Iteration protocol introduced under for-in syntax? [y]
2. Iteration protocol generalized to subsume enumeration? [y]
3. Enumeration special forms (for-each-in, for([k,v] in)) generalized through iteration protocol (this bug's request) [n]

I think decision (1) is correct. New syntax would be more obscure and ugly. The old for-in syntax is often misused, and will continue to be misused, by users expecting iteration rather than enumeration (enumeration is specified in ECMA-262 12.6.4, by prose that talks about prototype walking and shadowing, the DontEnum attribute, and properties deleted after the loop starts not being visited).

Decision (2) is debateable, but with correct implementation a matter of taste. Either we follow Python and do not put a default iterator::get on Object.prototype (Python does not define __iter__ for all types), making its lack trigger backward compatible enumeration (Python throws if you for-loop over a non-iterable type, but by decision (1) we must be backward compatible). Recent debate in TG1 led to unification of enumeration and iteration by defining a default Object.prototype iterator::get that returns an Enumerator instance.

Decision (3), this bug's concern if you ignore the wrinkle of JS1.7's boolean flag argument, which leads to uselessly wide type signatures for the relevant methods (iterator::get, the next method of the returned iterator), results in for-in and for-each-in being equivalent for non-default iterators. But the iteration protocol does not visit keys or values of an object: it returns arbitrary values. So there is no distinction between "in" and "each in" (not that those keywords clearly connote key vs. value *enumeration*). Therefore it seems best to unify iteration under one protocol for the future, and leave for-each-in for E4X compatibility and enumeration-of-values convenience.

Comments welcome.

/be

Comment 2

[Brendan Eich [:brendan]]

Another alternative is to throw a TypeError if anyone uses for-each-in on an object having a non-default iterator::get. Not good for users who face deployed code where all paths weren't tested, or for programmers intentionally writing generic code that wants either (a) value enumeration of backward-compatible objects; (b) value iteration for objects with custom iterator::get methods.

A SyntaxError would be better, but it can't be done because ES4/JS2 will never require static judgments of the type of a value operand (in this case the right operand of |in| in the loop head). So no error is the best result, and I do think value enumeration/iteration unity via for-each-in will be used when writing generic loops.

In Python, one must know the type to know whether values (lists, tuples) or keys (dicts) are iterated by a for-in construct:

>>> [i for i in (1,2,3)]
[1, 2, 3]
>>> [i for i in [1,2,3]]
[1, 2, 3]
>>> [i for i in {"p":1,"q":2,"r":3}]
['q', 'p', 'r']

Whereas in ES4/JS2, you always get keys given backward-compatible objects if you use for-in, and you always get values given any objects if you use for-each-in.

If you use for ([k,v] in o), you get destructuring of the key and value as a convenience for backward-compatible objects, otherwise value destructuring for objects with custom iterators. Python requires you to say what you mean here:

>>> [v for k,v in {"p":1,"q":2,"r":3}.items()]
[2, 1, 3]

JS1.7 and proposed ES4/JS2 do more with less explicit runtime type information (i.e., with the same operand on the right of |in|):

js> [k for (k in {p:1,q:2,r:3})]
p,q,r
js> [v for each (v in {p:1,q:2,r:3})]
1,2,3
js> [v for ([k,v] in {p:1,q:2,r:3})]
1,2,3

But only for backward-compatible objects -- those delegating to the default Object.prototype.iterator::get.

Backward compatibility means we are stuck with the for-each-in vs. for-in distinction. Adding a for-[k,v]-in distinction to the backward-compatible tine of the fork may be unwanted complexity. It certainly should not break induction on the number of destructuring array elements when iterating over objects with custom iterators (bug 366941). But we may drop it from ES4. Comments again welcome.

/be

Comment 3

[Igor Bukanov]

What about dropping "for each" and special treatment of for ([k,v] in obj) for default iterators from ES4 completely? Given that one would need to declare that the script is ES4, not ES4, that indicator can also be used to indicate that the code also wants to avoid the whole E4X iterator mess.

Comment 4

[Brendan Eich [:brendan]]

(In reply to comment #3)
> What about dropping "for each" and special treatment of for ([k,v] in obj) for
> default iterators from ES4 completely? Given that one would need to declare
> that the script is ES4, not ES4, that indicator can also be used to indicate
> that the code also wants to avoid the whole E4X iterator mess.

While it's true that reserved identifiers added for ES4 require version guards, we try not to raise the barrier to migration gratuitously, so are not proposing to remove for-each-in support in ES4 at this point.

for-each-in is quite popular in JS1.6+ and AS3. I think we should not remove it, even though it has vague connotation, complicates the surface syntax slightly, and raises the iteration protocol question.

for-[k,v]-in is something we could drop, for sure. We could even add an iterator::items function (along with iterator::keys and iterator::values).

/be

Comment 5

[Igor Bukanov]

(In reply to comment #4)
> so are not proposing
> to remove for-each-in support in ES4 at this point.

What about deprecating it or at least making 

  for each (i in obj)

exactly equivalent to 

  for (i in Iter::vals(obj))

so it would be just a syntax sugar.

Comment 6

[Igor Bukanov]

(In reply to comment #4)
> We could even add an
> iterator::items function (along with iterator::keys and iterator::values).

A library of useful iterator transformers would be a nice addition. 

Comment 7

[Brendan Eich [:brendan]]

(In reply to comment #5)
> (In reply to comment #4)
> > so are not proposing
> > to remove for-each-in support in ES4 at this point.
> 
> What about deprecating it or at least making 
> 
>   for each (i in obj)
> 
> exactly equivalent to 
> 
>   for (i in Iter::vals(obj))
> 
> so it would be just a syntax sugar.

Already done in the wiki -- need to re-export it -- but only if obj uses the default iterator::get. Seems you are proposing that for-each-in is just a macro for the long form proposed:

for each (i in o) === for (i in iterator::Itemizer(o))

but this breaks the case where o has a non-default iterator::get and the user wants the values returned by the iterator for o, not whatever values might be in enumerable properties of o (and along o's prototype unless shadowed, but not yet deleted, etc. -- enumeration sucks, especially if o *is* a custom iterator with only 'next' as its enumerable property).

(In reply to comment #6)
> (In reply to comment #4)
> > We could even add an
> > iterator::items function (along with iterator::keys and iterator::values).
> 
> A library of useful iterator transformers would be a nice addition. 

Working on it, based on the itertools module from Python. I'll attach the JS1.7 implementation here.

/be

Comment 8

[Robert Sayre]

http://mochikit.com/doc/html/MochiKit/Iter.html

Comment 9

[Igor Bukanov]

(In reply to comment #7)
> Seems you are proposing that for-each-in is just a macro
> for the long form proposed:
> 
> for each (i in o) === for (i in iterator::Itemizer(o))

No, I am proposing 

  for each (i in o) === for (i in iterator::values(o))

where iterator::values generates precisely the same iteration sequence as the current for each implementation. That is, iterator::values !== iterator::Itemizer when o has non-default iterator.   

Comment 10

[Brendan Eich [:brendan]]

(In reply to comment #8)
> http://mochikit.com/doc/html/MochiKit/Iter.html

Seenit ;-). It is not able to take advantage of generators, but the JS1.7 and ES4 itertools knock-offs can, and the latter can use types too.

(In reply to comment #9)
> (In reply to comment #7)
> > Seems you are proposing that for-each-in is just a macro
> > for the long form proposed:
> > 
> > for each (i in o) === for (i in iterator::Itemizer(o))
> 
> No, I am proposing 
> 
>   for each (i in o) === for (i in iterator::values(o))
> 
> where iterator::values generates precisely the same iteration sequence as the
> current for each implementation. That is, iterator::values !==
> iterator::Itemizer when o has non-default iterator. 

Oh I see -- that's cool, good idea. Thanks,

/be

Comment 11

[Yuh-Ruey Chen]

Ugh, all this iterator vs. enumerator distinction is confusing. I get what's
going on now, but if it takes me this long to understand how default
enumeration, new Iterator(), Iterator(), __iterator__, for-each-in vs. for-in,
and destructuring assignment interact, I can only imagine what it would be like
for the average JS developer. Backwards compatibility is a pita.

Dropping iterator::get's boolean flag and trying to dissuade usage (deprecate?)
of for-each-in does help though.

The main problem is the joining of iterators and enumerators into a common
for-in syntax. How can I distinguish between the two in code? I encountered
this problem when trying to define a function that works with any iterable or
iterator:

function sum(o) {
    if (o instanceof Iterator) {
        for (let x in o)
            // do something
    } else { // enumeration
        for (let [x, y] in o)
            // do something
    } 
}

The problem with the above code is that there is no way to distinguish between
an iterator created by |Iterator()| and an enumerator created by |new
Iterator()| - both are instances of Iterator! I know |new Iterator()| is going
to be replaced with an Enumerator class in ES4, which would be very helpful.

(Another problem is that generators aren't instances of Iterator, but that
could be fixed in SpiderMonkey with |(function() { yield;
}()).__proto__.__proto__ = Iterator.prototype|.)

That leaves one problem (a bit OT though): The difference between Enumerator
and the default enumerator. Is there any reason why we can't get a reference to
the default enumerator? (I'm not asking for a way to change the default
enumerator which I think was made impossible in bug 354750.) Alternatively, is
there a way to customize an Enumerator to do the same thing as the default
enumerator?

Perhaps there could be a way to customize for-in and for-each-in for
Enumerators, so that given e is an Enumerator:

|for (k in e)| calls iterator::keys(e)
|for each (v in e)| calls iterator::values(e)
|for ([k,v] in e)| and |for each ([k,v] in e)| calls iterator::items(e)

It just needs to be really clear that Iterators and Enumerators are different,
and the only things they share are that they both are in for-in and that both
can be returned from iterator::get/__iterator__.

In summary, this how each for-in would be processed:

|for (k in o)|:
if o is null, do nothing
else if o is an Iterator, iterate with it
else if o is an Enumerator, do |for (k in iterator::keys(o))|
else do |for (k in default_enumerator(o))|

|for each (k in o)|:
if o is null, do nothing
else if o is an Iterator, iterate with it
else if o is an Enumerator, do |for (k in iterator::values(o))|
else do |for each (k in default_enumerator(o))|

|for ([k,v] in o)| or |for each ([k,v] in o)|:
if o is null, do nothing
else if o is an Iterator, iterate with it
else if o is an Enumerator, do |for ([k,v] in iterator::items(o))|
else do |for ([k,v] in default_enumerator(o))|

Generators would be instances of Iterator, and enumerators are instances of
Enumerator (and not Iterator).

Comment 12

[Yuh-Ruey Chen]

Oops, I forgot a case in my last post.

|for (k in o)|:
if o is null, do nothing
else if o is an Iterator, iterate with it
else if o is an Enumerator, do |for (k in iterator::keys(o))|
else if o.iterator::get exists, do |for (k in o.iterator::get())|
else do |for (k in default_enumerator(o))|

|for each (v in o)|:
if o is null, do nothing
else if o is an Iterator, iterate with it
else if o is an Enumerator, do |for (v in iterator::values(o))|
else if o.iterator::get exists, do |for each (v in o.iterator::get())|
else do |for each (v in default_enumerator(o))|

|for ([k,v] in o)| or |for each ([k,v] in o)|:
if o is null, do nothing
else if o is an Iterator, iterate with it
else if o is an Enumerator, do |for ([k,v] in iterator::items(o))|
else if o.iterator::get exists, do |for ([k,v] in o.iterator::get())|
else do |for ([k,v] in default_enumerator(o))|

Comment 13

[Brendan Eich [:brendan]]

(In reply to comment #11)
> Ugh, all this iterator vs. enumerator distinction is confusing. I get what's
> going on now, but if it takes me this long to understand how default
> enumeration, new Iterator(), Iterator(), __iterator__, for-each-in vs. for-in,
> and destructuring assignment interact, I can only imagine what it would be like
> for the average JS developer. Backwards compatibility is a pita.

The idea is to unify iteration and enumeration so they don't have to worry.

In that light, dropping the for-[k,v]-in special form is a good idea. I'll raise this with ECMA TG1.

> Dropping iterator::get's boolean flag and trying to dissuade usage (deprecate?)
> of for-each-in does help though.

Making for-each-in just a macro for a kind of for-in is the ticket.

> The main problem is the joining of iterators and enumerators into a common
> for-in syntax. How can I distinguish between the two in code?

Why do you need to?

> I encountered
> this problem when trying to define a function that works with any iterable or
> iterator:
> 
> function sum(o) {
>     if (o instanceof Iterator) {
>         for (let x in o)
>             // do something
>     } else { // enumeration
>         for (let [x, y] in o)
>             // do something
>     } 
> }

You should not be testing instanceof any nominal type. Iteration is structurally typed (even in JS1.7), not nominally typed.

> The problem with the above code is that there is no way to distinguish between
> an iterator created by |Iterator()| and an enumerator created by |new
> Iterator()| - both are instances of Iterator!

This is a JS1.7 botch. In ES4, the default iterator is an Enumerator (iterator::Enumerator, to give its full name).

> I know |new Iterator()| is going
> to be replaced with an Enumerator class in ES4, which would be very helpful.

Ah, good.

> (Another problem is that generators aren't instances of Iterator, but that
> could be fixed in SpiderMonkey with |(function() { yield;
> }()).__proto__.__proto__ = Iterator.prototype|.)

Again, you don't want instanceof checks.  In ES4, you can use

  obj is iterator::IteratorType

to test whether obj implements the iteration protocol, that is, matches the structural type

  type IteratorType = {next: function .<T>() : T};

which is parameterized by T (again this is in namespace iterator). This is an object containing at least a next method that returns a T.

There is an IterableType too:

  type IterableType = {
    iterator::get: function .<T>() : function () : IteratorType.<T>
  };

These are not nominal types, any object may implement them. They're structural record types.
 
> That leaves one problem (a bit OT though): The difference between Enumerator
> and the default enumerator. Is there any reason why we can't get a reference to
> the default enumerator? (I'm not asking for a way to change the default
> enumerator which I think was made impossible in bug 354750.) Alternatively, is
> there a way to customize an Enumerator to do the same thing as the default
> enumerator?

It is probably best not to write "default enumerator" but only "default iterator", which is the original value of Object.prototype.iterator::get. It's a function that looks like this:

  function () { return new iterator::Enumerator(this); }

As noted in my earlier comment, it could be named by a const in namespace iterator, and probably should be so named.

> Perhaps there could be a way to customize for-in and for-each-in for
> Enumerators, so that given e is an Enumerator:
> 
> |for (k in e)| calls iterator::keys(e)
> |for each (v in e)| calls iterator::values(e)
> |for ([k,v] in e)| and |for each ([k,v] in e)| calls iterator::items(e)
> 
> It just needs to be really clear that Iterators and Enumerators are different,
> and the only things they share are that they both are in for-in and that both
> can be returned from iterator::get/__iterator__.

I don't see why the distinction between enumerators and any other kind of iterator must be exposed in this way. Enumeration is just the default iteration protocol implementation. Or, perhaps we agree, there *will* be nominal types implementing enumeration for the cases you care about: Enumerator (for-in), Itemizer (for-each-in), and Iterator (for-[k,v]-in, if we keep it around).

> In summary, this how each for-in would be processed:
> 
> |for (k in o)|:
> if o is null, do nothing

or if o is undefined -- note that this is a variance from ES1-3 based on real-world web browser inter-operation requirements (thank you IE).

> else if o is an Iterator, iterate with it
> else if o is an Enumerator, do |for (k in iterator::keys(o))|
> else do |for (k in default_enumerator(o))|

Ah, I see the confusion. You don't need this many lines to express the one, unified iteration protocol.  All you need is

  assert(o is IteratorType.<*>); iterate with it

That's it. The rest are particular implementations of the iteration protocol.

Does this make sense? You need to take the painful step of rejecting nominal types and instanceof, first ;-).

/be

Comment 14

[Brendan Eich [:brendan]]

Perhaps it would help to see class Enumerator from the (unexported since last November, IIRC) wiki proposal:

class Enumerator {
  function Enumerator(obj : Object)
    : initial_obj = obj
    , current_obj = obj
    , current_ids = magic::getEnumerableIds(obj);
  {
  }
 
  intrinsic static function invoke(obj : Object) : IteratorType.<string>
  {
    return new Enumerator(obj);
  }
 
  iterator function get() : IteratorType.<string>
    this;
 
  public function next() : string
  {
    if (current_obj === null)
      throw StopIteration;
 
  loop:
    while (true) {
      if (current_index === current_ids.length) {
        // No more properties in current_obj: try walking up the prototype chain.
        current_obj = magic::getPrototype(current_obj);
        if (current_obj === null)
          throw StopIteration;
 
        current_ids = magic::getEnumerableIds(current_obj);
        current_index = 0;
      }
 
      // Get the name of the next property in current_obj that lacks DontEnum.
      let name : string = current_ids[current_index++];
 
      // Check for a shadowing property from initial_obj to current_obj on the prototype chain.
      for (let obj = initial_obj; obj !== current_obj; obj = magic::getPrototype(obj)) {
        if (magic::hasOwnProperty(obj, name))
          continue loop;
      }
 
      // Check whether name is still bound in order to skip deleted properties.
      if (magic::hasOwnProperty(current_obj, name))
        return name;
    }
  }
 
  private var initial_obj : Object,
              current_obj : Object,
              current_ids : Array,
              current_index : uint;
}

This is ES4/JS2. Note how the class implements the iteration protocol -- that is, its instances are structural subtypes of iterator::IteratorType (and of course also of iterator::IterableType by virtue of the iterator function get definition).

There are not three special-case instanceof tests in the algorithm for for-in, again. There's just the null||undefined special case for web compatibility, and then the iterator::get call. The implementations are many and varied, and the Enumerator class is just what the default iterator::get (the one on Object.prototype) creates and returns.

/be

Comment 15

[Brendan Eich [:brendan]]

Note on the code in the last comment: the magic::getEnumerableIds and other magic helpers are implemented by the SML reference implementation. They are where the self-hosted ES4 implementation "bottoms out" and cannot express certain primitives in the new language, but must reach into the host runtime for help.

/be

Comment 16

[Brendan Eich [:brendan]]

Another note, for readers confused by:

  iterator function get() : IteratorType.<string>
    this;

This is an expression closure, a shorthand whereby you can write

  function square(x) { return x * x; }

as

  function square(x) x * x;

/be

Comment 17

[Yuh-Ruey Chen]

Okay, with ducktyping it becomes more manageable.

function sum(o) {
    let s = 0;
    if (o is iterator::IteratorType) {
        for (let x in o)
            s += x;
    } else {
        for each (let x in o)
            s += x;
    }
    return s;
}

It would be nice if that could be simplified down to:

function sum(o) {
    let s = 0;
    for each (let x in o)
        s += x;
    return s;
}

but if |for each(let x in o)| is translated into |for (let x in iterator::values(o)| and o is an iterator, then it would iterate over the iterator's values rather than o itself. Or am I wrong?

(In reply to comment #13)
> (In reply to comment #11)
> > Ugh, all this iterator vs. enumerator distinction is confusing. I get what's
> > going on now, but if it takes me this long to understand how default
> > enumeration, new Iterator(), Iterator(), __iterator__, for-each-in vs. for-in,
> > and destructuring assignment interact, I can only imagine what it would be like
> > for the average JS developer. Backwards compatibility is a pita.
> 
> The idea is to unify iteration and enumeration so they don't have to worry.
> 
> In that light, dropping the for-[k,v]-in special form is a good idea. I'll
> raise this with ECMA TG1.

What do you mean by "special form"? I hope you just mean that it won't be valid for enumerators. |for ([x,y] in o)| is really useful if o is an iterator.

> > That leaves one problem (a bit OT though): The difference between Enumerator
> > and the default enumerator. Is there any reason why we can't get a reference to
> > the default enumerator? (I'm not asking for a way to change the default
> > enumerator which I think was made impossible in bug 354750.) Alternatively, is
> > there a way to customize an Enumerator to do the same thing as the default
> > enumerator?
> 
> It is probably best not to write "default enumerator" but only "default
> iterator", which is the original value of Object.prototype.iterator::get. It's
> a function that looks like this:
> 
>   function () { return new iterator::Enumerator(this); }
> 
> As noted in my earlier comment, it could be named by a const in namespace
> iterator, and probably should be so named.

Right now, |new Iterator(o)| doesn't do the same thing as the default iterator. It doesn't go down the prototype chain - is that a bug? If it is, I find it a useful bug for some purposes :)

> > else if o is an Iterator, iterate with it
> > else if o is an Enumerator, do |for (k in iterator::keys(o))|
> > else do |for (k in default_enumerator(o))|
> 
> Ah, I see the confusion. You don't need this many lines to express the one,
> unified iteration protocol.  All you need is
> 
>   assert(o is IteratorType.<*>); iterate with it
> 
> That's it. The rest are particular implementations of the iteration protocol.

So let me get this straight. |for (x in o)| would go like this:
if x === null or x === undefined, do nothing
else if x is IteratorType, iterate with it
else do |for (x in o.iterator::get())|

(In reply to comment #14)
> Perhaps it would help to see class Enumerator from the (unexported since last
> November, IIRC) wiki proposal:

Seems like |new Iterator()|'s behavior is a bug then...

(In reply to comment #15)
> Note on the code in the last comment: the magic::getEnumerableIds and other
> magic helpers are implemented by the SML reference implementation. They are
> where the self-hosted ES4 implementation "bottoms out" and cannot express
> certain primitives in the new language, but must reach into the host runtime
> for help.
> 
> /be
> 

Is that magic namespace usable by script authors?

Comment 18

[Brendan Eich [:brendan]]

(In reply to comment #17)
> It would be nice if that could be simplified down to:
> 
> function sum(o) {
>     let s = 0;
>     for each (let x in o)
>         s += x;
>     return s;
> }
> 
> but if |for each(let x in o)| is translated into |for (let x in
> iterator::values(o)| and o is an iterator, then it would iterate over the
> iterator's values rather than o itself. Or am I wrong?

No, you've got it.  There is absolutely no reason to write the if (o is iterator::IteratorType) test of the then clause. Just use for-each-in.

> > In that light, dropping the for-[k,v]-in special form is a good idea. I'll
> > raise this with ECMA TG1.
> 
> What do you mean by "special form"? I hope you just mean that it won't be valid
> for enumerators. |for ([x,y] in o)| is really useful if o is an iterator.

Yes, absolutely -- that's what bug 366941 wants unregressed. It should always work to destructure a value returned from any iterator, including an enumerator. The special form I'm talking about is not the syntax for([k,v] in o) -- it is the meaning attached to it in JS1.7 that destructures key and value for o an object without an __iterator__ property.

> Right now, |new Iterator(o)| doesn't do the same thing as the default iterator.
> It doesn't go down the prototype chain - is that a bug? If it is, I find it a
> useful bug for some purposes :)

It is useful not to go over the prototype chain but simply iterate keys in the directly referenced object.

> So let me get this straight. |for (x in o)| would go like this:
> if x === null or x === undefined, do nothing
> else if x is IteratorType, iterate with it
> else do |for (x in o.iterator::get())|

No, the last two lines collapse because an iterator's iterator::get returns the iterator itself.

> (In reply to comment #14)
> > Perhaps it would help to see class Enumerator from the (unexported since last
> > November, IIRC) wiki proposal:
> 
> Seems like |new Iterator()|'s behavior is a bug then...

In hindsight, sure. It's a change for ES4/JS2.

> (In reply to comment #15)
> Is that magic namespace usable by script authors?

No, certainly not -- it's a specification device used in the reference implementation.

/be

Comment 19

[Yuh-Ruey Chen]

OIC, I wasn't paying attention to iter.iterator::get() returning iter. It makes sense now, thanks :)

(In reply to comment #18)
> (In reply to comment #17)
> > Right now, |new Iterator(o)| doesn't do the same thing as the default iterator.
> > It doesn't go down the prototype chain - is that a bug? If it is, I find it a
> > useful bug for some purposes :)
> 
> It is useful not to go over the prototype chain but simply iterate keys in the
> directly referenced object.

It is indeed useful, but if |new Iterator| is going to be replaced with the Enumerator class you posted above, does that mean we'll have to write our own enumerators that don't go down the prototype chain, or will there still be a built-in one?

Comment 20

[Igor Bukanov]

(In reply to comment #17)
> function sum(o) {
>     let s = 0;
>     if (o is iterator::IteratorType) {
>         for (let x in o)
>             s += x;
>     } else {
>         for each (let x in o)
>             s += x;
>     }
>     return s;
> }
> 
> It would be nice if that could be simplified down to:
> 
> function sum(o) {
>     let s = 0;
>     for each (let x in o)
>         s += x;
>     return s;
> }
> 
> but if |for each(let x in o)| is translated into |for (let x in
> iterator::values(o)| and o is an iterator, then it would iterate over the
> iterator's values rather than o itself. Or am I wrong?

"for each" (and proposed iterator::values will do the same) over obj always uses iterator::get to get the iterator. Since for iterators and generators obj.iterator::get() (obj.__iterator__() in JS 1.7) returns the obj itself, that check is unnecessary and simple 

     for each (let x in o)
         s += x;

works for the iterators / generators and normal objects both in JS1.7 and ES4.

Comment 21

[Igor Bukanov]

(In reply to comment #9)
> No, I am proposing 
> 
>   for each (i in o) === for (i in iterator::values(o))
> 
> where iterator::values generates precisely the same iteration sequence as the
> current for each implementation. That is, iterator::values !==
> iterator::Itemizer when o has non-default iterator.   

Perhaps better name for the method would be iterator::forEach rather then iterator::values?

Comment 22

[Jeff Walden (gone starting June 8)]

(In reply to comment #21)
> Perhaps better name for the method would be iterator::forEach rather then
> iterator::values?

I think forEach is already taken, conceptually, by Array[.prototype].forEach.  I could even easily see iterator::forEach having functionality analogous to Array.forEach, except for iterators, as a user-supplied extension or even as a built-in -- and in either case, I think that name's not available.

Comment 23

[Yuh-Ruey Chen]

(In reply to comment #20)
> "for each" (and proposed iterator::values will do the same) over obj always
> uses iterator::get to get the iterator. Since for iterators and generators
> obj.iterator::get() (obj.__iterator__() in JS 1.7) returns the obj itself, that
> check is unnecessary and simple 
> 
>      for each (let x in o)
>          s += x;
> 
> works for the iterators / generators and normal objects both in JS1.7 and ES4.
> 

Yeah, I can't believe I didn't try that in JS1.7. I just assumed it didn't work...

(In reply to comment #22)
> (In reply to comment #21)
> > Perhaps better name for the method would be iterator::forEach rather then
> > iterator::values?
> 
> I think forEach is already taken, conceptually, by Array[.prototype].forEach. 
> I could even easily see iterator::forEach having functionality analogous to
> Array.forEach, except for iterators, as a user-supplied extension or even as a
> built-in -- and in either case, I think that name's not available.
> 

Isn't each Array.x iterator going to have an iterator::x counterpart anyway in ES4 (e.g. iterator::filter)? And I thought Array.forEach was going to be renamed to Array.each (in which case it might make sense to have the for-each-in iterator be named iterator::each).

Comment 24

[Brendan Eich [:brendan]]

(In reply to comment #19)
> It is indeed useful, but if |new Iterator| is going to be replaced with the
> Enumerator class you posted above, does that mean we'll have to write our own
> enumerators that don't go down the prototype chain, or will there still be a
> built-in one?

Default-true enumerate flag parameter to items/keys/values and their underlying native class:

values(obj, false) => just the direct enumerable property values
values(obj, true)  or
values(obj)        => what |for each (value in obj)| iterates over

etc. Comments?

No plans to rename Array.forEach at this point. Do want itertools and other fns in the iterator namespace. More in a bit.

/be

Comment 25

[Igor Bukanov]

(In reply to comment #24)
> Default-true enumerate flag parameter to items/keys/values and their underlying
> native class:
> 
> values(obj, false) => just the direct enumerable property values
> values(obj, true)  or
> values(obj)        => what |for each (value in obj)| iterates over

What about iterator::default(obj, false|true)? "values" as a name does not capture the fact that the function uses iterator::get() to get the iterator.

Comment 26

[Cris Mooney]

This old thread becomes "as close to what I can find" that is relevant again as my website breaks on a recent Firefox release. Somewhere between v18 and v25 Firefox "for...in" on a form started returning index numbers for "indexes" whereas previously it returned "names" for inputs (matching IE). The simple test http://crism/standalone/mozilla_for_in.html iterates as "textA" and "textB" (and other extraneous properties) in Firefox v18 and IE 8, while Firefox v21 gives "0" and "1". Note: Win XP tested.

I'm unable to find where this was changed, and thus a better place to raised the issue. Can someone with better knowledge of the product either address this here, indicating this is an acceptable place for the talk, or direct me elsewhere (perhaps then removing this comment if it is felt to be extraneous to this defect)?

Thank you.

Comment 27

[Cris Mooney]

Typo corrections to above (I can't find an "edit"):
- I meant: index numbers for "inputs" whereas, not "indexes".
- WAN test URL is http://forus.com/csm/bugs/mozilla_for_in.html.
(In reply to Cris Mooney from comment #26)
> "indexes" whereas previously it returned "names" for inputs (matching IE).
...
> The simple test http://crism/standalone/mozilla_for_in.html iterates as

Comment 28

[Tom S. (please needinfo tschuster)]

__iterator__ should go away. bug 1098412