1577383 - Implement `async iterable` declaration in WebIDL

Status: RESOLVED FIXED

(Core :: DOM: Bindings (WebIDL), task, P3)

Description

This bug is to discuss about implementing async iterable[1].
[1] https://heycam.github.io/webidl/#idl-async-iterable

Since DOMCursor has removed, we are looking for a replacement to implement enumerate function of DeviceStorage API[2].

[2] https://developer.mozilla.org/en-US/docs/Archive/B2G_OS/API/DeviceStorage/enumerate

How DeviceStorage gets the next File with enumerate() is handled asynchronously, caller (usually content process) asks for the next file, sends a request to parent, parent creates a blob per request, sends it back to content. The closest replacement of DOMCursor I can think of, it to define a new interface and make it async iterable, for example:

interface BlobList {
  async iterable;
};

Hello bz,
I believe Fabrice has already shared some background on this topic, we look forward to hearing your feedbacks, many thanks!

Comment 1

[Boris Zbarsky [:bzbarsky]]

So in terms of implementing this... You can probably reuse a lot of the existing iterator machinery, but with the various types set to Promise<whatever> and some changes to the actual codegen for the iterator's next() method.

The contract with the C++ side will need some thought, since you can't use the "get length, get key at index, get value at index" approach sync iterators use. Worse yet is the opaque "iterator state" bit.

What will probably need to happen is that the generated iterator will have a void* member which it will initialize by calling into the backing C++ implementation (to allocate whatever), then get updated on every call into the C++ backend, but it's worth thinking a bit about whether we can do better.

Given that, what we'd probably want to do is just have the iterator pass the Promise it's about to return to the C++ impl and tell it whether we want the key, value, or key+value corresponding to the iterator state and let it decide how it wants to resolve the promise... We can either create a template helper for "resolve this promise with this array of two totally different things" to handle the key+value case, or codegen a per-binding helper using the type information from the IDL.

Comment 2

[Shelly Lin [:shelly]]

Attachment: Bug 1577383 - Implement async iterable declaration in WebIDL.

Comment 3

[Shelly Lin [:shelly]]

Comment on attachment 9191830 [details]
Bug 1577383 - Implement async iterable declaration in WebIDL.

Not sure whom should I ask for patch review, can anyone give me some suggestion on the reviewer?

Comment 4

[Boris Zbarsky [:bzbarsky]]

Shelly, I'd try :peterv or :edgar.

Comment 5

[Jan Varga [:janv]]

This is needed in bug 1750730, but we are adding a workaround for that.

[Exposed=(Window,Worker), SecureContext, LegacyNoInterfaceObject]
interface FileSystemDirectoryIterator {
  Promise<any> next();
};

[Exposed=(Window,Worker), SecureContext, Pref="dom.fs.enabled"]
interface FileSystemDirectoryHandle : FileSystemHandle {
  // This interface defines an async iterable, however that isn't supported yet
  // by the bindings. So for now just explicitly define what an async iterable
  // definition implies.
  //async iterable<USVString, FileSystemHandle>;
  FileSystemDirectoryIterator entries();
  FileSystemDirectoryIterator keys();
  FileSystemDirectoryIterator values();

  // ...
}

Not sure, if we can safely live with the workaround or real support for async iterable would be better.

Comment 6

[[:fabrice] Fabrice Desré]

Shelly's patch is used in the b2g fork (https://github.com/kaiostech/gecko-b2g) and works well so far.

Comment 7

[Olli Pettay [:smaug][[e-mail]]]

edgar, peterv, does the patch look like something we could use?

Comment 8

[Jari Jalkanen]

Any update on this?

Comment 9

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Peter is reviewing the patch.

Comment 10

[Shelly Lin [:shelly]]

Attachment: Bug 1577383 - Implement async iterable declaration in WebIDL. r=peterv

Comment 11

[Shelly Lin [:shelly]]

Attachment: TestInterfaceJSMaplikeSetlikeIterableBinding.h

Comment 12

[Shelly Lin [:shelly]]

Attachment: TestInterfaceJSMaplikeSetlikeIterableBinding.cpp

Comment 13

[Jari Jalkanen]

Attachment: WIP: Bug 1577383 - Implement async iterable declaration in WebIDL.

This is copy and refactoring of https://phabricator.services.mozilla.com/D142527
for the purpose of resolving conflicts in the OPFS patch stack.
It can be abandoned in favor of the original patch D142527

Depends on D143740

Comment 14

[Shelly Lin [:shelly]]

I have the patch updated but got the following error when using moz-phab:
I've tried moz-phab install-certificate but doesn't help, and I'm sorry for throwing such config problem here...

TypeError: sequence item 0: expected str instance, int found
Run moz-phab again with '--trace' to show debugging output
Traceback (most recent call last):
File "/home/shelly/.local/lib/python3.6/site-packages/mozphab/mozphab.py", line 70, in main
check_for_updates(with_arc=not args.no_arc)
File "/home/shelly/.local/lib/python3.6/site-packages/mozphab/updater.py", line 73, in check_for_updates
pypi_info["version"], ".".join(required_python_version)
TypeError: sequence item 0: expected str instance, int found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/shelly/.local/bin/moz-phab", line 33, in <module>
sys.exit(load_entry_point('MozPhab==1.0.0', 'console_scripts', 'moz-phab')())
File "/home/shelly/.local/lib/python3.6/site-packages/mozphab/mozphab.py", line 110, in run
main(sys.argv[1:], is_development=False)
File "/home/shelly/.local/lib/python3.6/site-packages/mozphab/mozphab.py", line 105, in main
report_to_sentry(e)
File "/home/shelly/.local/lib/python3.6/site-packages/mozphab/sentry.py", line 45, in report_to_sentry
or isinstance(e, ssl.SSLCertVerificationError)
AttributeError: module 'ssl' has no attribute 'SSLCertVerificationError'
Sentry is attempting to send 2 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

Comment 15

[Olli Pettay [:smaug][[e-mail]]]

I was told using python 3.6, which is a bit old, might be the reason for that. (Bug 1772578 is related)

Comment 16

[Edgar Chen [:edgar]]

(In reply to Shelly Lin [:shelly] from comment #14)

Traceback (most recent call last):
File "/home/shelly/.local/bin/moz-phab", line 33, in <module>
sys.exit(load_entry_point('MozPhab==1.0.0', 'console_scripts', 'moz-phab')())
File "/home/shelly/.local/lib/python3.6/site-packages/mozphab/mozphab.py", line 110, in run
main(sys.argv[1:], is_development=False)
File "/home/shelly/.local/lib/python3.6/site-packages/mozphab/mozphab.py", line 105, in main
report_to_sentry(e)
File "/home/shelly/.local/lib/python3.6/site-packages/mozphab/sentry.py", line 45, in report_to_sentry
or isinstance(e, ssl.SSLCertVerificationError)
AttributeError: module 'ssl' has no attribute 'SSLCertVerificationError'
Sentry is attempting to send 2 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

I vaguely remember I also hit this with python3.6 and got fixed after upgrading to 3.8. It looks like SSLCertVerificationError isn't available in 3.6.

Comment 17

[Shelly Lin [:shelly]]

Thank you very much! I got it all setup.

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.

Comment 19

[Suhaib Mujahid [:suhaib]]

Sorry, there was a problem with the detection of inactive users. I'm reverting the change.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d97a870b61d7
Implement async iterable declaration in WebIDL. r=peterv

Comment 21

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/d97a870b61d7

Comment 22

[Takanori MATSUURA]

Is D147738 really "Changes Planed"?

Comment 23

[Randell Jesup [:jesup] (needinfo me)]

That was a copy of the landed patch; we'll obsolete it soon