1572644 - Support "import" in worklet scripts

Status: RESOLVED FIXED

(Core :: DOM: Core & HTML, enhancement, P3)

Description

[Karl Tomlinson (:karlt)]

+++ This bug was initially created as a clone of Bug #1473463 +++

ScriptLoader supports fetching module script graphs but currently has a number of main-thread-only characteristics. The approach proposed in bug 1311726 would permit using this from worklet code.

Jon said in https://bugzilla.mozilla.org/show_bug.cgi?id=1290021#c34
"It would be really great if we could reuse some of the code in the script
loader (and would be useful for other efforts, e.g. bug 1308512). This is not
trivial unfortunately. I think it would require factoring out the module
loader into a separate class with a caller-provided a fetch interface passed
in."
This would be bug 1311726.

Andrea said in https://bugzilla.mozilla.org/show_bug.cgi?id=1472324#c1
'we must unify WorkletFetchHandler,
workerinternals::Load/LoadMainScript and mozilla::dom::ScriptLoader.'

Ben said in https://bugzilla.mozilla.org/show_bug.cgi?id=1442776#c1
"Would it not be possible to make worklets and workers completely separate?
Decoupling them seems like it would be nice for maintenance since they are
going to be fairly different."
That probably means that any shared code should be in or near ScriptLoader,
rather than in dom/worklers.

Houdini WG said in
https://github.com/w3c/css-houdini-drafts/issues/506#issuecomment-343276228
"RESOLVED: dynamic import() in worklets is a no-op that returns a rejected
promise (or equivalent after passing thru JS people)"

Web platform tests expect static import to work.
https://github.com/w3c/css-houdini-drafts/issues/506#issuecomment-342796390

Comment 1

[Paul Adenot (:padenot)]

It would be great if we could revisit priorities here. People expect imports to work in AudioWorklet, that we just enabled in Nightly (it's supposed to work per spec, and it works in Chromium). I'm seeing some websites that rely on this. AudioWorklet usage is still not widespread however.

https://superpowered.com/webbrowserlatency
https://mtg.github.io/essentia.js/examples
https://googlechromelabs.github.io/web-audio-samples/audio-worklet/

Comment 2

[Hsin-Yi Tsai (she/her) [:hsinyi]]

(In reply to Paul Adenot (:padenot) from comment #1)

It would be great if we could revisit priorities here. People expect imports to work in AudioWorklet, that we just enabled in Nightly (it's supposed to work per spec, and it works in Chromium). I'm seeing some websites that rely on this. AudioWorklet usage is still not widespread however.

https://superpowered.com/webbrowserlatency
https://mtg.github.io/essentia.js/examples
https://googlechromelabs.github.io/web-audio-samples/audio-worklet/

Thanks for raising this. Would take to our roadmap planning discussion. :)

Comment 3

[Paul Adenot (:padenot)]

https://ide.csound.com/

Comment 4

[Anne (:annevk)]

Also bringing this to the attention of Adam and Jens.

Comment 5

[johannes.feulner]

(In reply to Hsin-Yi Tsai [:hsinyi] from comment #2)

(In reply to Paul Adenot (:padenot) from comment #1)

It would be great if we could revisit priorities here. People expect imports to work in AudioWorklet, that we just enabled in Nightly (it's supposed to work per spec, and it works in Chromium). I'm seeing some websites that rely on this. AudioWorklet usage is still not widespread however.

Here is one more Website using WASM in AudioWorklet: https://ssc.scorio.com/localtest.html

Comment 6

[hjack]

I'm part of a team using WASM in AudioWorklets, via Superpowered, and would greatly appreciate this feature! We are trying to add Mozilla support and facing complications because import is not supported.

http://samples.landr.com/

Comment 7

[Jack Atherton]

I'm not sure if this is the right place for this.

Webchuck (ChucK in the browser) could also benefit from the ability to "import" in the AudioWorklet.

This version works in Chrome but not Firefox, exhibiting the "Module resolve hook not set" error message.

As suggested in 1636121, I've made a version that uses --extern-post-js to avoid the use of any imports. This faces the new bug "Module metadata hook not set", for which I haven't found any useful workarounds / bug reports.

This version works in Chrome but not Firefox, exhibiting the new bug. I'm not sure if this is completely unrelated, or if I've done something wrong in my attempt to eliminate import statements / ES6. Perhaps I can't use the -s MODULARIZE=1 option in Emscripten, even though I'm not using it alongside import statements?

Comment 8

[Jack Atherton]

It turns out I made a very silly mistake and did not remove the flag that was enabling ES6 support. I was able to flatten all of the imports, etc. into a single extern-post-js file, and things seem to work now. Here's the command that seems to be working for ChucK in Firefox, in case it helps anyone else facing this bug:

host_web/webchuck/js/webchuck.js: $(EMSCRIPTENSRCS)
	emcc -O3 -s DISABLE_EXCEPTION_CATCHING=0 \
        [many flags related to ChucK internals]
	-Wformat=0 \
        $(EMSCRIPTENSRCS) -o host_web/webchuck/js/webchuck.js \
        -s EXTRA_EXPORTED_RUNTIME_METHODS='["ccall", "cwrap", "getValue", "setValue", "addFunction", "removeFunction", "UTF8ToString", "stringToUTF8"]' \
        -s LINKABLE=1 -s MODULARIZE=1 -s 'EXPORT_NAME="ChucK"' \
        -s ALLOW_MEMORY_GROWTH=1 \
        -s --extern-post-js host_web/chucknode-postjs.js \
        -s RESERVED_FUNCTION_POINTERS=50 -s FORCE_FILESYSTEM=1

Comment 9

[Anne (:annevk)]

Clearing needinfos in favor of bug 1311726 comment 2 as that bug is the main thing that needs fixing here as I understand it.

Comment 10

[Maxime Chevalier-Boisvert]

Hello. I got here because I also stumbled upon this bug while trying to build a modular-synth-in-a-browser app that uses AudioWorklet. ES6 support in AudioWorklet would be much appreciated :)

Comment 11

[guest271314]

Still an issue on Firefox 96.

Error: Module resolve hook not set self-hosted:2022:31
    CallModuleResolveHook self-hosted:2022
    InnerModuleLinking self-hosted:2235
    declarationInstantiation self-hosted:2205
Uncaught (in promise) DOMException: The operation was aborted. 

Comment 12

[Hauke Krüger]

Hi, is there any recipe how to get wasm audioworklets to work with Firefox these days? It seems that even the links from above,

https://superpowered.com/webbrowserlatency
https://googlechromelabs.github.io/web-audio-samples/audio-worklet/

do not work with Firefox 101.

I tried so many options in the emcc command but none works. And it is always the same error message in the JS console. A proper error report or an explanation which options to use to get any example working would be greatly appreciated!

Comment 13

[Lukas Lihotzki]

rollup can be used to bundle worklet modules before publishing, so they can be loaded without imports.

rollup also runs in the browser. This can be used to implement a polyfill to work around this issue purely on the client side: https://gist.github.com/lukaslihotzki/b50ccb61ff3a44b48fc4d5ed7e54303f

Comment 14

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 1-1: Move WorkletFetchHandler to its own file.

This is a refactoring to move WorkletFetchHandler to its own file, so
later other classes can include its header.

Comment 15

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 2: Init WorkletModuleLoader when WorkletGlobalScope is created.

Comment 16

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 3: Add StartModuleLoadRunnable.

Add a Runnable class called StartModuleLoadRunnable, which is dispatched
from main thread to worklet thread to call
ModuleLoadRequest::StartModuleLoad.

Comment 17

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 4: Add WorkletLoadContext and nsMainThreadPtrHandle to delegate WorkletFetchHandler on the worklet thread.

Add WorkletLoadContext to delegate WorkletFetchHandler.

Comment 18

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 5-2: Implement WorkletModuleLoader::StartFetch with WorkletFetchHandler::StartFetch method.

Move the fetch code into WorkletFetchHandler::StartFetch, which will be
called by WorkletModuleLoader::StartFetch.

Comment 19

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 6: Set CORS and referrer on the Request.

Comment 20

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 7-2: Add FetchCompleteRunnable to call OnFetchComplete.

Add a class called WorkletScriptHandler to handle the loading of each
module script.

Add a Runnable class called FetchCompleteRunnable to be dispatched from
main thread to worklet thread to call
ModuleLoadRequest::OnFetchComplete.

Comment 21

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 8: Compile fetched module.

The compile options is from ExecutionRunnable::ParseAndLinkModule of the
previous patch, or could be found in
https://searchfox.org/mozilla-central/rev/fb9a504ca73529fa550efe488db2a012a4bf5169/dom/worklet/Worklet.cpp#386-389

Comment 22

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 9: Create static import.

Comment 23

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 10: Implement WorkletModuleLoader::OnModuleLoadComplete.

Comment 24

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 11-1: Add a virtual method GetResolveFailureMessage in ModuleLoaderBase.

Comment 25

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 12-1: Using StructuredCloneHolder to pass a JS::Value from a worklet thread to main thread.

Comment 26

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 13: Update ini files in wpt/.

There are still some failed test cases in audio-worklet-referrer.https.html.ini,
which is due to https://bugzilla.mozilla.org/show_bug.cgi?id=1808189.

Comment 27

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 5-1: Remove mErrorStatus and mURL from WorkletFetchHandler.

Unify to reject the promise with AbortError per spec requirement.

Also WorkletFetchHandler will fetch not only the top level module script
but also its dependent modules, so remove the mURL as well.

Comment 28

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 1-2: Add WorkletScriptHandler.

Add a class called WorkletScriptHandler to handle the loading of each
module script.

Comment 29

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 7-1: Add DispatchFetchCompleteToWorklet.

Dispatch error to worklet thread.

Comment 30

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 7-5: Add WorkletLoadContext into CC.

To fix the compilation error when traversing mHandler in
WorkletLoadcontext.

Comment 31

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 7-1: Add a hash table to map from nsIURI to ModuleLoadRequest.

Add a nsRefPtrHashtable<nsURIHashKey, ModuleLoadRequest> called mFetchingRequests
to map from nsIURI to ModuleLoadRequest.

When WorkletModuleLoader::StartFetch is called, it will insert an entry
into the table. Later when the main thread finishes the fetching of the module
script, we will use the URI to get the ModuleLoadRequest on worklet
thread and call ModuleLoadRequest::OnFetchComplete.

Comment 32

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 13: Get the localized message on the main thread.

On Worklet thread, we just return the error message with "{0},{1}",
where {0} is the ResolveErrorInfo message and {1} is the specifier.

Later when we are on the main thread, we extract the error info and
specfier and use them to get the localized error message.

Comment 33

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 11-2: Preload localized error msg and format it when resolving failed.

Comment 34

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 12-2: Add mErrorToRethrow to hold the JS::Value in WorkletFetchHandler.

Comment 35

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Attachment: Bug 1572644 - Part 14: Add test case for fetch a child module script failed.

Comment 36

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Thanks for jonco's and yulia's reviews.
As soft code freeze is coming, I'll push the patches next week.

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7890e4826b89
Part 1-1: Move WorkletFetchHandler to its own file. r=jonco
https://hg.mozilla.org/integration/autoland/rev/81fea615d358
Part 1-2: Add WorkletScriptHandler. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/b19342581003
Part 2: Init WorkletModuleLoader when WorkletGlobalScope is created. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/24c5f2c86eda
Part 3: Add StartModuleLoadRunnable. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/e030ffd2f51c
Part 4: Add WorkletLoadContext and nsMainThreadPtrHandle to delegate WorkletFetchHandler on the worklet thread. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/27da560927f4
Part 5-1: Remove mErrorStatus and mURL from WorkletFetchHandler. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/01b87c41932c
Part 5-2: Implement WorkletModuleLoader::StartFetch with WorkletFetchHandler::StartFetch method. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/001fd6c7a59a
Part 6: Set CORS and referrer on the Request. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/074a822a5c64
Part 7-1: Add a hash table to map from nsIURI to ModuleLoadRequest. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/e42a869bfdf2
Part 7-2: Add FetchCompleteRunnable to call OnFetchComplete. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/85d060c1e1f2
Part 8: Compile fetched module. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/3581667122b1
Part 9: Create static import. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/ca1c57e77708
Part 10: Implement WorkletModuleLoader::OnModuleLoadComplete. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/a7f760bb61cb
Part 11-1: Add a virtual method GetResolveFailureMessage in ModuleLoaderBase. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/556b3b7ef319
Part 11-2: Preload localized error msg and format it when resolving failed. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/7b674db6a007
Part 12-1: Using StructuredCloneHolder to pass a JS::Value from a worklet thread to main thread. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/7a9b0e1c02a8
Part 12-2: Add mErrorToRethrow to hold the JS::Value in WorkletFetchHandler. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/4c9f55efa65e
Part 13: Update ini files in wpt/. r=jonco,yulia
https://hg.mozilla.org/integration/autoland/rev/2189a8fa813a
Part 14: Add test case for fetch a child module script failed. r=jonco

Comment 38

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/7890e4826b89
https://hg.mozilla.org/mozilla-central/rev/81fea615d358
https://hg.mozilla.org/mozilla-central/rev/b19342581003
https://hg.mozilla.org/mozilla-central/rev/24c5f2c86eda
https://hg.mozilla.org/mozilla-central/rev/e030ffd2f51c
https://hg.mozilla.org/mozilla-central/rev/27da560927f4
https://hg.mozilla.org/mozilla-central/rev/01b87c41932c
https://hg.mozilla.org/mozilla-central/rev/001fd6c7a59a
https://hg.mozilla.org/mozilla-central/rev/074a822a5c64
https://hg.mozilla.org/mozilla-central/rev/e42a869bfdf2
https://hg.mozilla.org/mozilla-central/rev/85d060c1e1f2
https://hg.mozilla.org/mozilla-central/rev/3581667122b1
https://hg.mozilla.org/mozilla-central/rev/ca1c57e77708
https://hg.mozilla.org/mozilla-central/rev/a7f760bb61cb
https://hg.mozilla.org/mozilla-central/rev/556b3b7ef319
https://hg.mozilla.org/mozilla-central/rev/7b674db6a007
https://hg.mozilla.org/mozilla-central/rev/7a9b0e1c02a8
https://hg.mozilla.org/mozilla-central/rev/4c9f55efa65e
https://hg.mozilla.org/mozilla-central/rev/2189a8fa813a

Comment 39

[Ryan VanderMeulen [:RyanVM]]

Is this something we should add a relnote for?

Comment 40

[Paul Adenot (:padenot)]

Yes please, this improves compatibility for web pages using AudioWorklet and has been long awaited by the Web Audio API community.

Comment 41

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #39)

Is this something we should add a relnote for?

Yeah, but I don't know what's the next step for writing a release note,
setting dev-doc-needed? is enough?

Comment 42

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Release Note Request (optional, but appreciated)
[Why is this notable]: Module scripts can import other module scripts on worklets. Originally the module script provided in Worklet.AddModule() won't be loaded if it has 'static import'.
[Affects Firefox for Android]: Yes
[Suggested wording]: ES modules on worklets.
[Links (documentation, blog post, etc)]: None.

Comment 43

[Ryan VanderMeulen [:RyanVM]]

Added to the Nightly 113 relnotes to ride out to release, thanks.