1152856 - [meta] DevTools support for shared memory for workers

Status: NEW

(DevTools :: General, defect, P3)

Description

[Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8]]

Not sure what the tooling should look like for shared memory between workers, but it sure seems like the kind of thing devs would _really_ appreciate when hacking on this stuff.

Comment 1

[Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8]]

ni some folks for ideas

Comment 2

[Luke Wagner [:luke]]

Coincidentally, Jukka was just making a set of requests yesterday.

Comment 3

[Jim Blandy :jimb]

By swapping in instrumented versions of the shared arrays, we should be able to do all kinds of nice tracking. "Who wrote this byte last?" It might be expensive, but if you need to know, you really need to know.

Comment 4

[Lars T Hansen [:lth]]

Worker debugging.  (I know, this is on its way.)

Deadlock / wait detection - provide a view to workers that are waiting on locations on arrays, and (once we have the "synchronic" APIs), what condition they're waiting for.  Deadlock detection proper is a little tricky perhaps but waiting-for-a-long-time may be an OK proxy.

Performance - which locations are hot for waiting, how long do waiters wait there, etc.

Performance - can we tie this in with CPU event counters and report memory traffic on shared memory location.

(I started a mail thread about this a while ago and got some sensible answers, I'll try to summarize here when I have a moment.)

Comment 5

[Jukka Jylänki]

Lots of ideas:

- When we abort execution on slow script dialog, print out call stacks for all workers where they were executing at, not just the main thread. (not sure if this was done yet?)

- Have the ability to somehow (debug button?) abort the execution of all workers, i.e. it might be that the main thread is not hung and is not showing the slow script dialog, but one or more workers have hung. Note that we are looking forward to executing infinite loops (blocking game main loop) in workers, so in some scenarios a worker that never yields is intended behavior.

- Each web worker currently has a single parent, and can only communicate with that. This creates a tree hierarchy for workers. Have the ability to inspect what this tree hierarchy is, and which workers are idle and not executing any code, and which are. Are any workers waiting for a futex, and which address, if so.

- Since futexes are raw addresses, it is difficult to understand looking at a UI what a "worker (http://server.com/page.js) is waiting for a futex at address 0x00042315" means. It would be great to have a JS API like "debugger.setWorkerName(worker, 'myPhysicsProcessingThread');" and "debugger.setFutexName(mySharedArrayBuffer, 0x00042315, 'physicsTaskWait');" that would allow JS code to cooperatively annotate workers and addresses with what these mean.

- Ability to abort/kill individual workers.

- Ability to pause and resume individual worker execution.

- Ability to see memory and cpu usage of individual workers.

- Cpu profiling to workers - the Performance view that shows CPU profiles for the main thread would be great for this!
   - Would be great to be simultaneously able to profile all workers, and i.e. not just a single chosen worker at a time.

- Ability to profile how contended different futex addresses have been: how long a thread has waited for futexes overall, how long did it wait on an individual futex?

- Break and step through code running in workers. See what script code is loaded in workers.

- Ability to break upon the creation of a new worker in the first line of the worker (in order to be able to set up new breakpoints and debug the "main" startup of the script)

- Ability to execute custom JS code live in the context of a specific worker via a mechanism similar to the page console interpreter.

- Ability to trace/profile the postMessage() communication that occurred between workers - I recently had a bug where I accidentally postMessage()d hundreds of identical messages to a worker when I only intended to send one message, and everything worked fine, except that there was redundant work being done when the worker did the same reply again and again (replacing the previous), a scenario which I caught much later by accident.

- Ability to craft and send custom postMessage()s to workers from the debugger. (this can be done from workers that the main thread owns by retaining a global var to the worker, but not from workers that workers own?)

- Ability to profile memory allocations that occurred in a worker. (just like the excellent devtool we now have for the main thread!)

Comment 6

[Eddy Bruel [:ejpbruel]]

Break on shared memory access (read/write) seems like it would be really useful.

Other than that, I don't have much to add to what Jukka already wrote in comment #5.

Comment 7

[Kannan Vijayan [:djvj]]

(In reply to Nick Fitzgerald [:fitzgen][:nf] from comment #1)
> ni some folks for ideas

Jukka covered a whole lot of the important stuff.  The only thing I can think of that really expands on that is chained breakpoints.

Sometimes when working with multithreaded programs, you want to break on some line of code, but only in the context of some other event happening.  I.e. you want to know where thread A hits line X, but only after thread B doe something that you expect A to respond to.

For example, you have a bug that manifests in the main thread after a worker thread handles a particular kind of request, does some work, and messages the main thread.  You want to break on the main thread, but only after that particular kind of request has been handled on the worker.

This can potentially be implemented through a "pseudo" breakpoint notion.  You set a breakpoint that doesn't actually break, but instead triggers the enabling or disabling of another breakpoint somewhere else.

Comment 8

[Eddy Bruel [:ejpbruel]]

This feature will eventually become important, but for now the timetable is still unclear, and in any case we have more pressing issues such as service worker debugging. I'm going to tentatively mark this as P3. We can up the priority once this becomes more important.

Comment 9

[Lars T Hansen [:lth]]

I can afford to be opinionated since I won't be doing the work :) but IMO basic profiling is a must-have -- computational workers will be used largely for performance reasons, without profiling programmers will be in the dark.  It's a non-option to move the code to the main page to get profiling results.

Comment 10

[Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8]]

(In reply to Lars T Hansen [:lth] from comment #9)
> I can afford to be opinionated since I won't be doing the work :) but IMO
> basic profiling is a must-have -- computational workers will be used largely
> for performance reasons, without profiling programmers will be in the dark. 
> It's a non-option to move the code to the main page to get profiling results.

I think most of the devtools team either (a) doesn't have the expertise, or (b) doesn't have the free cycles to expose APIs for profiling shared memory workers.

My understanding is that exposing the existing profiling machinery we have (SPS profiler, and tracing instrumentation) for worker threads in the profiler's UI is happening in bug 1240836 (or maybe bug 1132525?). That will not have any shared memory specific features, however; just our existing profiling infrastructure.

If you want to help expose shared memory specific features and tooling (eg futex contention), the best way to get there would be to help us out by exposing APIs that tools could build on top of. It is unlikely to happen anytime soon otherwise. If you want to volunteer to build APIs, then we should probably sit down and talk about the design of the APIs, what minimal requirements for shipping might be, etc.

Comment 11

[Lars T Hansen [:lth]]

Fair point, though I really did intend to ask only for straightforward JS profiling within the workers, not contention etc.  I was hoping that a proxy for contention might be simply the time spent in futexWait, for example.  If that's already happening then I think that's great.

I can give some thought to APIs that might expose interesting behaviors.  wait/wake behavior is probably most important in that regard.  Do you have a place to point to where similar introspection APIs are defined, that I can use as a model?

Comment 12

[Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8]]

(In reply to Lars T Hansen [:lth] from comment #11)
> Fair point, though I really did intend to ask only for straightforward JS
> profiling within the workers, not contention etc.  I was hoping that a proxy
> for contention might be simply the time spent in futexWait, for example.  If
> that's already happening then I think that's great.

Is futexWait self hosted or a JSNative? If it is self-hosted then it should show up automatically via the usual stack capturing. If it is a JSNative, then we will need some extra instrumentation which should be really easy to add with AutoSPSEntry:

https://dxr.mozilla.org/mozilla-central/source/js/src/vm/SPSProfiler.h?from=AutoSPSEntry#283
https://dxr.mozilla.org/mozilla-central/source/js/src/jsarray.cpp#1314

> I can give some thought to APIs that might expose interesting behaviors. 
> wait/wake behavior is probably most important in that regard.  Do you have a
> place to point to where similar introspection APIs are defined, that I can
> use as a model?

If you want to add "fake" frames to the JS stacks captured by the sampling profiler, use `AutoSPSEntry` (linked above).

If you want to precisely trace an individual operation to expose it in the waterfall graph, then we have the `mozilla::TimelineMarker` infrastructure. Unfortunately, this lives in docshell/ and so adding tracing to JS operations involves defining new JSAPI methods to set callbacks. There is some talk about moving the markers into js/ and making them part of the Debugger API so that we won't have this problem (and others) anymore.

If it isn't adding frames to stacks in the profiler, or exact tracing of an operation for the waterfall graph, then usually it is exposed via the Debugger API: js/src/vm/Debugger.{h,cpp} and js/src/vm/DebuggerMemory.{h,cpp}.

Let me know if you have questions about any of these bits, or if there is anything I can help clarify!

Comment 13

[Jim Blandy :jimb]

(In reply to Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8] from comment #12)
> If it isn't adding frames to stacks in the profiler, or exact tracing of an
> operation for the waterfall graph, then usually it is exposed via the
> Debugger API: js/src/vm/Debugger.{h,cpp} and
> js/src/vm/DebuggerMemory.{h,cpp}.

... and documented in js/src/doc/Debugger (Debugger-API.md in that directory is the front page).