1341811 - Consider hooking pthread_create (or another technique) so that we can profile threads launched by libraries

Status: NEW

(Core :: Gecko Profiler, enhancement, P3)

Description

[Markus Stange [:mstange]]

Threads launched by media libraries or WebRTC don't go through NS_NewNamedThread, so we don't automatically register them to the profiler.

We should investigate whether we can hook pthread_create (and the Windows equivalent) in order to automatically register all threads with the profiler.

Comment 1

[Markus Stange [:mstange]]

Glandium had some ideas about this in Austin.

Comment 2

[Mike Hommey [:glandium]]

Oh, my, this has been ni? for way too long. Sorry about that.

Windows is, interestingly, the easiest to handle. You just need a DllMain in one of our dlls (xul.dll), and have it handle the DLL_THREAD_ATTACH event. We have that in toolkit/library/nsDllMain.cpp.

On linux, we can export a pthread_create wrapper from our executable, but that won't catch all thread creations. Anything using the system call directly (and there may be) won't be caught. I guess that's still better than nothing?

On mac, we can wrap pthread_create, but that won't catch any threads created by system libraries, and there are. OTOH, OSX has APIs to enumerate all threads: http://web.mit.edu/darwin/src/modules/xnu/osfmk/man/task_threads.html

Comment 3

[Mike Hommey [:glandium]]

Also note threads can be enumerated on linux by reading the directory entries in /proc/self/task. Arguably, on OSes where we can enumerate threads, we don't need to have them registered to the profiler.

Comment 4

[David Teller [:Yoric] - still alive but not very active]

For what it's worth:

• unnamed threads are now visible in about:processes;
• we can easily test that all threads are OS-level named in the unit tests for about:processes.

Unfortunately, as of today, there are still a few unnamed threads. I'm tracking them as blockers of bug 1630898.

Comment 5

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Thank you David.

Obviously about:processes must have a way to get the list of threads (ids and names), I hope the profiler could use the same technique or even the same code, in which case it should be possible to record them.

I think this bug here should depend on bug 1630898 instead of blocking it (i.e., the profiler depends on threads being named before it can profile them) -- But please let me know if you disagree.

Comment 6

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Oh, while it's great if we can get a list of threads, I'm guessing it may not be cheap to do that, so the profiler may not be able to do it at every sample point!
If my assumption is sadly correct we will still want to try and catch thread creations (by hooking pthread_create or other means).

But in the meantime, it could still be valuable to have an option to regularly (1 Hz?) look at the list of threads, and profile new ones at they appear.

And on the flip side, we'll want to catch thread terminations; but that should (hopefully) be trivial as they'll just not be there anymore when we try to sample them. (Another assumption: Thread ids won't be reused in-between two samplings.)

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

Yoric (or whomever): where are threads in about:processes? Also, we're talking raw threads/pthreads, not nsThreads which we can already enumerate.

Comment 8

[Florian Quèze [:florian]]

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #7)

Yoric (or whomever): where are threads in about:processes?

They are shown if you set toolkit.aboutProcesses.showThreads to true in about:config.

Comment 9

[David Teller [:Yoric] - still alive but not very active]

They are hidden by default. You need pref toolkit.aboutProcesses.showThreads = true to show them.

Comment 10

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

I've just filed bug 1742522, which will discover unregistered threads the same way about:processes does.
It should help with profiling library-controlled threads, but not as soon as threads are started, so I will keep this bug here open in case we can actually hook pthread_create (or equivalent/better, and equivalent on other platforms).

Comment 11

[David Teller [:Yoric] - still alive but not very active]

Note that the thread discovery in about:processes is pretty slow.

Comment 12

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

(In reply to David Teller [:Yoric] - still alive but not very active from comment #11)

Note that the thread discovery in about:processes is pretty slow.

True, it's not be perfect but hopefully it will give some useful data on all platforms, until someone can get this bug here done!

Comment 13

[Markus Stange [:mstange]]

The thread discovery implemented in bug 1742522 takes 4ms per check on my Samsung Galaxy S21 so this is not something we can realistically do on a per-sample basis.

(In reply to Mike Hommey [:glandium] from comment #2)

On linux, we can export a pthread_create wrapper from our executable, but
that won't catch all thread creations. Anything using the system call
directly (and there may be) won't be caught. I guess that's still better
than nothing?

It should be quite a bit better than nothing - do you know if this would catch Rust threads? Also, would this work on Android?
Also, we may want to hook pthread_setname_np as well, so that if we have a thread name filter, we can start sampling a thread immediately when it gets renamed to a name that matches the filter.

Comment 14

[Markus Stange [:mstange]]

Oh, also, do we have other executable-exported wrapper functions that I can look at for inspiration?

Comment 15

[Mike Hommey [:glandium]]

We're actually hooking pthread_create already, now, since bug 1678152. Bug 1752703 moved it out of the crash reporter and into mozglue and works to some extent on Android (it doesn't catch system libraries calling pthread_create).
It should be catching Rust threads, but it's android/linux only at the moment.