Implement new C++ way to define marker types
Categories
(Core :: Gecko Profiler, task, P2)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox82 | --- | fixed |
People
(Reporter: mozbugz, Assigned: mozbugz)
References
(Depends on 2 open bugs, Blocks 1 open bug)
Details
Attachments
(20 files, 1 obsolete file)
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review |
We want to make it easier to define new types of markers from the C++ side.
(The front-end dependency will be handled by other bugs, in particular the formatting schema, see bug 1640999.)
Currently developers need to create a derived class from ProfilerMarkerPayload, and define functions to:
- Construct a payload.
- Compute the serialization size.
- Serialize to the profile buffer, including taking care to serialize common properties in the base class.
- Deserialize from the profile buffer, including reconstructing the payload.
- Stream to JSON.
With only a developer-provided stream-to-json function, I believe it is possible to automatically handle everything else.
For more information and a proof-of-concept, see the "C++ Gecko code" section in https://docs.google.com/document/d/1CCvTL2O_9YF7sd6ef8pWvm23EVgArOsmoNG0zqXFk9w/edit#heading=h.6nbh7k46bsm1
| Assignee | ||
Comment 1•5 years ago
|
||
One thing to consider is a simpler option to capture backtraces when adding a marker. In most cases the backtrace is done on the spot, in which case it should be possible to store its data on the stack and avoid an allocation for that (see bug 1578792).
Also, it should be possible to avoid actually constructing a marker structure on the stack; This would effectively "fix" bug 1636802.
| Assignee | ||
Comment 2•5 years ago
|
||
This patch introduces all new files that contain the new markers C++ API and implementation.
They are mostly empty at this time, only including each other as eventually needed, and with #ifdef MOZ_GECKO_PROFILER guards.
Rough inclusion diagram: (header <-- includer)
BaseProfilerMarkerPrerequesites.h <-- ProfilerMarkerPrerequesites.h (Useful types: Input string view, marker options)
^ ^
BaseProfilerMarkerDetail.h <-- ProfilerMarkerDetail.h (Implementation details)
^ ^
BaseProfilerMarkers.h <-- ProfilerMarkers.h (Main API)
^ ^--------- ^ ^---------
BaseProfilerMarkerTypes.h | <-- ProfilerMarkerTypes.h | (Common marker types)
^ ^-- BaseProfiler.h <-- GeckoProfiler.h (Existing main profiler headers)
Depends on D87240
| Assignee | ||
Comment 3•5 years ago
|
||
These are similar to std::string_view, but they are optimized to be serialized, and later deserialized and streamed to JSON.
They accept literal strings, and keep them as unowned raw pointers and sizes.
They also accept any substring reference, assuming that they will only be used as function parameters, and therefore the dependent string will live during the function call where these StringView's are used.
Internally, they also allow optional string ownership, which is only used during deserialization and streaming.
This is hidden, so that users are not tempted to use potentially expensive string allocations during profiling; it's only used after profiling, so it's less of an impact to allocate strings then. (But it could still be optimized later on, as part of bug 1577656.)
Depends on D87241
| Assignee | ||
Comment 4•5 years ago
|
||
This marker option captures a given thread id.
If left unspecified (by default construction) during the add-marker call, the current thread id will be used then.
Depends on D87242
| Assignee | ||
Comment 5•5 years ago
|
||
This moves the existing MarkerTiming class introduced in bug 1640969 to the BaseProfilerMarkersPrerequesites.h header, and can be used as a marker option.
Some minor clarifying changes:
Instant()is split into two functions:InstantNow()andInstantAt(TimeStamp).Interval(TimeStamp, TimeStamp)must be given both start and end, otherwiseIntervalUntilNowFrom(TimeStamp)takes the start only and ends "now".IsUnspecified()indicates ifMarkerTimingwas default-constructed with no timestamps at all; In this case the add-marker code will replace it withInstantNow().
The serialization only contains the phase, and one or two timestamps as needed, to save space for non-interval timings.
Depends on D87244
| Assignee | ||
Comment 6•5 years ago
|
||
IsEmpty() makes it easier for users to determine if there is anything stored in the ProfileChunkedBuffer.
And ProfileChunkedBuffer can now be serialized from a raw pointer, which will be useful when adding markers with an optional stack.
Deserialization should be done to a UniquePtr<ProfileChunkedBuffer>, which is already implemented.
Depends on D87245
| Assignee | ||
Comment 7•5 years ago
|
||
This marker option allows three cases:
- By default, no stacks are captured.
- The caller can request a stack capture, and the add-marker code will take care of it in the most efficient way.
- The caller can still provide an existing backtrace, for cases where a marker reports something that happened elsewhere.
Depends on D87246
| Assignee | ||
Comment 8•5 years ago
|
||
This option can take an inner window id.
Depends on D87247
| Assignee | ||
Comment 9•5 years ago
|
||
A MarkerOptions must be constructed with a known sub-category from ProfilingCategoryList.h, e.g.: MarkerOptions::OTHER(), this will be required for all markers.
MarkerOptions also contains defaulted MarkerThreadId, MarkerTiming, MarkerStack, and MarkerInnerWindowId. They may be changed by chaining Set() functions, e.g.:
PROFILER_MARKER<...>("name", MarkerOptions::Other().Set(MarkerThreadId(otherThread)).Set(MarkerStack::Capture()), ...)
Depends on D87248
| Assignee | ||
Comment 10•5 years ago
|
||
This is similar to the existing deserializer table in ProfilerMarkerPayload.*:
Each type of marker (except in the NoPayload one) has its corresponding deserializer in a table, and the index is used as a tag in the serialization buffer.
Depends on D87249
| Assignee | ||
Comment 11•5 years ago
|
||
Marker types will be defined with a simple struct that contains (amongst other things) one Stream(JSONWriter&, ...) function.
This patch introduces a type-traits-like helper to examine that function. This will be used to check the arguments given to the upcoming add-marker function, to serialize and deserialize them.
Depends on D87250
| Assignee | ||
Comment 12•5 years ago
|
||
NoPayload will be mostly used internally when adding markers without payload data.
It has an empty specialization of the MarkerTypeHelper (mainly to catch misuses), and the add-marker code will need to have different compile-time paths to handle it.
Depends on D87251
| Assignee | ||
Comment 13•5 years ago
|
||
A new entry kind is needed to serialize the new markers, because they are not encoded the same way.
Depends on D87252
| Assignee | ||
Comment 14•5 years ago
|
||
DeserializeAfterKindAndStream() is the main function that extracts all the marker data (past the already-read entry kind), and streams it to JSON using the user-provided Stream(JSONWriter&, ...) function in the appropriate marker type definition.
It currently requires two external functions to stream the name and the optional backtrace, because these are different between the two profilers. This may change in the future.
(Deserialization is implemented before serialization, because the Deserialize() function is needed during serialization to get a marker type tag.)
Depends on D87253
| Assignee | ||
Comment 15•5 years ago
|
||
Main marker serialization function, which accepts the same arguments as the marker type's Stream(JSONWriter&, ...) function, and stores them after the marker name and options.
Depends on D87254
| Assignee | ||
Comment 16•5 years ago
|
||
The upcoming profiler-specific add-marker function will need to know which ProfileChunkedBuffer to serialize to, these "detail" functions provide access.
Depends on D87255
| Assignee | ||
Comment 17•5 years ago
|
||
This is the main public marker API:
AddMarkerToBuffercan be used to store a marker to any buffer. This could be useful to code that wants to store markers outside of the default profiler buffers.AddMarker/profiler_add_markerstore a marker to the appropriate profiler buffer.- BASE_PROFILER_MARKER and PROFILER_MARKER do the same, but are also available (and empty) when MOZ_GECKO_PROFILER is not #defined.
All these take a name, marker options, a marker type, and the type's expected arguments if any.
Extra helpers for the most common types:
- BASE_PROFILER_MARKER_UNTYPED and PROFILER_MARKER_UNTYPED store a marker with no data payload.
- BASE_PROFILER_MARKER_TEXT and PROFILER_MARKER_TEXT store a text marker.
baseprofiler::markers::Textshows how new marker types can be defined.
Depends on D87256
| Assignee | ||
Comment 18•5 years ago
|
||
RAII class that may be used to capture a time interval in a marker.
To stay efficient, it requires some awkward lambda work in the call.
Users should instead store a start timestamp and later add a marker with an interval from that start timestamp; but in some cases (e.g., multiple return paths) this Auto class may be more appropriate.
Depends on D87257
| Assignee | ||
Comment 19•5 years ago
|
||
This ports existing ProfilerMarkerPayload types to the new API. This is just a starting point, they may be changed later on as needed.
Depends on D87258
| Assignee | ||
Comment 20•5 years ago
|
||
Depends on D87259
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
| Assignee | ||
Comment 21•5 years ago
|
||
The main, and only compulsory, marker option is the MarkerCategory, which captures the "category pair", as well as the parent category.
Depends on D87256
| Assignee | ||
Comment 22•5 years ago
|
||
profiler_capture_backtrace(ProfileChunkedBuffer&) renamed to profiler_capture_backtrace_into(ProfileChunkedBuffer&) (notice "_into"), which is clearer.
New function profiler_capture_backtrace() returns a UniquePtr<ProfileChunkedBuffer>, which can later be given to MarkerStack.
profiler_get_backtrace() (returning a UniqueProfilerBacktrace) now uses profiler_capture_backtrace().
This reduces most duplicate code between these functions.
Depends on D87246
Comment 23•5 years ago
|
||
Comment 24•5 years ago
|
||
Backed out for bustages on TestBaseProfiler.cpp
Backout link: https://hg.mozilla.org/integration/autoland/rev/bb7ec19eaf346a09e1e4a02abd15f74d6ea1f253
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=314529145&repo=autoland&lineNumber=64541
Comment 25•5 years ago
|
||
| Assignee | ||
Updated•5 years ago
|
Comment 26•5 years ago
|
||
Backed out 20 changesets (bug 1646266) for build bustages at TestBaseProfiler.cpp.
https://hg.mozilla.org/integration/autoland/rev/9a8d66b99960ecd148e891029d4f183424dbc696
Failure log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=314536441&repo=autoland&lineNumber=30706
| Assignee | ||
Comment 27•5 years ago
|
||
Thank you Narcis and Cristian for the backouts.
Please note that I did test on Try first, but didn't see these errors there. I filed bug 1662328 about the differing build results.
I'm working on fixing these errors here again, I'll avoid literal string pointers this time...
Comment 28•5 years ago
|
||
Comment 29•5 years ago
|
||
Backed out 20 changesets (bug 1646266) for build bustage at baseprofiler/core/ProfilerMarkers.cpp
Backout: https://hg.mozilla.org/integration/autoland/rev/3d00542b5bd1dfeeff35e9d31927d512fd21e044
Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=a2734d73264c7208c6e1720f21fbdc95bfe1b28d
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=314545088&repo=autoland&lineNumber=14
Comment 30•5 years ago
|
||
Comment 31•5 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/a5fffd9f8649
https://hg.mozilla.org/mozilla-central/rev/b511ca601b74
https://hg.mozilla.org/mozilla-central/rev/6af847204dc5
https://hg.mozilla.org/mozilla-central/rev/05dd7a1c5279
https://hg.mozilla.org/mozilla-central/rev/baaa8d69cf5d
https://hg.mozilla.org/mozilla-central/rev/19230cc586cf
https://hg.mozilla.org/mozilla-central/rev/764280d95b6f
https://hg.mozilla.org/mozilla-central/rev/611e97a5af26
https://hg.mozilla.org/mozilla-central/rev/ececccb1c743
https://hg.mozilla.org/mozilla-central/rev/89cdbaea408a
https://hg.mozilla.org/mozilla-central/rev/476acea9fa53
https://hg.mozilla.org/mozilla-central/rev/8f8d2845d089
https://hg.mozilla.org/mozilla-central/rev/9ee79e7bd226
https://hg.mozilla.org/mozilla-central/rev/8581a81b51fa
https://hg.mozilla.org/mozilla-central/rev/ac8060521a4f
https://hg.mozilla.org/mozilla-central/rev/75543e56e2e8
https://hg.mozilla.org/mozilla-central/rev/300dcabd8b53
https://hg.mozilla.org/mozilla-central/rev/9a2249f099bb
https://hg.mozilla.org/mozilla-central/rev/17bc1530415f
https://hg.mozilla.org/mozilla-central/rev/44ed92e28c34
| Assignee | ||
Updated•5 years ago
|
Description
•