1923143 - Create an OpenTelemetry C++-based tracing component

Status: RESOLVED FIXED

(Toolkit :: Gecko Trace, task)

Description

[Michael van Straten [:michael]]

Monitoring error failures in Firefox calls for a comprehensive tracing system. Developing a custom error tracer risks inefficiently using developer resources and potentially duplicating established, approved solutions. Additionally, without integrating a comprehensive tracing system, we risk creating redundant systems and hindering the progress of overall observability within Firefox.

Objective

The objective of this bug is to implement an OpenTelemetry-based tracing component within Gecko, providing an API for the instrumentation of other components.

This new component will enable us to address diverse observability needs across various components and thread/process boundaries. Below are some features that Gecko developers may find beneficial:

• Obtaining a view of commonly hard-to-debug error paths from real clients
• Tracing operations across threads and IPC boundaries
• Attaching tracing information to track the state and execution of runnables
• Observing the resolution and callbacks of promises

Why Logging Falls Short: While logging is useful for recording discrete events or errors, it lacks the structured context necessary for tracing complex, multi-threaded operations across various boundaries. Logs provide snapshots rather than the continuous, contextualized view that tracing enables, which is crucial for debugging asynchronous issues and understanding system-wide interactions.

This bug tracks the first milestone of the Error Tracing Proposal for Firefox.

Comment 1

[Michael van Straten [:michael]]

Attachment: Bug 1923143 part 1 - Vendor opentelemety-cpp library

Comment 2

[Michael van Straten [:michael]]

Attachment: Bug 1923143 part 2 - Patch opentelemetry-cpp with mozbuild files

Comment 3

[Michael van Straten [:michael]]

Attachment: Bug 1923143 part 3 - Add a OpenTelemetry C++ based tracing component

Comment 4

[Florian Quèze [:florian]]

Can you explain which problem this is intending to solve, and why what we already have in place (telemetry, logging, profiling) is not usable to solve it?

Comment 5

[Michael van Straten [:michael]]

Sorry for the confusion, this bug is part of an internal project of the LWS DOM team.

Comment 6

[Jan Varga [:janv]]

Please consider using a different bugzilla component for this.

Comment 7

[Michael van Straten [:michael]]

I've just created a request for a new bug component (Bug 1930663).

Comment 8

[Jan Varga [:janv]]

Thanks

Comment 9

[Michael van Straten [:michael]]

Attachment: WIP: Bug 1923143 - Test shared library loading and unloading under memory pressure

Comment 10

[Jeff Muizelaar [:jrmuizel]]

I'm doing something that will give functionality similar to this in bug 1919845

Comment 11

[Michael van Straten [:michael]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #10)

I'm doing something that will give functionality similar to this in bug 1919845

I see that this is scoped a bit broader than bug 1919845, with support for operations outside the profiler sampling context.

If you're interested in collaborating on this, please feel free to reach out. I'd be happy to discuss further.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fb3772f8a616
part 1 - Vendor opentelemety-cpp library r=jari,nika
https://hg.mozilla.org/integration/autoland/rev/45e612b7d4b6
part 2 - Patch opentelemetry-cpp with mozbuild files r=jari,nika
https://hg.mozilla.org/integration/autoland/rev/7510845c07da
part 3 - Add a OpenTelemetry C++ based tracing component r=jari,nika

Comment 13

[amarc]

Backed out for causing hazard bustages

• Backout link
• Push with failures
• Failure Log
• Failure line:

Exception: Process executed with non-0 exit code 1: /builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py --js /builds/worker/workspace/obj-haz-shell/dist/bin/js

Comment 14

[Alex Finder]

(In reply to amarc from comment #13)

Backed out for causing hazard bustages

• Backout link
• Push with failures
• Failure Log
• Failure line:

Exception: Process executed with non-0 exit code 1: /builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py --js /builds/worker/workspace/obj-haz-shell/dist/bin/js

Perfherder has detected a build_metrics performance change from push 20c9e4dea27103afdae102935cd0ed9815151f56.

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
3%	compiler_metrics num_static_constructors	linux64		112.00 -> 109.00

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

You can run these tests on try with ./mach try perf --alert 43126

For more information on performance sheriffing please see our FAQ.

Comment 15

[Michael van Straten [:michael]]

Apologies for the delay.

The issue was caused by a header file that forward-declared a class and then used that forward declaration in an std::shared_ptr. However, the header file that actually defines the class was never included in the build process. As a result, the analysis failed because it could not guarantee that the class had a non-deleted destructor.

This issue has been resolved by removing the forward declaration. To verify the fix, here is a try job:
Try Job Link

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3d3d9f95e477
part 1 - Vendor opentelemety-cpp library r=jari,nika
https://hg.mozilla.org/integration/autoland/rev/02a5f49f0b2f
part 2 - Patch opentelemetry-cpp with mozbuild files r=jari,nika
https://hg.mozilla.org/integration/autoland/rev/1ec7a7034971
part 3 - Add a OpenTelemetry C++ based tracing component r=jari,nika

Comment 17

[amarc]

https://hg.mozilla.org/mozilla-central/rev/3d3d9f95e477
https://hg.mozilla.org/mozilla-central/rev/02a5f49f0b2f
https://hg.mozilla.org/mozilla-central/rev/1ec7a7034971