1930954 - Implement Collection and Processing Mechanism for Traces Captured by Gecko Trace

Status: RESOLVED FIXED

(Toolkit :: Gecko Trace, task, P2)

Description

[Michael van Straten [:michael]]

Gecko Trace must be instrumented to collect useful traces in a central location. Traces need to be aggregated, meaning similar signatures will be merged to reduce the volume of data.

At specified intervals, the aggregated traces should be handed over to Glean for submission to Telemetry.

On the backend, traces from individual clients must be further merged, with client identifiers removed and the data further reduced. On a scheduled basis, all aggregates should be reported to their respective meta bugs on Bugzilla.

This bug tracks the second milestone of the Error Tracing Proposal for Firefox.

Comment 1

[Michael van Straten [:michael]]

Attachment: Bug 1930954 - Part 1: Set up metrics/pings for Glean telemetry integration r=jari

Comment 2

[Michael van Straten [:michael]]

Attachment: Bug 1930954 - Part 3: Vendor opentelemetry-proto definitions r=jari

Comment 3

[Michael van Straten [:michael]]

Attachment: Bug 1930954 - Part 4.1: `moz.yaml` setup for toolkit/components/protobuf component r=tjr,nika

Comment 4

[Michael van Straten [:michael]]

Attachment: Bug 1930954 - Part 7: Add telementry trace aggregation r=nika

Comment 5

[Michael van Straten [:michael]]

Attachment: Bug 1930954 - Part 2: Add gecko-trace linter for schema validation r=#linter-reviewers

Comment 6

[Michael van Straten [:michael]]

Attachment: Bug 1930954 - Part 5: Vendor OtlpRecordable from OTel upstream r=jari

This patch vendors the OtlpRecordable and associated utilities for storing
span data within a protobuf-encodable container. This enables us to use this
recordable in subsequent patches to export traces to the parent process,
leveraging the protobuf files vendored in the previous patches.

Comment 7

[Michael van Straten [:michael]]

Attachment: Bug 1930954 - Part 6: Add pref to enable tracing system at runtime r=jari

This patch adds an off-by-default static-pref to enable the gecko-trace
tracing system at runtime.

Comment 8

[Michael van Straten [:michael]]

Attachment: Bug 1930954 - Part 4.2: Integrate opentelemetry-proto with protobuf component UPGRADE_NSS_RELEASE r=nika

This patch runs the added/updated generation scripts in the
toolkit/components/protobuf component.

It also integrates the generated files into the mozbuild system for
later use.

Comment 9

[Michael van Straten [:michael]]

Attachment: WIP: Bug 1930954 - Add Glean glue code for gecko-trace

Comment 10

[Michael van Straten [:michael]]

Attachment: WIP: Bug 1930954 - Part 9: Test new features added in this stack r=jari

Comment 11

[Michael van Straten [:michael]]

Attachment: Bug 1930954 - Part 8: Add support for array attribute types r=jari

This patch adds support for array attribute types in gecko-trace.ymal
files. It was previously forgotten.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/5db40ed7fb52
https://hg.mozilla.org/integration/autoland/rev/a5d28bed3401
Part 1: Set up metrics/pings for Glean telemetry integration r=jari
https://github.com/mozilla-firefox/firefox/commit/78c0656ffad5
https://hg.mozilla.org/integration/autoland/rev/757e536bd695
Part 2: Add gecko-trace linter for schema validation r=linter-reviewers,ahal
https://github.com/mozilla-firefox/firefox/commit/1a48ef555569
https://hg.mozilla.org/integration/autoland/rev/baa6425bf629
Part 3: Vendor opentelemetry-proto definitions r=jari
https://github.com/mozilla-firefox/firefox/commit/ba2cec89e40b
https://hg.mozilla.org/integration/autoland/rev/d5234bd38999
Part 4.1: `moz.yaml` setup for toolkit/components/protobuf component r=tjr
https://github.com/mozilla-firefox/firefox/commit/d9251f91ada4
https://hg.mozilla.org/integration/autoland/rev/878a48c023ee
Part 4.2: Integrate opentelemetry-proto with protobuf component UPGRADE_NSS_RELEASE r=nika,tjr,bbeurdouche
https://github.com/mozilla-firefox/firefox/commit/1d752429e3c6
https://hg.mozilla.org/integration/autoland/rev/6bc80eec034f
Part 5: Vendor OtlpRecordable from OTel upstream r=jari
https://github.com/mozilla-firefox/firefox/commit/fd208fe761d7
https://hg.mozilla.org/integration/autoland/rev/5d498bf86c54
Part 6: Add pref to enable tracing system at runtime r=jari

Comment 13

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/a5d28bed3401
https://hg.mozilla.org/mozilla-central/rev/757e536bd695
https://hg.mozilla.org/mozilla-central/rev/baa6425bf629
https://hg.mozilla.org/mozilla-central/rev/d5234bd38999
https://hg.mozilla.org/mozilla-central/rev/878a48c023ee
https://hg.mozilla.org/mozilla-central/rev/6bc80eec034f
https://hg.mozilla.org/mozilla-central/rev/5d498bf86c54

Comment 14

[Michael van Straten [:michael]]

The top part of the patch stack is still missing.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/83bbfd922880
https://hg.mozilla.org/integration/autoland/rev/27fa35b27921
Part 7: Add telementry trace aggregation r=nika

Comment 16

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/27fa35b27921

Comment 18

[Phabricator Automation]

Comment on attachment 9510704 [details]
Bug 1930954 - Part 8: Add support for array attribute types r=jari

Revision D263303 was moved to bug 1984029. Setting attachment 9510704 [details] to obsolete.

Comment 19

[Phabricator Automation]

Comment on attachment 9508124 [details]
WIP: Bug 1930954 - Part 9: Test new features added in this stack r=jari

Revision D261726 was moved to bug 1984092. Setting attachment 9508124 [details] to obsolete.