1646165 - Multi-language architecture Implementation

Status: RESOLVED FIXED

(Toolkit :: Telemetry, task, P1)

Description

[Chris H-C :chutten]

Following the design eked out in bug 1618905, build out the multi-language architecture. This includes

• Auto-generating the Metric ID tables using the glean_parser (work may have started on this in bug 1635260 for Rust, but by the end of this bug metric ids must be useful in JS, C++, and Rust across processes)
• Establishing how best to build out the MLA FFI layer. Is it similar enough to mozilla/glean's ffi that we should adopt its approach? tools? code?
• Tests
• Documentation (update https://firefox-source-docs.mozilla.org/toolkit/components/glean/code_organization.html as necessary)

This does not include:

• Writing or generating the C++ or JS API except as needed for prototyping and testing

Comment 1

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Generate Metric ID -> Metric instance map in Rust. r?chutten!

Metric IDs are just a monotonically increasing number assigned to each metric.
This is stable for a single compilation and should thus be enough to
uniquely identify a metric.

We create one hash map per metric type implementation in the generated
Rust code as global statics, so we can later look up metrics.
These maps are not modified at runtime.
We map from the metric ID to a reference of lazy value of the global metric.
This means we don't instantiate the metric until accessed.
We also ensure to lazily instantiate the map itself to avoid the
allocation at start, it happens at first access of the metric type.
This makes access time non-deterministic, but for now this might be ok.
We can come up with a more static layout when needed.

Comment 2

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Generate C++ code for Glean metrics based on IDs. r?chutten!

Based on the same Metric ID generation we now generate C++ code.
As tested in the C++ API design document these are constexpr of the
metric type implementations, wrapping the metric ID.
Right now everything is generated into a single big GleanMetrics.h header file.

Depends on D89722

Comment 3

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Implement 3 metric types in C++, deferring to Rust code. r?chutten!

Implementation of metric types in C++ is reasonably simple, as it defers
only to the FFI counterparts in Rust.

Right now this is a most simplistic approach to try it out:

• It's all in one header file instead of split up into one file per type
• It lacks docs
• It lacks the test API

We also manually implement the FFI API.
We should be able to leverage a macro to reduce the boilerplate we have
to actually type, both for the C++ implementation as well as the Rust FFI part.
This will probably land in a follow-up.

Depends on D89723

Comment 4

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - [dev only] Record FOG metric in C++ code. r?chutten!

Just for testing purposes. This code will go away before landing this
patch set.

Depends on D89724

Comment 5

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Prepare to generate Metric Name -> Metric ID for JavaScript. r?chutten!

NOTE: This is not implemented yet. Included to express my thoughts
before I get to it.

For the JavaScript part we need to become a bit smarter.
As we rely on named getters in WebIDL we need string -> ID maps.

1. We need to be able to distinguish between valid and invalid categories (to allow for the API: glean.category.metric.set(3)).
   That requires some map.
2. We then need to map the full name (category.name) to the metric ID
   we generated earlier.

PHF (Perfect Hash Table) as available in the build system and to C++
code can take most of this work off our hands. Firefox Telemetry already uses it.

Optimization note:
Can we drop the map?
If we generate perfect hashes, we can use that hash as the Metric ID instead.
A lookup will then require to run the hash function (and a check if the hash doesn't exist), but no double indirection (name -> hash -> metric id).

Depends on D89725

Comment 6

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Generate Metric ID -> Metric instance map in Rust. r?chutten!

Metric IDs are just a monotonically increasing number assigned to each metric.
This is stable for a single compilation and should thus be enough to
uniquely identify a metric.

We create one hash map per metric type implementation in the generated
Rust code as global statics, so we can later look up metrics.
These maps are not modified at runtime.
We map from the metric ID to a reference of lazy value of the global metric.
This means we don't instantiate the metric until accessed.
We also ensure to lazily instantiate the map itself to avoid the
allocation at start, it happens at first access of the metric type.
This makes access time non-deterministic, but for now this might be ok.
We can come up with a more static layout when needed.

Comment 7

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Implement FOG's JavaScript API via WebIDL. r?chutten!

This is only the basic outline.
It doesn't do anything yet, but compile.
As there are no metrics generated for it it can't look up anything.

To note: Actual metric types are implemented in XPIDL later.

The following (priviliged) JavaScript code will soon work (if the
corresponding metrics would be defined):

const { Glean } = ChromeUtils.import("resource://gre/modules/Glean.jsm");
Glean.shared.test_only.count_things.add(1);

Depends on D92210

Comment 8

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Generate C++ code for Glean metrics based on IDs. r?chutten!

Based on the same Metric ID generation we now generate C++ code.
As tested in the C++ API design document these are constexpr of the
metric type implementations, wrapping the metric ID.
Right now everything is generated into a single big GleanMetrics.h header file.

Depends on D92211

Comment 9

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Implement code generation for the JavaScript API. r?chutten!

This is a bit more complex because of the multiple layers we have.
For JavaScript we need to know:

1. Does an arbitrary string identify a valid category?
2. If yes, does another arbitrary string identify a valid metric inside that checked category?
3. If yes, Which type does that metric have?
4. If you got that, give me a type that has the right methods and
   forwards them to the right metric instance.

So we create 2 lookup tables and encode additional information into the
entries.

1. lookup table:
   Category name -> Index,
   where the index points into a string table, so we can ensure it's correct.

2. lookup table:
   Metric identifier (category name . metric name) -> Entry,
   where Entry is a 64-bit integer.
   This integer encodes:

3. An index into the string table to check for string equality with a search key

4. Type information to instantiate the correct C++ class

5. The metric's actual ID to lookup the underlying instance.

With the 64 bits available we dedicate:

1. 32 bit to the string table offset. More than enough for a large string table.
2. 5 bit for the type. That allows for 32 metric types. We're not even close to that yet.
3. 27 bit for the metric ID. That allows for 130 million metrics. Let's not go there.

Depends on D92212

Comment 10

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Implement the XPIDL and C++ parts of a counter metric. r?chutten!,brizental!

This is the very first metric to gain support across all 3 languages (C++, JavaScript, Rust).
There's no user yet, so we can't properly test it (a C++ test will come later).

As its the first it also brings the XPIDL definition and some
nice-to-haves with it (like a header file including all other metric type header files).

Depends on D92213

Comment 11

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Implement the XPIDL and C++ parts of a timespan metric. r?chutten!

Depends on D92214

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1b5253d54219
Generate Metric ID -> Metric instance map in Rust. r=chutten
https://hg.mozilla.org/integration/autoland/rev/a72fe6375f80
Implement FOG's JavaScript API via WebIDL. r=chutten,webidl,emilio,smaug
https://hg.mozilla.org/integration/autoland/rev/2b75c70b1f2d
Generate C++ code for Glean metrics based on IDs. r=chutten
https://hg.mozilla.org/integration/autoland/rev/af5e9b06ff01
Implement code generation for the JavaScript API.  r=chutten,emilio
https://hg.mozilla.org/integration/autoland/rev/0cc93b4162bb
Implement the XPIDL and C++ parts of a counter metric. r=chutten,brizental
https://hg.mozilla.org/integration/autoland/rev/e312798400e8
Implement the XPIDL and C++ parts of a timespan metric.  r=chutten

Comment 13

[Jan-Erik Rediger [:janerik]]

Attachment: Bug 1646165 - Make the test metrics file known to sphinx.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3a61ec27f575
Make the test metrics file known to sphinx. r=janerik CLOSED TREE

Comment 15

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/1b5253d54219
https://hg.mozilla.org/mozilla-central/rev/a72fe6375f80
https://hg.mozilla.org/mozilla-central/rev/2b75c70b1f2d
https://hg.mozilla.org/mozilla-central/rev/af5e9b06ff01
https://hg.mozilla.org/mozilla-central/rev/0cc93b4162bb
https://hg.mozilla.org/mozilla-central/rev/e312798400e8

Comment 16

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/3a61ec27f575