1662868 - Bring the Rust language binding into the Glean repositories

Status: RESOLVED FIXED

(Data Platform and Tools :: Glean: SDK, task, P1)

Description

[Chris H-C :chutten]

For the purposes of expediency many parts of the Rust Language Binding used by FOG are resident in mozilla-central. Now's the time to split off the pieces that are common across Rust consumers and factor them out into the glean repositories.

These pieces include:

• The Rust metrics API generation template and outputter for glean_parser
• The dispatcher and pre-init queue (see bug 1659109)
• Off-thread init
• Core metrics persistence
• The global GLEAN object
• The uploader interfaces (maybe not including the viaduct implementation)
• Anything else in toolkit/components/glean/src/api.rs or toolkit/components/glean/api/src that looks relevant
  • Not IPC, though. Or the files that include the generated Rust APIs.
• API documentation (not present in FOG, but you'll want to write it)
• Tests (watch out for the ones that expect IPC to be present)

Ideally you'll also reconfigure FOG to use these crates and outputter templates as a part of this effort, but maybe we'll need a bug in Toolkit::Telemetry for that part.

Comment 1

[Alessio Placitelli [:Dexter]]

Will leave this assigned to me, but the work is happening in other bugs.

Comment 2

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use the inherent crate in FOG and update Cargo.lock. r?janerik

Comment 3

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use Glean's Boolean implementation. r?janerik

Depends on D97545

Comment 4

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use Glean's Counter implementation. r?janerik

Depends on D97546

Comment 5

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use Glean's String implementation. r?janerik

Depends on D97547

Comment 6

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - fixme: various

Depends on D97548

Comment 7

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - REMOVE ME: Update to WIP glean

Depends on D97549

Comment 8

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - test2

Depends on D97550

Comment 9

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use the inherent crate in FOG and update Cargo.lock. r?janerik

Comment 10

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use Glean's Boolean implementation. r?janerik

Depends on D98059

Comment 11

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use Glean's Counter implementation. r?janerik

Depends on D98060

Comment 12

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use Glean's String implementation. r?janerik

Depends on D98061

Comment 13

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use Glean's Labeled implementation. r?janerik

In addition to use the labeled type from the RLB, this defines
new sealed traits to be used in m-c for restricting the possible
types that are allowed to be labeled.

Depends on D98062

Comment 14

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use Glean's General API. r?janerik

Depends on D98063

Comment 15

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use Glean's Uuid implementation. r?janerik

Depends on D98064

Comment 16

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Use Glean's Ping type. r?janerik

Depends on D98065

Comment 17

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Create a Viaduct-based Glean uploader. r?janerik

This implements the Glean ping uploader trait to provide Firefox-specific
Viaduct-based uploading capabilities. This is mostly what was in the
'register_uploader' function, with little changes to adjust it to the
new trait-based structure.

Depends on D98066

Comment 18

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Remove unused or obsolete FOG code. r?janerik

Depends on D98067

Comment 19

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Fixup the test utils to use the Glean RLB. r?janerik

Depends on D98068

Comment 20

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Disable Rust tests on non-RLB types for FOG. r?janerik

This is needed otherwise rusttests will hang: non-RLB types use
a different dispatcher, which locks up the execution of tests.
These tests are safe to be disabled, since we are not planning
on using these metrics before the RLB are fully back in m-c.

Depends on D98069

Comment 21

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Disable xpcshell tests on non-RLB types for FOG. r?janerik

Depends on D98070

Comment 22

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - REMOVE ME: vendor the Glean SDK from GH

This will need to be removed before merging. It's only here for
convenience, to enable reviewers testing this stack locally.

Depends on D98071

Comment 23

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Disable gtest on non-RLB types for FOG. r?janerik

Depends on D98071

Comment 24

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Vendor Glean 33.5.0. r?chutten

Depends on D98299

Comment 25

[Alessio Placitelli [:Dexter]]

Try run: https://treeherder.mozilla.org/jobs?repo=try&revision=bbb4f4a99c9cc7443e8d80622ebcf7a9f0fa7029&selectedTaskRun=Y-q4H9DBRlmwe-dok3qziw.0

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2f342eaea13e
Use the inherent crate in FOG and update Cargo.lock. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/6f4740140fe1
Use Glean's Boolean implementation. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/82c6c1b7a24a
Use Glean's Counter implementation. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/6a5e0257086c
Use Glean's String implementation. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/f631a1d04d0a
Use Glean's Labeled implementation. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/b727f5c658f5
Use Glean's General API. r=janerik
https://hg.mozilla.org/integration/autoland/rev/43245bce408a
Use Glean's Uuid implementation. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/6e38bfcb8587
Use Glean's Ping type. r=janerik
https://hg.mozilla.org/integration/autoland/rev/98415dd8ee7e
Create a Viaduct-based Glean uploader. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/1803b5acd0f8
Remove unused or obsolete FOG code. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/be7a168ee182
Fixup the test utils to use the Glean RLB. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/33a5ec2378cd
Disable Rust tests on non-RLB types for FOG. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/21f7b675b3b9
Disable xpcshell tests on non-RLB types for FOG. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/99df04f55be1
Disable gtest on non-RLB types for FOG. r=janerik
https://hg.mozilla.org/integration/autoland/rev/fe1462617a8d
Vendor Glean 33.5.0. r=chutten

Comment 27

[Alexandru Michis [:malexandru]]

Backed out 15 changesets (Bug 1662868) for causing xpcshell thread leaks.

Backout link: https://hg.mozilla.org/integration/autoland/rev/6fa5620ea6c598bf6adac559f8e5d43e4a3ea695

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&fromchange=1e6b3c1288586627d5905d6ce87764daf4f7bc2c&searchStr=linux%2C18.04%2Cx64%2Ctsan%2Copt%2Cxpcshell%2Ctests%2Ctest-linux1804-64-tsan%2Fopt-xpcshell-e10&tochange=18db050b9f5606ba641067039e1182d262b137f9&group_state=expanded&selectedTaskRun=TzfhvhleQFOtjKGPjxZoPA.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=323322485&repo=autoland&lineNumber=3630

Comment 28

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1662868 - Add a TSAN suppression for the Glean dispatcher. r?decoder

We will investigate the root cause of the TSAN problem in bug
1680474, but we need to land this massive stack first.

Depends on D98356

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e730de1209c7
Use the inherent crate in FOG and update Cargo.lock. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/2d0782cd3750
Use Glean's Boolean implementation. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/f41cdd40b9d4
Use Glean's Counter implementation. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/4a11c9af0b7f
Use Glean's String implementation. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/c8888e6f5402
Use Glean's Labeled implementation. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/f8c46a80f6a3
Use Glean's General API. r=janerik
https://hg.mozilla.org/integration/autoland/rev/17ed3d6338c9
Use Glean's Uuid implementation. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/9ee8a6765b48
Use Glean's Ping type. r=janerik
https://hg.mozilla.org/integration/autoland/rev/2f61e330b136
Create a Viaduct-based Glean uploader. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/ba78ebabdb7e
Remove unused or obsolete FOG code. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/16e411f7f49e
Fixup the test utils to use the Glean RLB. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/27fc2db19c00
Disable Rust tests on non-RLB types for FOG. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/db4c369f4e30
Disable xpcshell tests on non-RLB types for FOG. r=janerik,chutten
https://hg.mozilla.org/integration/autoland/rev/e4cc2a12b7fe
Disable gtest on non-RLB types for FOG. r=janerik
https://hg.mozilla.org/integration/autoland/rev/7f2e65c912fe
Vendor Glean 33.5.0. r=chutten
https://hg.mozilla.org/integration/autoland/rev/8fc94e3eefff
Add a TSAN suppression for the Glean dispatcher. r=decoder

Comment 30

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/e730de1209c7
https://hg.mozilla.org/mozilla-central/rev/2d0782cd3750
https://hg.mozilla.org/mozilla-central/rev/f41cdd40b9d4
https://hg.mozilla.org/mozilla-central/rev/4a11c9af0b7f
https://hg.mozilla.org/mozilla-central/rev/c8888e6f5402
https://hg.mozilla.org/mozilla-central/rev/f8c46a80f6a3
https://hg.mozilla.org/mozilla-central/rev/17ed3d6338c9
https://hg.mozilla.org/mozilla-central/rev/9ee8a6765b48
https://hg.mozilla.org/mozilla-central/rev/2f61e330b136
https://hg.mozilla.org/mozilla-central/rev/ba78ebabdb7e
https://hg.mozilla.org/mozilla-central/rev/16e411f7f49e
https://hg.mozilla.org/mozilla-central/rev/27fc2db19c00
https://hg.mozilla.org/mozilla-central/rev/db4c369f4e30
https://hg.mozilla.org/mozilla-central/rev/e4cc2a12b7fe
https://hg.mozilla.org/mozilla-central/rev/7f2e65c912fe
https://hg.mozilla.org/mozilla-central/rev/8fc94e3eefff