1591555 - Audit Glean SDK Dependencies and Compare to crates in mozilla-central

Status: RESOLVED FIXED

(Data Platform and Tools :: Glean: SDK, task, P1)

Description

[Chris H-C :chutten]

We need to make a list of all the dependencies the Glean SDK currently has (including their versions), and compare them to the libraries and versions already available in mozilla-central.

For example, we already know we use an older version of rkv than mozilla-central does (because of binary size reasons).

This bug need not also cover us making decisions about how to resolve any differences between the lists. That can be a follow-up if we want.

But we might want to include some work under this bug about documenting how we did this. Other products might have similar questions, so if there are any tools or tricks about how to generate and compare these lists, we might benefit from noting them down in the Book of Glean.

Comment 1

[Jan-Erik Rediger [:janerik]]

Note: the rkv version mismatch solved itself now (its dependency that caused us problems, lmdb-sys, was updated with a fix). We're updating our dependency.

Comment 2

[Alessio Placitelli [:Dexter]]

Note: something like cargo-tree can be used to get Glean SDK dependencies.

Comment 3

[Alessio Placitelli [:Dexter]]

Attachment: glean-core-deps.txt

These are the dependencies as detected by cargo tree.

These are the ones obtained by stitching together the cargo files (1, 2):

glean-core

serde = { version = "1.0.102", features = ["derive"] }
serde_json = "1.0.41"
rkv = "0.10.2"
bincode = "1.1.3"
log = "0.4.6"
uuid = { version = "0.8.1", features = ["v4"] }
failure = "0.1.5"
ffi-support = "0.3.5"
regex = { version = "1.3.0", default-features = false, features = ["std"] }
lazy_static = "1.3.0"
chrono = { version = "0.4.6", features = ["serde"] }
once_cell = "1.0.2"

glean-ffi

lazy_static = "1.3.0"
ffi-support = "0.3.5"
log = "0.4.6"
uuid = { version = "0.8.1", features = ["v4"] }

Comment 4

[Alessio Placitelli [:Dexter]]

These are the versions from the third_party/rust directory for the crates from comment 3:

Package	glean-core version	glean-ffi version	mozilla-central version	Mismatch?
serde	1.0.102		1.0.88	Mismatch
serde_json	1.0.41		1.0.26	Mismatch
rkv	0.10.2		0.10.2	Matching
bincode	1.1.3		1.0.0	Mismatch
log	0.4.6	0.4.6	0.4.6	Matching
uuid	0.8.1	0.8.1	0.7.4	Mismatch
failure	0.1.5		0.1.3	Mismatch
ffi-support	0.3.5	0.3.5		Missing in m-c
regex	1.3.0		1.1.9	Mismatch
lazy_static	1.3.0	1.3.0	1.4.0	Mismatch
chrono	0.4.6		0.4.6	Matching
once_cell	1.0.2		1.2.0	Mismatch

Comment 5

[Alessio Placitelli [:Dexter]]

Is comment 4 enough for this? Looks like we have a few version mismatches and a missing package. How should this be handled? Who would know?

Comment 6

[Chris H-C :chutten]

Well, if I may co-opt your table format:

Package	glean-core version	glean-ffi version	mozilla-central version	Mismatch?
serde	1.0.102		1.0.88	m-c too old
serde_json	1.0.41		1.0.26	m-c too old
rkv	0.10.2		0.10.2	Matching
bincode	1.1.3		1.0.0	m-c too old
log	0.4.6	0.4.6	0.4.6	Matching
uuid	0.8.1	0.8.1	0.7.4	m-c too old
failure	0.1.5		0.1.3	m-c too old
ffi-support	0.3.5	0.3.5		Missing in m-c
regex	1.3.0		1.1.9	m-c too old
lazy_static	1.3.0	1.3.0	1.4.0	m-c newer
chrono	0.4.6		0.4.6	Matching
once_cell	1.0.2		1.2.0	m-c newer

It looks as though we have three separate cases: the case where we need to update m-c, the case where we need to update glean-core or glean-ffi or both, and the case where we need to add to m-c.

My previous research for the latter case suggests that so long as we have an inclination that the to-be-added crate is well-maintained and likely to not be risky, we can add it to m-c. Since the one crate this applies to, ffi-support is a mozilla thing, I think we can meet those criteria.

As for updating Glean crates to match m-c, I assume you are up to the task, Alessio? (or Jan-Erik will be?)

For the last point about updating crates in-use in m-c... this is where I ask Lina if there's docs or prior art we can follow along with.

Comment 7

[Lina Butler [:lina]]

I think you can import multiple versions of the same crate (looking at third_party/rust, I see a few—miow, proc-macro2, quote, rand, semver, syn, unicode-xid, and winapi), but bug 1404894 wants to move away from that.

However, that means you'd either need to downgrade Glean to pull in the older m-c versions (yuck 😑), or bump and rerun mach vendor rust in m-c, and fix any in-tree Rust code if there's breakage (also yuck 😱). Bug 1404894 notes this exact problem:

Even totally unrelated projects (e.g. rust-url and webrender) that are vendored into m-c would need to make sure their entire dependency trees use compatible versions which IMO might be excessive.

...But I don't think we have a good answer. Bug 1322798, comment 11 has some nice suggestions; a separate bug to bump the m-c versions and update all dependents would simplify review. I think our general approach has been "if you need it, you bump it," though, which does increase the maintenance burden.

Nathan, can you think of a better approach, or any docs or prior art here?

Comment 8

[Nathan Froyd [:froydnj]]

I don't think we have docs or strong prior art here. I would encourage bringing the dependencies up to m-c's level where possible.

For places where m-c has older versions, we are not averse to bumping them when necessary. serde and friends just need to be updated with some mozilla-specific patches, I think. Patch-level version bumps (e.g. for failure) for bringing m-c closer to your crate level are probably OK.

We would like to only have single versions of vendored crates, but as Lina pointed out, this is a Hard Thing to do.

Does that answer your question?

Comment 9

[Chris H-C :chutten]

This answers my questions, thank you. Seems like it boils down to "Pick the least-worst option for your workflow" : )

Alessio, I guess we might need to have a discussion about what we want to do with each of these crates (bring them into agreement, add yet another copy into the tree) and then file some bugs. And then we call this one done.

Thanks for your help, Lina and :froydnj!

Comment 10

[Chris H-C :chutten]

Okidoki, discussion happened.

The plan is thus:

1. Upgrade the Glean SDK crates to at least match m-c's versions
2. See how many of the crates in m-c that need updating can be updated simply (and update them)
3. Any that can't, we'll vendor separate versions for the FOGotype (and trust that, since we're building this only in Nightly, any costs/dangers associated with multiple versions are borne only by Nightly)

Alessio's filing bugs for 1, I'll file a bug for 2 and 3.