1675534 - Vendor 'glean' in mozilla-central

Status: RESOLVED FIXED

(Data Platform and Tools :: Glean: SDK, task, P1)

Description

[Alessio Placitelli [:Dexter]]

We should bump glean-core in m-c to the latest version of glean-core and additionally vendor the glean crate.

Comment 1

[Alessio Placitelli [:Dexter]]

Ok, I finally managed to get this working on a fresh VM.

On a blank, recent m-c, ./mach vendor rust reports the following:

Package instant has a license that is approved for build-time dependencies: BSD-3-Clause
but the package itself is not whitelisted as being a build-time only package.

If your package is build-time only, please add it to the whitelist of build-time
only packages. Otherwise, you need to request license review on the package's license.
If the package's license is approved, please add it to the whitelist of suitable licenses.

Which seems to be fallout from bug 1674917, that added instant in this revision.

Comment 2

[Alessio Placitelli [:Dexter]]

Attempting to bump the version of glean-core to 33.1.2 both here and here results in new dependencies being added, specifically:

• adler v0.2.3
• autocfg v1.0.1

Both seem to come from flate2 versions > 1.0.12 , which we recently upgraded to v1.0.19 (updated dependencies are due to the version bump of miniz_oxide which happened in v1.0.13).

Jan-Erik, what should we do here?

• Should we update flate2 in-tree and add the 2 new dependencies?
• Should we rollback the version of flate2 used by the SDK?

Comment 3

[Chris H-C :chutten]

flate2 is depended on by zip and object, and are asking for "1" or "1.0" so that should be fine. I say upgrade flate2 in-tree and pull in those two new libs (so long as their licences are good). flate2 is housed under the rust-lang github org, so we can probably trust that they're keeping an eye on their deps.

Comment 4

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1675534 - Bump glean-core to 33.1.2. r?chutten

This additionally brings in two more Rust dependencies:
adler and autocfg. They are included in one of the latest
flate2 version.

Comment 5

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1675534 - Vendor glean (Rust Language Bindings). r?chutten

This is vendoring the Glean Rust Language Bindings (built on the
top of glean-core), providing a nice Glean SDK Rust API for consumers,
for using in FOG.

Depends on D96226

Comment 6

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1675534 - Update glean-parser to v1.29.0. r?chutten

This version bump is required by the Rust glean-core version
bump to v33.1.2.

Depends on D96227

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/35d4d9bddadd
Bump glean-core to 33.1.2. r=chutten
https://hg.mozilla.org/integration/autoland/rev/2c75ab4daa47
Vendor glean (Rust Language Bindings). r=chutten
https://hg.mozilla.org/integration/autoland/rev/ee45a6abdadf
Update glean-parser to v1.29.0. r=janerik

Comment 8

[Natalia Csoregi [:nataliaCs]]

Backed out for causing build bustage.

backout: https://hg.mozilla.org/integration/autoland/rev/e1209a7ebf0da07b48eeffd492a585a2c28a4cc4

push: https://treeherder.mozilla.org/jobs?repo=autoland&revision=ee45a6abdadf6e61898dd77d07a4662dc8771f78

failure log: https://treeherder.mozilla.org/logviewer?job_id=321171972&repo=autoland&lineNumber=42164

[task 2020-11-09T11:31:12.364Z] 11:31:12 INFO - Compiling style_traits v0.0.1 (/builds/worker/checkouts/gecko/servo/components/style_traits)
[task 2020-11-09T11:31:12.366Z] 11:31:12 INFO - Running CARGO=/builds/worker/fetches/rustc/bin/cargo CARGO_CRATE_NAME=style_traits CARGO_MANIFEST_DIR=/builds/worker/checkouts/gecko/servo/components/style_traits CARGO_PKG_AUTHORS='The Servo Project Developers' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=style_traits CARGO_PKG_REPOSITORY='' CARGO_PKG_VERSION=0.0.1 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=0 CARGO_PKG_VERSION_PATCH=1 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/builds/worker/workspace/obj-build/debug/deps:/builds/worker/fetches/rustc/lib:/builds/worker/fetches/clang/lib' /builds/worker/fetches/sccache/sccache /builds/worker/fetches/rustc/bin/rustc --crate-name style_traits servo/components/style_traits/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=1 -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on --cfg 'feature="gecko"' -C metadata=e8f7efaaf4fda1dc -C extra-filename=-e8f7efaaf4fda1dc --out-dir /builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps --target x86_64-apple-darwin -C linker=/builds/worker/checkouts/gecko/build/cargo-linker -L dependency=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps -L dependency=/builds/worker/workspace/obj-build/debug/deps --extern app_units=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libapp_units-effdacc423d9138d.rmeta --extern bitflags=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libbitflags-65be83f25adaa42d.rmeta --extern cssparser=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libcssparser-c6f714a395cc2953.rmeta --extern euclid=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libeuclid-94c44f1d803e1a63.rmeta --extern lazy_static=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/liblazy_static-97100ecff2467a05.rmeta --extern malloc_size_of=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libmalloc_size_of-dc27041b98576f54.rmeta --extern malloc_size_of_derive=/builds/worker/workspace/obj-build/debug/deps/libmalloc_size_of_derive-92e30b38ad7dd1da.so --extern selectors=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libselectors-f3c2c58fe841c40a.rmeta --extern serde=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libserde-9e1dc95d74be46a0.rmeta --extern servo_arc=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libservo_arc-d5e4d38c661ee8b6.rmeta --extern to_shmem=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libto_shmem-4ac6281c333212f4.rmeta --extern to_shmem_derive=/builds/worker/workspace/obj-build/debug/deps/libto_shmem_derive-267ef3fdcef80113.so -C opt-level=2 -C debuginfo=2 -C force-frame-pointers=yes -Dwarnings -C codegen-units=1
[task 2020-11-09T11:31:12.366Z] 11:31:12 INFO - error: use of deprecated function bincode::config: please use options() instead
[task 2020-11-09T11:31:12.366Z] 11:31:12 INFO - --> media/audioipc/audioipc/src/codec.rs:173:25
[task 2020-11-09T11:31:12.366Z] 11:31:12 INFO - |
[task 2020-11-09T11:31:12.366Z] 11:31:12 INFO - 173 | if let Err(e) = bincode::config()
[task 2020-11-09T11:31:12.366Z] 11:31:12 INFO - | ^^^^^^^^^^^^^^^
[task 2020-11-09T11:31:12.367Z] 11:31:12 INFO - |
[task 2020-11-09T11:31:12.367Z] 11:31:12 INFO - = note: -D deprecated implied by -D warnings
[task 2020-11-09T11:31:12.367Z] 11:31:12 ERROR - error: aborting due to previous error
[task 2020-11-09T11:31:12.367Z] 11:31:12 INFO - error: could not compile audioipc.
[task 2020-11-09T11:31:12.367Z] 11:31:12 INFO - Caused by:
[task 2020-11-09T11:31:12.369Z] 11:31:12 INFO - process didn't exit successfully: CARGO=/builds/worker/fetches/rustc/bin/cargo CARGO_CRATE_NAME=audioipc CARGO_MANIFEST_DIR=/builds/worker/checkouts/gecko/media/audioipc/audioipc CARGO_PKG_AUTHORS='Matthew Gregan <[e-mail]>:Dan Glastonbury <[e-mail]>' CARGO_PKG_DESCRIPTION='Remote Cubeb IPC' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=audioipc CARGO_PKG_REPOSITORY='' CARGO_PKG_VERSION=0.2.5 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=2 CARGO_PKG_VERSION_PATCH=5 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/builds/worker/workspace/obj-build/debug/deps:/builds/worker/fetches/rustc/lib:/builds/worker/fetches/clang/lib' /builds/worker/fetches/sccache/sccache /builds/worker/fetches/rustc/bin/rustc --crate-name audioipc --edition=2018 media/audioipc/audioipc/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=1 -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C debug-assertions=on -C metadata=e08bc9c5e47181a6 -C extra-filename=-e08bc9c5e47181a6 --out-dir /builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps --target x86_64-apple-darwin -C linker=/builds/worker/checkouts/gecko/build/cargo-linker -L dependency=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps -L dependency=/builds/worker/workspace/obj-build/debug/deps --extern audio_thread_priority=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libaudio_thread_priority-b24bf5cc4a4a2e04.rlib --extern bincode=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libbincode-8e31d6eaf7b7c62a.rmeta --extern bytes=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libbytes-dca8d63abb86b153.rmeta --extern cubeb=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libcubeb-98c82856f28d141b.rmeta --extern error_chain=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/liberror_chain-97e697a4cb0526e4.rmeta --extern futures=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libfutures-a493ce893bceaa96.rmeta --extern iovec=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libiovec-e413138183c77f45.rmeta --extern libc=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/liblibc-aa7601272941c09e.rmeta --extern log=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/liblog-23a071dbc06f62dd.rmeta --extern memmap=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libmemmap-d3ff3e6f3c2b3a09.rmeta --extern mio=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libmio-eda031a1241c7f62.rmeta --extern mio_uds=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libmio_uds-2bed04365e9290ed.rmeta --extern serde=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libserde-9e1dc95d74be46a0.rmeta --extern serde_derive=/builds/worker/workspace/obj-build/debug/deps/libserde_derive-93d247e202fcca87.so --extern tokio=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libtokio-0d66c68bb747a1bc.rmeta --extern tokio_io=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libtokio_io-2c63ac3910b74967.rmeta --extern tokio_reactor=/builds/worker/workspace/obj-build/x86_64-apple-darwin/debug/deps/libtokio_reactor-0094d01e295b0e39.rmeta -C opt-level=2 -C debuginfo=2 -C force-frame-pointers=yes -Dwarnings -C codegen-units=1 (exit code: 1)
[task 2020-11-09T11:31:12.370Z] 11:31:12 INFO - warning: build failed, waiting for other jobs to finish...

Comment 9

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1675534 - Fix audioipc deprecation warning due to bincode update. r?kinetik

glean-core requires updating bincode from 1.2.1 to 1.3.1. The new version
deprecates 'config' and introduces 'options', which means that a build time
warning is emitted. This ends up breaking the builds on TC.

Depends on D96305

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/72fd081de136
Bump glean-core to 33.1.2. r=chutten
https://hg.mozilla.org/integration/autoland/rev/242953f682a5
Vendor glean (Rust Language Bindings). r=chutten
https://hg.mozilla.org/integration/autoland/rev/8af8c98229a8
Update glean-parser to v1.29.0. r=janerik
https://hg.mozilla.org/integration/autoland/rev/27ee1ce4cfc7
Fix audioipc deprecation warning due to bincode update. r=kinetik

Comment 11

[Cosmin Sabou [:CosminS]]

Backed out for causing mass test failures on tree.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=27ee1ce4cfc7c2389fa5cd2a1582cd421b136c93&selectedTaskRun=Nm7moFnIQlCRQ9CmX8A7Kg.0

Failure logs: https://treeherder.mozilla.org/logviewer?job_id=321276276&repo=autoland
https://treeherder.mozilla.org/logviewer?job_id=321277597&repo=autoland
https://treeherder.mozilla.org/logviewer?job_id=321277284&repo=autoland
https://treeherder.mozilla.org/logviewer?job_id=321277582&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/ca5fb38e53452f3330659689f4e64cc6e836f9da

Comment 12

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1675534 - Bump glean-core to 33.3.0. r?chutten

This additionally brings in two more Rust dependencies:
adler and autocfg. They are included in one of the latest
flate2 version.

Comment 13

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1675534 - Vendor glean (Rust Language Bindings). r?chutten

This is vendoring the Glean Rust Language Bindings (built on the
top of glean-core), providing a nice Glean SDK Rust API for consumers,
for using in FOG.

Depends on D96833

Comment 14

[Alessio Placitelli [:Dexter]]

Attachment: Bug 1675534 - Update glean-parser to v1.29.0. r?chutten

This version bump is required by the Rust glean-core version
bump to v33.3.3.

Depends on D96834

Comment 15

[Alessio Placitelli [:Dexter]]

All right, this was trickier than expected. There was an incompatibility between bincode and the rkv version used in m-c, so we had to cut a new release of the SDK. Thanks Jan-Erik for figuring this out and fixing it.

here's the new try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=dfd69362625a917c1546054e1f81a228e43efe64

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/133512f263d0
Bump glean-core to 33.3.0. r=chutten
https://hg.mozilla.org/integration/autoland/rev/b088e8256f9c
Vendor glean (Rust Language Bindings). r=chutten
https://hg.mozilla.org/integration/autoland/rev/8bbb403aad3f
Update glean-parser to v1.29.0. r=chutten

Comment 17

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/133512f263d0
https://hg.mozilla.org/mozilla-central/rev/b088e8256f9c
https://hg.mozilla.org/mozilla-central/rev/8bbb403aad3f