1164126 - Install rust toolchain for tooltool

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ralph Giles (:rillian)]

Several gecko teams are experimenting with using rust language code. We'd like to get the rust toolchain installed on build machines for some subset of tier-1 platforms so we can use this code in continuous integration.

If I understand tooltool correctly we should be able to use the official rust builds (or repackage them) just by adding them to the server and using the corresponding hashes in our build manifests.

Official rust toolchain builds are available at well known urls:

https://static.rust-lang.org/dist/rust-${version}-${arch}.tar.gz

Checksums and signatures for verification are available as similar urls:

https://static.rust-lang.org/dist/rust-${version}-${arch}.tar.gz.sha256
https://static.rust-lang.org/dist/rust-${version}-${arch}.tar.gz.asc

There are also text manifest files listing the be base products for each channel:

https://static.rust-lang.org/dist/channel-rust-nightly
https://static.rust-lang.org/dist/channel-rust-beta
https://static.rust-lang.org/dist/channel-rust-stable

Finally, for a download tool, see https://github.com/rust-lang/rustup

Comment 1

[Ralph Giles (:rillian)]

We'll want the 1.0.x stable release after it comes out on Friday, but for experimentation in the meantime the latest release is 1.0.0-beta.5.

https://static.rust-lang.org/dist/rust-1.0.0-beta.5-x86_64-unknown-linux-gnu.tar.gz
https://static.rust-lang.org/dist/rust-1.0.0-beta.5-x86_64-apple-darwin.tar.gz
https://static.rust-lang.org/dist/rust-1.0.0-beta.5-i686-pc-windows-gnu.tar.gz

I suggest we start with just these platforms for now. Rust doesn't support arm or android in official builds yet, so we're not ready for full tier-1 support anyway. If you want to reduce scope I'd prefer just linux64 or/and just macosx as an initial trial.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Ralph Giles (:rillian) from comment #0)
> If I understand tooltool correctly we should be able to use the official
> rust builds (or repackage them) just by adding them to the server and using
> the corresponding hashes in our build manifests.

Yup!  And you can do all of that yourself :)

For the uploading:
  https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool#How_To_Upload_To_Tooltool

For using them in your build manifests, I don't know the details of the the build system, but Mozharness has support for tooltool, for example.

Comment 3

[Ralph Giles (:rillian)]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)

> Yup!  And you can do all of that yourself :)

Huzzah for self service! Thanks for the link, I had no idea that was possible now.

Comment 4

[Ralph Giles (:rillian)]

1.0.0-beta.5 uploaded for linux64, mac, win32. Many thank to jlund for granting me upload permission.

[
{
"size": 101630664, 
"visibility": "public", 
"digest": "a39dbeadc75c20d14f47f3b64092b834015d07a22746fbe5edfd4a0301dfb11ff3957927ce5b9695670d0e53bdf761c073cbc660537394529a42af51efa6084f", 
"algorithm": "sha512", 
"filename": "rust-1.0.0-beta.5-x86_64-unknown-linux-gnu.tar.gz"
}, 
{
"size": 92563477, 
"visibility": "public", 
"digest": "6ae677cdb1a65c22303da8a0acd5f299a83aff36bc0b2c415a3bdc79a3618a75b9e815c0ae01557b839b3830be697964ce1eb98d0b324fdd5619ed723f945140", 
"algorithm": "sha512", 
"filename": "rust-1.0.0-beta.5-x86_64-apple-darwin.tar.gz"
}, 
{
"size": 108236324, 
"visibility": "public", 
"digest": "20dbd22ae38e253ea16e8a45a84738f8a07b377f47db367a13ac6d76333e4d916d50a9d52c3de8f0346dd6945aedd062360a6ae1bf515b02a4e6773ddabd1125", 
"algorithm": "sha512", 
"filename": "rust-1.0.0-beta.5-i686-pc-windows-gnu.tar.gz"
}
]

Comment 5

[Ralph Giles (:rillian)]

Here's an updated tooltool setup.sh to unpack the linux64 rustc build into build/src/rustc.

{
"size": 173,
"visibility": "public",
"digest": "4f4d5a303f5be1f5c7d4c4106f0ff8c5958a55ea41abbceefc87cc76e63d8cf646a2308ce19aec7f45a8d380efd5d575473fe0734202bbcb63a3dda697376869",
"algorithm": "sha512",
"filename": "setup.sh"
}

Comment 6

[Ralph Giles (:rillian)]

Attachment: Example patch to enable rustc on try

Here's a patch which successfully compiles the trivial in-tree gtest calling rust code. linux64 only so far; we need a smarter setup.sh in tooltool to handle more platforms.

Required changes:

- Update tooltool manifest to reference the rustc toolchain and the setup script to unpack it.
- Override default mozconfig to add rustc location to PATH and LD_LIBRARY_PATH.
- Override default mozconfig to pass --enable-rust to configure.

From https://treeherder.mozilla.org/#/jobs?repo=try&revision=a8a5d82c4db7

16:53:55     INFO -  INFO - File rust-1.0.0-beta.5-x86_64-unknown-linux-gnu.tar.gz retrieved from local cache /builds/tooltool_cache
[...]
16:54:04     INFO -  + rm -rf rustc
16:54:04     INFO -  + tar xf rust-1.0.0-beta.5-x86_64-unknown-linux-gnu.tar.gz --strip-components=1
[...]
16:54:17     INFO -  checking for rustc... /builds/slave/try-l64-d-00000000000000000000/build/src/rustc/bin//rustc
[...]
17:19:24     INFO - TEST-START | rust.CallFromCpp
17:19:24     INFO - TEST-PASS | rust.CallFromCpp | test completed (time: 0ms)

Comment 7

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

\o/