861283 - initial build support for Servo

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

This bug is to track initial setup of builds of Servo. Basic requirements are:
* 64-bit Linux and Mac builds
* Builds per commit to the Servo repo (https://github.com/mozilla/servo)
* TBPL or other dashboard for results

Other information:
* Prerequisites are here: https://github.com/mozilla/servo/blob/master/README.md
* RoR is on github, likely need to mirror into our own Mercurial servers
** Including the submodules.
** vcs2vcs might need support for submodules
* Current build pulls down a known good rust to build rust with, we may need to preinstall or rehost that to avoid external dependencies.

Much more to come, this is just an initial braindump.

Comment 1

[David Caabeiro [:dcaabeiro]]

Just for anyone's interest, I just built servo on my Ubuntu 12.04 (x86_64) according to instructions at https://github.com/mozilla/servo and had to perform some additional steps:

sudo apt-get install libfreetype6-dev
sudo apt-get install libfontconfig1-dev
sudo apt-get install libglib2.0-dev
sudo ln -s /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0 /usr/lib/libpango-1.0.so

Of course YMMV but might be worth confirming these and adding them to README if proved relevant.

Comment 2

[[e-mail] (:bhearsum)]

Braindump of conversation with Graydon, Brian, and Jack today:
* Main developer interface is github pull requests. Current workflow is:
** When a new PR is created, bors creates a new commit for it
** When the PR is r+'ed, bors creates a new commit for it, merged against the latest master, and pushes it to a repository
** Buildbot notices the commit and runs a set of jobs against it
** If all jobs complete successfully and the commit can be fastforwarded to the latest master, the commit is landed in the master repository
** If any job fails an error is reported into the pull request with a link to the build on buildbot.

An example of this is: https://github.com/mozilla/rust/pull/5726.

Other notes:
* All of the above is done by a script that Graydon wrote called "bors".
* The Buildbot that does all the tests is publicly accessible. The parts of it that are used are:
** log viewing
** retriggering jobs
** triggering "try" builds with properties
* We should focus exclusively on Servo automation for now, but it's important that Rust+Servo automation stays in sync. This may mean moving rust automation onto RelEng infrastructure later
* Future considerations to take in mind: rust/servo want to be able to offer CI services for rust based projects.

Comment 3

[[e-mail] (:bhearsum)]

Brian and I chatted some more yesterday, here's the plan we're going to move forward with:
* RelEng will setup buildbot+bors on a machine within our network
* RelEng will dedicate some slaves for running the jobs (currently thinking 2 mac, and on-demand EC2 instances for Linux)
* Barring no security or other blocking issues, we'll somehow make the Buildbot WebStatus available publicly. This is an interim step until we can have a TBPL or similar job status display.
* Some developers will have direct access to the master + slaves to enable poking at them if something goes wrong. This will be reevaluated in the future, particularly if we get to a place where Servo builds are happening on our primary pool of masters/slaves.
* Fixes/enchancements to the Servo buildbot master + bors will primarily be done by Servo people for now. Generally, deployment  of those changes will happen through RelEng.

Brian, does this match your understanding of what we talked about yesterday? Is there anything I missed?

Comment 4

[Brian Anderson]

Ben, yes I am in agreement.

I will be holding off on adding a mac bot to our current buildbot setup in anticipation of switching to RelEng infrastructure. Next week I do intend to see if I can set up bors on the existing buildbot, as I don't have any experience with that script yet and I imagine it will need to be adapted for Servo.

Comment 5

[[e-mail] (:bhearsum)]

I had an initial conversation with opsec about the security considerations. They'll be getting back to me on Monday.

Comment 6

[[e-mail] (:bhearsum)]

Still working with opsec on how we're going to organize some things. Hope to get that figured out today/tomorrow. I've also been working on some of the puppet manifests for deploying the buildbot master + slaves. Should have a better timeline on when this will be done after things are firmed up with opsec.

Comment 7

[[e-mail] (:bhearsum)]

Status update: We've settled on requirements with opsec and are now waiting for the new vlans to be created (bug 866212). Once those are ready we can start bringing up machinery and getting buildbot/bors running. I'm PTO Wed-Fri this week, so this definitely won't be completed this week.

Comment 8

[[e-mail] (:bhearsum)]

It's been awhile, but Bors + Buildbot are setup and working fine. We've seen many changes landed in the past week or two.

Still to do, all tracked in dep bugs:
* Bring mac machines online. This is blocked on re-imaging of the machines, which is possibly blocked on some network stuff.
* Security review of the Servo networks. This is waiting on OpSec, and we need to reverify that everything still works after it's been completed.
* Proxy a read-only WebStatus to a public location. This is blocked on a security review of Buildbot.

Comment 9

[[e-mail] (:bhearsum)]

Not blocked on 760093 anymore. The mac machines are already reimaged. I enabled mac as one of the builders that Bors requires before merging to master in https://hg.mozilla.org/users/dmitchell_mozilla.com/puppet320/rev/56f3b98a6aab

Comment 10

[[e-mail] (:bhearsum)]

The remaining two bugs aren't real blockers. The Servo CI has been in use for weeks now, and is working very well as far as I know.

I do expect the public interface (bug 886356) to be available soon, but I don't know when the sec review (bug 874637) is happening.