Closed Bug 891870 Opened 11 years ago Closed 11 years ago

use in-tree configs whenever possible for servo builds

Categories

(Release Engineering :: General, defect, P3)

x86_64
Linux
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: jyeo)

Details

Attachments

(6 files)

So that devs can do bugs like 891652 at the same time as the new code lands, and avoid unnecessary burning.
Priority: -- → P3
OK, so we can't do everything in tree. We can do something similar to b2g though, and move some things to an in-tree file. Basically, this would involve:
* Choosing a place to put per-build configs in tree (somewhere in https://github.com/mozilla/servo)
* Moving mock_packages, and concurrency from https://github.com/mozilla/build-mozharness/blob/master/configs/servo/linux.py into a new in-tree file.
* Moving concurrency from https://github.com/mozilla/build-mozharness/blob/master/configs/servo/mac.py into a new in tree file (and deleting that one, since it will be empty).
* Add a new step called "read-servo-config" to servo_build.py, and read the file into memory in it.
* Add another new step called "update-mock-packages", which looks at the servo config and calls self.install_mock_packages to install them. It will still need to look at self.config for the target. This step will need to be added to Linux's add_actions: https://github.com/mozilla/build-mozharness/blob/master/configs/servo/linux.py#L6
* Adjust concurrency references to look at servo config instead of self.config

Aki, does the above sound sane to you? My ideal world has the entire mozharness config in tree, but because we need mock before we have the Servo repo, we have a chicken+egg problem.
Flags: needinfo?(aki)
Summary: use in-tree mozharness configs for servo builds → use in-tree configs whene possible for servo builds
(In reply to Ben Hearsum [:bhearsum] from comment #1)
> * Choosing a place to put per-build configs in tree (somewhere in
> https://github.com/mozilla/servo)

Jack, any preference where the new configs go in the repo? We'll be starting with one for linux and one for mac, but eventually we may need one per platform+build type.
Flags: needinfo?(jack)
I proposed `servo/bld/{mac,linux}.cfg` or something like that. If you send me the files I can get them checked in, or just do a PR on Github.
Flags: needinfo?(jack)
(In reply to Ben Hearsum [:bhearsum] from comment #1)
> OK, so we can't do everything in tree. We can do something similar to b2g
> though, and move some things to an in-tree file. Basically, this would
> involve:
> * Choosing a place to put per-build configs in tree (somewhere in
> https://github.com/mozilla/servo)
> * Moving mock_packages, and concurrency from
> https://github.com/mozilla/build-mozharness/blob/master/configs/servo/linux.
> py into a new in-tree file.
> * Moving concurrency from
> https://github.com/mozilla/build-mozharness/blob/master/configs/servo/mac.py
> into a new in tree file (and deleting that one, since it will be empty).
> * Add a new step called "read-servo-config" to servo_build.py, and read the
> file into memory in it.
> * Add another new step called "update-mock-packages", which looks at the
> servo config and calls self.install_mock_packages to install them. It will
> still need to look at self.config for the target. This step will need to be
> added to Linux's add_actions:
> https://github.com/mozilla/build-mozharness/blob/master/configs/servo/linux.
> py#L6
> * Adjust concurrency references to look at servo config instead of
> self.config
> 
> Aki, does the above sound sane to you? My ideal world has the entire
> mozharness config in tree, but because we need mock before we have the Servo
> repo, we have a chicken+egg problem.

This sounds sane.  It helps that the mock methods allow for overriding the list of packages, rather than only checking self.config.  We have support for everything you need here, today.

An additional solution that we don't currently support (but could add support for) would be allowing for a url to a config file.

python path/to/script -c http://link/to/raw/config

That would be your full-in-tree config.  The link to the config would have to specify a branch/revision, as pointing at tip is poor for production reproducibility.
Flags: needinfo?(aki)
Assignee: nobody → yshun
Status: NEW → ASSIGNED
Attachment #777120 - Flags: review?(aki)
Comment on attachment 777120 [details] [diff] [review]
download config file

This looks good. Have you tested it?
Attachment #777120 - Flags: review?(aki) → review+
Yup I've tested and it is able to generate a similar set of config other than the config_files entry. The entry is an url if we choose to download the config. Otherwise, it contains a path to the config on the local filesystem. Would that be a problem?
(In reply to Jason Yeo [:jyeo] from comment #9)
> Yup I've tested and it is able to generate a similar set of config other
> than the config_files entry. The entry is an url if we choose to download
> the config. Otherwise, it contains a path to the config on the local
> filesystem. Would that be a problem?

I think that's ok; I don't think anything uses that afterwards yet.
If we need the file path saved later, we can update our config dict appropriately at that point in mozharness.base.config.
Merged to production.
Submitted a pull request: https://github.com/mozilla/servo/pull/594
The pull request just landed.
Attachment #777849 - Flags: review?(bhearsum)
Comment on attachment 777849 [details] [diff] [review]
replace_cfg_with_url.diff

Review of attachment 777849 [details] [diff] [review]:
-----------------------------------------------------------------

This looks fine. Going to land early tomorrow when things are quiet.
Attachment #777849 - Flags: review?(bhearsum) → review+
Attachment #777849 - Flags: checked-in+
Summary: use in-tree configs whene possible for servo builds → use in-tree configs whenever possible for servo builds
Jason, we hit an issue with the Buildbot patch. I did some live debugging and came up with this.

Sorry we don't have a staging environment for you to work through things like this in :(.
Attachment #778454 - Flags: checked-in+
This is working now. Huge thanks to Jason for making downloadable Mozharness configs possible!
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: