use in-tree configs whenever possible for servo builds

RESOLVED FIXED

Status

Release Engineering
General Automation
P3
normal
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: bhearsum, Assigned: jyeo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(6 attachments)

(Reporter)

Description

5 years ago
So that devs can do bugs like 891652 at the same time as the new code lands, and avoid unnecessary burning.
(Reporter)

Updated

5 years ago
Priority: -- → P3
(Reporter)

Comment 1

5 years ago
OK, so we can't do everything in tree. We can do something similar to b2g though, and move some things to an in-tree file. Basically, this would involve:
* Choosing a place to put per-build configs in tree (somewhere in https://github.com/mozilla/servo)
* Moving mock_packages, and concurrency from https://github.com/mozilla/build-mozharness/blob/master/configs/servo/linux.py into a new in-tree file.
* Moving concurrency from https://github.com/mozilla/build-mozharness/blob/master/configs/servo/mac.py into a new in tree file (and deleting that one, since it will be empty).
* Add a new step called "read-servo-config" to servo_build.py, and read the file into memory in it.
* Add another new step called "update-mock-packages", which looks at the servo config and calls self.install_mock_packages to install them. It will still need to look at self.config for the target. This step will need to be added to Linux's add_actions: https://github.com/mozilla/build-mozharness/blob/master/configs/servo/linux.py#L6
* Adjust concurrency references to look at servo config instead of self.config

Aki, does the above sound sane to you? My ideal world has the entire mozharness config in tree, but because we need mock before we have the Servo repo, we have a chicken+egg problem.
Flags: needinfo?(aki)
Summary: use in-tree mozharness configs for servo builds → use in-tree configs whene possible for servo builds
(Reporter)

Comment 2

5 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #1)
> * Choosing a place to put per-build configs in tree (somewhere in
> https://github.com/mozilla/servo)

Jack, any preference where the new configs go in the repo? We'll be starting with one for linux and one for mac, but eventually we may need one per platform+build type.
Flags: needinfo?(jack)

Comment 3

5 years ago
I proposed `servo/bld/{mac,linux}.cfg` or something like that. If you send me the files I can get them checked in, or just do a PR on Github.
Flags: needinfo?(jack)

Comment 4

5 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #1)
> OK, so we can't do everything in tree. We can do something similar to b2g
> though, and move some things to an in-tree file. Basically, this would
> involve:
> * Choosing a place to put per-build configs in tree (somewhere in
> https://github.com/mozilla/servo)
> * Moving mock_packages, and concurrency from
> https://github.com/mozilla/build-mozharness/blob/master/configs/servo/linux.
> py into a new in-tree file.
> * Moving concurrency from
> https://github.com/mozilla/build-mozharness/blob/master/configs/servo/mac.py
> into a new in tree file (and deleting that one, since it will be empty).
> * Add a new step called "read-servo-config" to servo_build.py, and read the
> file into memory in it.
> * Add another new step called "update-mock-packages", which looks at the
> servo config and calls self.install_mock_packages to install them. It will
> still need to look at self.config for the target. This step will need to be
> added to Linux's add_actions:
> https://github.com/mozilla/build-mozharness/blob/master/configs/servo/linux.
> py#L6
> * Adjust concurrency references to look at servo config instead of
> self.config
> 
> Aki, does the above sound sane to you? My ideal world has the entire
> mozharness config in tree, but because we need mock before we have the Servo
> repo, we have a chicken+egg problem.

This sounds sane.  It helps that the mock methods allow for overriding the list of packages, rather than only checking self.config.  We have support for everything you need here, today.

An additional solution that we don't currently support (but could add support for) would be allowing for a url to a config file.

python path/to/script -c http://link/to/raw/config

That would be your full-in-tree config.  The link to the config would have to specify a branch/revision, as pointing at tip is poor for production reproducibility.
Flags: needinfo?(aki)
(Assignee)

Comment 5

5 years ago
Created attachment 777120 [details] [diff] [review]
download config file
Assignee: nobody → yshun
Status: NEW → ASSIGNED
Attachment #777120 - Flags: review?(aki)

Comment 6

5 years ago
Comment on attachment 777120 [details] [diff] [review]
download config file

This looks good. Have you tested it?
Attachment #777120 - Flags: review?(aki) → review+
(Assignee)

Comment 7

5 years ago
Created attachment 777280 [details]
Logs from mozharness when using config url.
(Assignee)

Comment 8

5 years ago
Created attachment 777282 [details]
Logs from mozharness when using local config
(Assignee)

Comment 9

5 years ago
Yup I've tested and it is able to generate a similar set of config other than the config_files entry. The entry is an url if we choose to download the config. Otherwise, it contains a path to the config on the local filesystem. Would that be a problem?
(Assignee)

Comment 10

5 years ago
Created attachment 777308 [details]
Output from mozharness when there's no network connection

Comment 11

5 years ago
(In reply to Jason Yeo [:jyeo] from comment #9)
> Yup I've tested and it is able to generate a similar set of config other
> than the config_files entry. The entry is an url if we choose to download
> the config. Otherwise, it contains a path to the config on the local
> filesystem. Would that be a problem?

I think that's ok; I don't think anything uses that afterwards yet.
If we need the file path saved later, we can update our config dict appropriately at that point in mozharness.base.config.
Merged to production.
(Assignee)

Comment 13

5 years ago
Submitted a pull request: https://github.com/mozilla/servo/pull/594
The pull request just landed.
(Assignee)

Comment 15

5 years ago
Created attachment 777849 [details] [diff] [review]
replace_cfg_with_url.diff
Attachment #777849 - Flags: review?(bhearsum)
(Reporter)

Comment 16

5 years ago
Comment on attachment 777849 [details] [diff] [review]
replace_cfg_with_url.diff

Review of attachment 777849 [details] [diff] [review]:
-----------------------------------------------------------------

This looks fine. Going to land early tomorrow when things are quiet.
Attachment #777849 - Flags: review?(bhearsum) → review+
(Reporter)

Updated

5 years ago
Attachment #777849 - Flags: checked-in+
(Reporter)

Updated

5 years ago
Summary: use in-tree configs whene possible for servo builds → use in-tree configs whenever possible for servo builds
(Reporter)

Comment 17

5 years ago
Created attachment 778454 [details] [diff] [review]
use custom renderer to make mozharness_config get properly interpreted

Jason, we hit an issue with the Buildbot patch. I did some live debugging and came up with this.

Sorry we don't have a staging environment for you to work through things like this in :(.
Attachment #778454 - Flags: checked-in+
(Reporter)

Comment 18

5 years ago
This is working now. Huge thanks to Jason for making downloadable Mozharness configs possible!
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.