Use mozmake for some windows builds

RESOLVED FIXED

Status

defect
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: glandium, Assigned: glandium)

Tracking

({feature})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 3 obsolete attachments)

Assignee

Description

6 years ago
Once a new mozillabuild is deployed, we'll be able to switch off pymake to mozmake. This however will need to ride the trains, since it relies on build system changes.
Assignee

Comment 1

6 years ago
I've done this more or less blindly, since i don't really know how this all works. The intent here is to add a tristate make_type holding either make, mozmake or pymake, allowing to switch between any of those. enable_pymake is kept for compatibility with what currently is in buildbot-configs, allowing for smoother updates.

Note, I'm not sure how we should handle try. Switching to mozmake really makes a world of difference on turnaround times, but pushing anything that doesn't have the patches required for that to work would fail (think, pushing aurora to try, or something like that)

Also note this is just preparative work, we're not going to switch to mozmake just yet. Heck, a new MozillaBuild with it hasn't even been built yet, let alone deployed.
Attachment #818158 - Flags: review?(catlee)
Assignee

Updated

6 years ago
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Try can branch on whether the current tip is a descendant of the changeset that added mozmake support.

$ hg contains -c tip afae5911a1e0

http://mercurial.selenic.com/wiki/ContainsExtension

(We should do this for practically everything that diverges over time if possible.)
When enabled, this flag will cause windows binaries to be created using a new make process. RelEng wants to be sure there aren't any hidden issues with the changes, especially those not easily visible to developers (such as l10n repacks).

Setting the "feature" keyword to ensure visibility to QA
Keywords: feature
Comment on attachment 818158 [details] [diff] [review]
Add a tristate make_type value

Review of attachment 818158 [details] [diff] [review]:
-----------------------------------------------------------------

looks good to me. have you run this through checkconfig or dump masters yet?

note that we can't examine hg ancestry from within the build factory, so for try, we won't know which make to use.

try is generally considered to be a mozilla-central-like repo, so whenever we make changes to m-c and project branches, we also change try's configuration. this does make it hard if you want to push aurora/beta/other code to try. I don't think we should try and solve that problem in general here, but if there are some workarounds we can do for make, I'm open to ideas!
Attachment #818158 - Flags: review?(catlee) → review+
Blocks: 928193
Assignee

Comment 5

6 years ago
(In reply to Chris AtLee [:catlee] from comment #4)
> looks good to me. have you run this through checkconfig or dump masters yet?

How do I do that?
Assignee

Comment 6

6 years ago
So, I have another idea, which would avoid the try problem:

- Add an environment variable to the buildbot config. Say, PYMAKE_IS_MOZMAKE=1. On every branch, twig, try, etc. That's not going to affect anything that doesn't have what follows.
- Patch pymake's make.py to invoke mozmake.exe when PYMAKE_IS_MOZMAKE is set. Except for the environment variable part, that's what I'm using on birch to make it use mozmake.

[The reason for the environment variable thing is to keep pymake a normally working pymake when people invoke it manually.]

Thoughts?
Why do we need to patch pymake? Can't we add something to config.mk and have make reinvoke self if it's launched with a suboptimal make implementation?
Assignee

Comment 8

6 years ago
(In reply to Gregory Szorc [:gps] from comment #7)
> Why do we need to patch pymake? Can't we add something to config.mk and have
> make reinvoke self if it's launched with a suboptimal make implementation?

That still leaves pymake as the middle man.
Assignee

Updated

6 years ago
Depends on: 942375
Assignee

Updated

6 years ago
Depends on: 942377
Assignee

Comment 9

6 years ago
(In reply to Mike Hommey [:glandium] from comment #8)
> (In reply to Gregory Szorc [:gps] from comment #7)
> > Why do we need to patch pymake? Can't we add something to config.mk and have
> > make reinvoke self if it's launched with a suboptimal make implementation?
> 
> That still leaves pymake as the middle man.

It's also hard to make that work for every invocation of make/pymake the buildbots do.
Assignee

Updated

6 years ago
Depends on: 943728
QA Contact: john+bugzilla
Assignee

Comment 10

5 years ago
This more or less implements the idea from comment 6, except it relies on the existing TINDERBOX_OUTPUT variable instead of adding a new one. I know this is a hack, but it is also:
- easier to backout than a buildbot-side change
- way less tricky than trying to reexecute the right thing from config.mk
- lessens the risk of running mozmake for some commands and pymake for others in the same builds.
Attachment #8396166 - Flags: review?(gps)
Assignee

Updated

5 years ago
Attachment #818158 - Attachment is obsolete: true
Comment on attachment 8396166 [details] [diff] [review]
Force pymake to fall back to mozmake when run on build slaves

Review of attachment 8396166 [details] [diff] [review]:
-----------------------------------------------------------------

I don't feel all cozy about hacking up make.py to do this. But it is the easiest solution and is progress.

::: build/pymake/make.py
@@ +11,5 @@
>  
>  import gc
>  
>  if __name__ == '__main__':
> +  if 'TINDERBOX_OUTPUT' in os.environ:

Please add a comment here about what this code path is used for.
Attachment #8396166 - Flags: review?(gps) → review+
Assignee

Updated

5 years ago
Attachment #8396166 - Attachment is obsolete: true
Comment on attachment 8398169 [details] [diff] [review]
Force pymake to fall back to mozmake when run on build slaves

Review of attachment 8398169 [details] [diff] [review]:
-----------------------------------------------------------------

r+++
Attachment #8398169 - Flags: review?(gps) → review+
Assignee

Comment 14

5 years ago
Attachment #8398172 - Flags: review?(mshal)
Comment on attachment 8398172 [details] [diff] [review]
Get mozmake from tooltool

This looks fine. We still need to get mozmake.exe itself reviewed for tooltool, though I'm not sure how we would review an exe...
Attachment #8398172 - Flags: review?(mshal) → review+
Assignee

Comment 16

5 years ago
B2G and c-c don't have tooltool yet, so that still needs to be conditional :(
Attachment #8398210 - Flags: review?(gps)
Assignee

Updated

5 years ago
Attachment #8398169 - Attachment is obsolete: true
Assignee

Comment 17

5 years ago
(In reply to Michael Shal [:mshal] from comment #15)
> Comment on attachment 8398172 [details] [diff] [review]
> Get mozmake from tooltool
> 
> This looks fine. We still need to get mozmake.exe itself reviewed for
> tooltool, though I'm not sure how we would review an exe...

That executable was built as per bug 944649, so in practice, we can consider it reviewed.
Attachment #8398210 - Flags: review?(gps) → review+
Assignee

Updated

5 years ago
Depends on: 941994
Blocks: 990739
I attempted to land this on comm-central, but it didn't go so well.
https://tbpl.mozilla.org/php/getParsedLog.php?id=37325971&tree=Thunderbird-Trunk

What needs to be done in addition to adding the tooltool manifests?
Oh, that's right, tooltool is explicitly disabled there. Wonderful. So if and when we land pymake-breaking patches on m-c, we're going to break c-c?
Assignee

Comment 22

5 years ago
The first thing that needs happening on cc is ccrework. Without that, cc can't build with anything but pymake. We can, however, start removing pymake support from mach.
You need to log in before you can comment on or make changes to this bug.