Last Comment Bug 828317 - Require pymake for building on Windows
: Require pymake for building on Windows
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: All Windows 7
: -- normal (vote)
: mozilla24
Assigned To: Gregory Szorc [:gps]
:
Mentors:
Depends on: 842445
Blocks: 775939
  Show dependency treegraph
 
Reported: 2013-01-09 06:41 PST by Ted Mielczarek [:ted.mielczarek]
Modified: 2013-05-30 11:42 PDT (History)
14 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Require pymake to build on Windows, v1 (2.72 KB, patch)
2013-02-18 16:04 PST, Gregory Szorc [:gps]
ted: review+
Details | Diff | Review
Require pymake to build on Windows, v2 (3.59 KB, patch)
2013-02-21 11:15 PST, Gregory Szorc [:gps]
ted: review+
Details | Diff | Review

Description Ted Mielczarek [:ted.mielczarek] 2013-01-09 06:41:45 PST
We should stop supporting GNU make on Windows and require Pymake.

Since mach uses Pymake automatically, the error message could instruct the user to use "./mach build" instead.
Comment 1 Honza Bambas (:mayhemer) 2013-01-09 11:46:21 PST
Just a note that when a bit mysterious bug 824004 that disappears and reappears at its own will is not resolved I may be completely blocked from build the tree.  It means that all pymake related issues must be resolved or a override for this requirement must be available.

Personally I don't understand why you want to remove the GNU make supports at all.
Comment 2 Gregory Szorc [:gps] 2013-01-09 11:50:52 PST
(In reply to Honza Bambas (:mayhemer) from comment #1)
> Personally I don't understand why you want to remove the GNU make supports
> at all.

It is easier to support and debug N - 1 build configurations.
Comment 3 Honza Bambas (:mayhemer) 2013-01-09 11:58:58 PST
OK, if that is just about "not supporting" in means of not maintaining the code by some paid mozilla contributor, then I think we could just claim the support is dead but leave the option be supported by the community (here me).  If that is possible of course and doesn't mean to still leave effort on build config developers.

I'm just worried to get blocked from work by being dependent on faulty pymake with no other option to build.
Comment 4 Ted Mielczarek [:ted.mielczarek] 2013-01-09 12:11:41 PST
Our tinderbox machines are building with pymake, so GNU make is no longer a well-tested solution. Furthermore, GNU make on Windows has a long-standing bug where it hangs with multi-core builds, meaning that it's a huge potential footgun. Finally, because of path differences between pymake and gmake, supporting both means that it's hard (or impossible) to get a bunch of corner cases right (for example, bug 773423).
Comment 5 Ted Mielczarek [:ted.mielczarek] 2013-01-09 12:12:28 PST
I would like this configuration to error out so that users don't accidentally use an unsupported tool and find themselves with strange errors as a result.
Comment 6 Honza Bambas (:mayhemer) 2013-01-09 13:06:49 PST
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> Our tinderbox machines are building with pymake, so GNU make is no longer a
> well-tested solution. Furthermore, GNU make on Windows has a long-standing
> bug where it hangs with multi-core builds, meaning that it's a huge
> potential footgun. 

Yeah, I know this very well ;)

> Finally, because of path differences between pymake and
> gmake, supporting both means that it's hard (or impossible) to get a bunch
> of corner cases right (for example, bug 773423).

This is an argument then.

Ted, I hope I can count on you to help with any issues that would prevent build the tree only with pymake where make might work.
Comment 7 Ted Mielczarek [:ted.mielczarek] 2013-01-09 17:16:47 PST
Absolutely. Pymake is our officially supported build platform for Windows, so bugs there are always important.
Comment 8 Gregory Szorc [:gps] 2013-02-18 16:04:15 PST
Created attachment 715276 [details] [diff] [review]
Require pymake to build on Windows, v1

This bug could use a patch!

I don't have access to a Windows machine ATM, so this patch is untested. Would appreciate testing as part of review. Or, I can wait until Wednesday to test on Windows myself and/or use Try.
Comment 9 Mike Hommey [:glandium] 2013-02-18 22:20:57 PST
Comment on attachment 715276 [details] [diff] [review]
Require pymake to build on Windows, v1

Why not just run pymake instead of complaining?
Comment 10 Mike Hommey [:glandium] 2013-02-18 22:34:22 PST
(In reply to Mike Hommey [:glandium] from comment #9)
> Comment on attachment 715276 [details] [diff] [review]
> Require pymake to build on Windows, v1
> 
> Why not just run pymake instead of complaining?

for clarification: "why not just have make run pymake instead of erroring-out?" is what was meant above.
Comment 11 Gregory Szorc [:gps] 2013-02-18 22:49:53 PST
Is there a robust way to obtain all the arguments passed into make? Yes, you can use $(MAKEFLAGS). But how do you get at -C and/or -f? $(firstword $(MAKEFILE_LIST))?

My personal opinion here is people should start using an intelligent driver (like mach) that automatically selects the appropriate build backend for them (which may not be make/pymake in the future).
Comment 12 neil@parkwaycc.co.uk 2013-02-19 01:32:28 PST
(In reply to Gregory Szorc from comment #11)
> Is there a robust way to obtain all the arguments passed into make? Yes, you
> can use $(MAKEFLAGS). But how do you get at -C and/or -f? $(firstword
> $(MAKEFILE_LIST))?

FYI, -C doesn't apply because gmake will already have changed to the appropriate directory. The only time I use -f is for client.mk which I don't think includes rules.mk anyway. (This makes it easy to search shell history for -f to find the command to rebuild the whole tree!)
Comment 13 Masatoshi Kimura [:emk] 2013-02-19 18:39:06 PST
(In reply to Mike Hommey [:glandium] from comment #10)
> > Why not just run pymake instead of complaining?
> 
> for clarification: "why not just have make run pymake instead of
> erroring-out?" is what was meant above.

It's great. I'm tired to type "./build/pymake/make.py".

(In reply to Gregory Szorc [:gps] from comment #11)
> My personal opinion here is people should start using an intelligent driver
> (like mach) that automatically selects the appropriate build backend for
> them (which may not be make/pymake in the future).

I'm usually using ./mach, but I still have to use make at the moment. For example, bug 842387. Another example is building only the part of the tree. If I have to rebuild the entire tree whenever I changed something, it would be virtually impossible to develop Firefox on my slow laptop.
Comment 14 Gregory Szorc [:gps] 2013-02-19 18:40:44 PST
(In reply to Masatoshi Kimura [:emk] from comment #13)
> I'm usually using ./mach, but I still have to use make at the moment. For
> example, bug 842387. Another example is building only the part of the tree.
> If I have to rebuild the entire tree whenever I changed something, it would
> be virtually impossible to develop Firefox on my slow laptop.

$ ./mach build toolkit browser

That will perform an incremental build.
Comment 15 Honza Bambas (:mayhemer) 2013-02-20 08:03:48 PST
(In reply to Gregory Szorc [:gps] from comment #14)
> (In reply to Masatoshi Kimura [:emk] from comment #13)
> > I'm usually using ./mach, but I still have to use make at the moment. For
> > example, bug 842387. Another example is building only the part of the tree.
> > If I have to rebuild the entire tree whenever I changed something, it would
> > be virtually impossible to develop Firefox on my slow laptop.
> 
> $ ./mach build toolkit browser
> 
> That will perform an incremental build.

I just checked it works also for submodules like
src$ mach build netwerk/protocol/http/

But I hope that doing...

src/_obj$ pymake -C network/protocol/http

... is going to be preserved as well?
Comment 16 Ted Mielczarek [:ted.mielczarek] 2013-02-20 10:57:19 PST
(In reply to Honza Bambas (:mayhemer) from comment #15)
> But I hope that doing...
> 
> src/_obj$ pymake -C network/protocol/http
> 
> ... is going to be preserved as well?

We have no short-term plans to stop supporting that. Longer-term we hope to get rid of Makefiles as the build backend, so think of "mach build" as the carrot to change your habits so you don't get broken eventually. :)
Comment 17 Ted Mielczarek [:ted.mielczarek] 2013-02-20 11:14:56 PST
Comment on attachment 715276 [details] [diff] [review]
Require pymake to build on Windows, v1

Review of attachment 715276 [details] [diff] [review]:
-----------------------------------------------------------------

Can you put the same check in client.mk so we can error out early there as well?
Comment 18 Honza Bambas (:mayhemer) 2013-02-20 11:18:07 PST
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #16)
> We have no short-term plans to stop supporting that. Longer-term we hope to
> get rid of Makefiles as the build backend, so think of "mach build" as the
> carrot to change your habits so you don't get broken eventually. :)

I'm asking because I sometimes have more then just one _obj dir under my source dir, with different configs (browser, b2g X debug, optimized, tests etc..) and I work with both build configs using the same sources.  Then I just cd to the _obj I want to build, do my build commands for modules.  And when I want to rebuild the same modules in other build config, I just cd ../_obj2 and use bash history to quickly repeat build commands.  

If mach build can also support switch between .mozconfigs (actually having an option to specify a custom .mozconfig would be great!) then I'm OK.
Comment 19 Masatoshi Kimura [:emk] 2013-02-20 11:36:09 PST
(In reply to Gregory Szorc [:gps] from comment #14)
> (In reply to Masatoshi Kimura [:emk] from comment #13)
> > I'm usually using ./mach, but I still have to use make at the moment. For
> > example, bug 842387. Another example is building only the part of the tree.
> > If I have to rebuild the entire tree whenever I changed something, it would
> > be virtually impossible to develop Firefox on my slow laptop.
> 
> $ ./mach build toolkit browser
> 
> That will perform an incremental build.

Oh, when I tried it before, it didn't work.
Is there an corresponding mach command to "./build/pymake/make.py -C path/to/objdir/ export"?
I'm often using it to reduce the turn-around time.
Comment 20 Gregory Szorc [:gps] 2013-02-21 09:45:43 PST
Justin: I think you mentioned on IRC that SeaMonkey and possibly comm-central doesn't build with pymake yet. I don't want to push this out before you are ready. But, I don't want to wait indefinitely either. What timetable is good for you?
Comment 21 Gregory Szorc [:gps] 2013-02-21 09:47:59 PST
(In reply to Honza Bambas (:mayhemer) from comment #18)
> If mach build can also support switch between .mozconfigs (actually having
> an option to specify a custom .mozconfig would be great!) then I'm OK.

mach supports the MOZCONFIG environment variable to define which mozconfig is in effect:

MOZCONFIG=foo ./mach build
MOZCONFIG=bar ./mach build

(In reply to Masatoshi Kimura [:emk] from comment #19)
> Oh, when I tried it before, it didn't work.
> Is there an corresponding mach command to "./build/pymake/make.py -C
> path/to/objdir/ export"?
> I'm often using it to reduce the turn-around time.

This is bug 837824.
Comment 22 Justin Wood (:Callek) 2013-02-21 09:51:15 PST
(In reply to Gregory Szorc [:gps] from comment #20)
> Justin: I think you mentioned on IRC that SeaMonkey and possibly
> comm-central doesn't build with pymake yet. I don't want to push this out
> before you are ready. But, I don't want to wait indefinitely either. What
> timetable is good for you?

SeaMonkey is indeed not ready for pymake yet, we're working on it. But we're also broken due to Bug 842341 right now, so long as we don't have pymake. :ted doesn't seem willing to fix that though (see c#3)

So my priority is "fix that" so we can get SeaMonkey trunk and aurora building again.

If it is deemed wontfix we can land this, since we're broken anyway.

Either way, we're actively working on pymake in Bug 842445, and expect within the next few weeks to be switched over on trunk.
Comment 23 Gregory Szorc [:gps] 2013-02-21 11:15:02 PST
Created attachment 716677 [details] [diff] [review]
Require pymake to build on Windows, v2

Now with client.mk integration.
Comment 24 Gregory Szorc [:gps] 2013-02-21 11:16:05 PST
Waiting on SeaMonkey (until we get impatient).
Comment 25 Gregory Szorc [:gps] 2013-03-26 23:03:21 PDT
It's been a month. Is SeaMonkey ready?
Comment 26 Justin Wood (:Callek) 2013-03-29 23:21:53 PDT
(In reply to Gregory Szorc [:gps] from comment #25)
> It's been a month. Is SeaMonkey ready?

We discussed in IRC recently that its not *only* waiting on SeaMonkey now, it was waiting on l10n bustage. We switched over a while ago for our main builds but l10n (also for Firefox) was broken when we required pymake.

As discussed in IRC then enabling pymake my default is ok if you exclude l10n (e.g. --with-l10n-basedir and --disable-compile-environment as seperate switches to allow without pymake)
Comment 27 Gregory Szorc [:gps] 2013-05-20 17:35:43 PDT
Ted is sitting next to me and we both agreed there is no time like the present.

https://hg.mozilla.org/integration/mozilla-inbound/rev/c750d5d003dd
Comment 28 Ryan VanderMeulen [:RyanVM] 2013-05-21 10:46:14 PDT
https://hg.mozilla.org/mozilla-central/rev/c750d5d003dd

Note You need to log in before you can comment on or make changes to this bug.