774109 - bootstrap support for OS X

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

Mach should have support for bootstrapping the development environment on OS X.

1) Start with a fresh install of OS X.
2) Obtain source code for mozilla-central (either a source bundle or repository clone)
3) Run ./mach and all build prerequisites are downloaded, installed, and configured properly.

The goal is to make it super simple for people to build mozilla-central. Your terminal-literate grandmother should be able to do it.

This bug essentially involves automating https://developer.mozilla.org/En/Developer_Guide/Build_Instructions/Mac_OS_X_Prerequisites. Basically,

1) Install Xcode
2) Install Homebrew or MacPorts
3) Install required packages from Homebrew or MacPorts

Ideally, bootstrap support would not require a full copy of mozilla-central. We could provide a one-liner for someone to `curl | python` and magic would ensue. This could, of course, just talk to hg.mozilla.org over HTTPS (not using Mercurial) and pull down the latest .py files needed to start performing more magic. This minimal bootstrap is beyond the scope of this bug, but it should be considered during implementation.

Comment 1

[Gregory Szorc [:gps]]

Since we now build with Clang on OS X and since we have the SVN revision of LLVM/Clang we build with checked into the repository, it would be splendidly awesome if bootstrap also ensured the appropriate version of Clang were installed.

Bug 768879 tracks exposing the Clang builds Mozilla uses on its official infrastructure to the public internet. Bootstrap could theoretically fetch binaries from there.

Comment 2

[Gregory Szorc [:gps]]

I just discovered B2G has a bootstrap script for OS X. We should consider stealing parts. https://github.com/mozilla-b2g/B2G/blob/master/scripts/bootstrap-mac.sh

Comment 3

[Gregory Szorc [:gps]]

Attachment: Bootstrap backend for OS X, v1

It bootstraps a vanilla OS X install to the point where it can build m-c.

I've only tested this on OS X 10.8. I could probably check this in without review b/c it isn't part of the build. But, I'd like just a cursory glance over the code.

Things like user prompting and support for additional OS X versions will come in follow-up bugs. Something is better than nothing.

Comment 4

[Gregory Szorc [:gps]]

This isn't part of mach.

Comment 5

[Richard Newman [:rnewman]]

Doesn't apply to current m-c…

unable to find 'python/mozboot/mozboot/base.py' for patching
1 out of 1 hunks FAILED -- saving rejects to file python/mozboot/mozboot/base.py.rej
unable to find 'python/mozboot/mozboot/bootstrap.py' for patching
1 out of 1 hunks FAILED -- saving rejects to file python/mozboot/mozboot/bootstrap.py.rej
unable to find 'python/mozboot/mozboot/osx.py' for patching
1 out of 1 hunks FAILED -- saving rejects to file python/mozboot/mozboot/osx.py.rej

Missing dependency?

Comment 6

[Gregory Szorc [:gps]]

(In reply to Richard Newman [:rnewman] from comment #5)
> Doesn't apply to current m-c…
> 
> unable to find 'python/mozboot/mozboot/base.py' for patching
> 1 out of 1 hunks FAILED -- saving rejects to file
> python/mozboot/mozboot/base.py.rej
> unable to find 'python/mozboot/mozboot/bootstrap.py' for patching
> 1 out of 1 hunks FAILED -- saving rejects to file
> python/mozboot/mozboot/bootstrap.py.rej
> unable to find 'python/mozboot/mozboot/osx.py' for patching
> 1 out of 1 hunks FAILED -- saving rejects to file
> python/mozboot/mozboot/osx.py.rej
> 
> Missing dependency?

Those files are in inbound and haven't been merged to central yet.

Comment 7

[Gregory Szorc [:gps]]

Attachment: Bootstrap backend for OS X, v2

Performed some general code cleanup by refactoring the printed content to not be inline.

More importantly, I added a check for the OS X command line tools version. Previously, we allowed older, non-functional compilers in /usr/bin to exist. Now, we prompt to upgrade the Xcode command line tools version.

I've still only tested this on 10.8. But, I'll soon have my hands on a 10.5, 10.6, and 10.7 machine. If the instructions need refined on older OS versions, I'm fine with these being follow-up bugs.

Comment 8

[Gregory Szorc [:gps]]

Attachment: Bootstrap backend for OS X, v3

I /think/ this will work with OS X 10.6. Although, I haven't actually tested. Apply on top of m-c (merge finally happened, yay) and let me know if it works.

Comment 9

[Richard Newman [:rnewman]]

(In reply to Gregory Szorc [:gps] from comment #8)

> I /think/ this will work with OS X 10.6. Although, I haven't actually
> tested. Apply on top of m-c (merge finally happened, yay) and let me know if
> it works.

Running "python python/mozboot/mozboot/bootstrap.py" on 10.6.8 with current m-c:

---
Traceback (most recent call last):
  File "python/mozboot/mozboot/bootstrap.py", line 10, in <module>
    from mozboot.centos import CentOSBootstrapper
ImportError: No module named mozboot.centos

Comment 10

[Gregory Szorc [:gps]]

Run bin/bootstrap.py instead.

Comment 11

[Richard Newman [:rnewman]]

(In reply to Gregory Szorc [:gps] from comment #10)
> Run bin/bootstrap.py instead.

Traceback (most recent call last):
  File "python/mozboot/bin/bootstrap.py", line 134, in <module>
    sys.exit(main(sys.argv))
  File "python/mozboot/bin/bootstrap.py", line 125, in main
    dasboot.bootstrap()
  File "python/mozboot/mozboot/bootstrap.py", line 54, in bootstrap
    instance.install_system_packages()
  File "python/mozboot/mozboot/osx.py", line 15, in install_system_packages
    raise NotImplementedError('OS X bootstrap not yet implemented.')
NotImplementedError: OS X bootstrap not yet implemented.

Comment 12

[Richard Newman [:rnewman]]

Wait, I qpopped. I actually get:

Traceback (most recent call last):
  File "python/mozboot/bin/bootstrap.py", line 134, in <module>
    sys.exit(main(sys.argv))
  File "python/mozboot/bin/bootstrap.py", line 125, in main
    dasboot.bootstrap()
  File "python/mozboot/mozboot/bootstrap.py", line 68, in bootstrap
    instance.install_system_packages()
  File "python/mozboot/mozboot/osx.py", line 98, in install_system_packages
    self.ensure_xcode()
  File "python/mozboot/mozboot/osx.py", line 105, in ensure_xcode
    if not os.path.exsts('/Developer/Applications/Xcode.app'):
AttributeError: 'module' object has no attribute 'exsts'

Let me fix that typo…

Comment 13

[Richard Newman [:rnewman]]

With typo fixed I get:

Homebrew needs XQuartz installed in order to build some dependencies. Please
download and install XQuartz from the following URL:

    http://xquartz.macosforge.org/downloads/SL/XQuartz-2.7.3.dmg
When that has finished, please relaunch bootstrap.

This seems inaccurate, because I have a mozilla-central build running with a HomeBrew-installed clang *right now*. Correct?

Comment 14

[Gregory Szorc [:gps]]

Attachment: Bootstrap backend for OS X, v4

Conditionally require XQuartz depending on whether Python is already installed.

This may not be perfect. I'm trying to sort out what Homebrew is doing...

Comment 15

[Richard Newman [:rnewman]]

Comment on attachment 660703 [details] [diff] [review]
Bootstrap backend for OS X, v4

This works. It ignored the hg 1.8.2 that I had on my path, installing 2.3 from HomeBrew, and installed autoconf-2.13.

Comment 16

[Gregory Szorc [:gps]]

https://hg.mozilla.org/mozilla-central/rev/e80ec1d9b1b1

Comment 17

[Gregory Szorc [:gps]]

Docs updated:

https://developer.mozilla.org/en-US/docs/Developer_Guide/Build_Instructions/Mac_OS_X_Prerequisites
https://developer.mozilla.org/en-US/docs/Simple_Firefox_build

Comment 18

[goxberry]

I was looking at the code you guys wrote for bootstrapping the development environment on OS X while Googling around to see if anyone had scripted a bootstrapper for Homebrew itself. I'm in the middle of writing a script that bootstraps a development stack on my OS X machine myself.

I love the Python code; I've been doing things in bash so far, but thought about using Python, and like your approach better. I think the script could be improved if you pulled the command line tools binaries off of Apple's web site, dumped them into SourceForge or something like that, and then downloaded and installed them as part of your bootstrapping script. Doing so might raise Apple's hackles, but should be legal due to the open source nature of the software, and would fully automate your script. At the same time, such an install should not interfere with an existing Xcode installation; it is the command line equivalent downloading the latest version of Xcode and installing the command line tools from within the GUI. (There could be issues if the command line tools downloaded did not match the version of an existing Xcode install, I suppose, but that scenario could be tested.)

If you guys think it's a reasonable idea, I'd be happy to contribute patches sometime over the next couple weeks, since I'm in the process of doing the same thing in my spare time anyway.

Comment 19

[Gregory Szorc [:gps]]

(In reply to goxberry from comment #18)
> I love the Python code; I've been doing things in bash so far, but thought
> about using Python, and like your approach better. I think the script could
> be improved if you pulled the command line tools binaries off of Apple's web
> site, dumped them into SourceForge or something like that, and then
> downloaded and installed them as part of your bootstrapping script. Doing so
> might raise Apple's hackles, but should be legal due to the open source
> nature of the software, and would fully automate your script. At the same
> time, such an install should not interfere with an existing Xcode
> installation; it is the command line equivalent downloading the latest
> version of Xcode and installing the command line tools from within the GUI.
> (There could be issues if the command line tools downloaded did not match
> the version of an existing Xcode install, I suppose, but that scenario could
> be tested.)

Fully automating things would be desirable. Avoiding the Xcode install would be nice.

That being said, I do worry about legal implications. We shouldn't have the bootstrapper bypass ToS acceptance, etc.

> If you guys think it's a reasonable idea, I'd be happy to contribute patches
> sometime over the next couple weeks, since I'm in the process of doing the
> same thing in my spare time anyway.

If there is a way to do things that doesn't upset lawyers, we'd likely be interested.

Also, commenting on old resolved bugs is generally frowned upon. If you have additional comments or patches, please file a new bug.

Comment 20

[Reuben Morais [:reuben]]

The command line tools don't include the OS X SDKs, so I don't think you can build m-c just from that. Might also be worth mentioning to the user in the bootstrap process that Mozilla's OS X builds use the 10.6 SDK, so if you just install latest Xcode, you won't be building the same code as the test and release builds do.