Closed Bug 774109 Opened 7 years ago Closed 7 years ago

bootstrap support for OS X

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla18

People

(Reporter: gps, Assigned: gps)

References

(Depends on 1 open bug)

Details

(Keywords: dev-doc-complete)

Attachments

(1 file, 3 obsolete files)

Mach should have support for bootstrapping the development environment on OS X.

1) Start with a fresh install of OS X.
2) Obtain source code for mozilla-central (either a source bundle or repository clone)
3) Run ./mach and all build prerequisites are downloaded, installed, and configured properly.

The goal is to make it super simple for people to build mozilla-central. Your terminal-literate grandmother should be able to do it.

This bug essentially involves automating https://developer.mozilla.org/En/Developer_Guide/Build_Instructions/Mac_OS_X_Prerequisites. Basically,

1) Install Xcode
2) Install Homebrew or MacPorts
3) Install required packages from Homebrew or MacPorts

Ideally, bootstrap support would not require a full copy of mozilla-central. We could provide a one-liner for someone to `curl | python` and magic would ensue. This could, of course, just talk to hg.mozilla.org over HTTPS (not using Mercurial) and pull down the latest .py files needed to start performing more magic. This minimal bootstrap is beyond the scope of this bug, but it should be considered during implementation.
Since we now build with Clang on OS X and since we have the SVN revision of LLVM/Clang we build with checked into the repository, it would be splendidly awesome if bootstrap also ensured the appropriate version of Clang were installed.

Bug 768879 tracks exposing the Clang builds Mozilla uses on its official infrastructure to the public internet. Bootstrap could theoretically fetch binaries from there.
Depends on: 768879
I just discovered B2G has a bootstrap script for OS X. We should consider stealing parts. https://github.com/mozilla-b2g/B2G/blob/master/scripts/bootstrap-mac.sh
Attached patch Bootstrap backend for OS X, v1 (obsolete) — Splinter Review
It bootstraps a vanilla OS X install to the point where it can build m-c.

I've only tested this on OS X 10.8. I could probably check this in without review b/c it isn't part of the build. But, I'd like just a cursory glance over the code.

Things like user prompting and support for additional OS X versions will come in follow-up bugs. Something is better than nothing.
Assignee: nobody → gps
Status: NEW → ASSIGNED
Attachment #660328 - Flags: review?(rnewman)
This isn't part of mach.
No longer blocks: machfeatures
Summary: mach: bootstrap support for OS X → bootstrap support for OS X
Whiteboard: [mach]
Doesn't apply to current m-c…

unable to find 'python/mozboot/mozboot/base.py' for patching
1 out of 1 hunks FAILED -- saving rejects to file python/mozboot/mozboot/base.py.rej
unable to find 'python/mozboot/mozboot/bootstrap.py' for patching
1 out of 1 hunks FAILED -- saving rejects to file python/mozboot/mozboot/bootstrap.py.rej
unable to find 'python/mozboot/mozboot/osx.py' for patching
1 out of 1 hunks FAILED -- saving rejects to file python/mozboot/mozboot/osx.py.rej

Missing dependency?
(In reply to Richard Newman [:rnewman] from comment #5)
> Doesn't apply to current m-c…
> 
> unable to find 'python/mozboot/mozboot/base.py' for patching
> 1 out of 1 hunks FAILED -- saving rejects to file
> python/mozboot/mozboot/base.py.rej
> unable to find 'python/mozboot/mozboot/bootstrap.py' for patching
> 1 out of 1 hunks FAILED -- saving rejects to file
> python/mozboot/mozboot/bootstrap.py.rej
> unable to find 'python/mozboot/mozboot/osx.py' for patching
> 1 out of 1 hunks FAILED -- saving rejects to file
> python/mozboot/mozboot/osx.py.rej
> 
> Missing dependency?

Those files are in inbound and haven't been merged to central yet.
Attached patch Bootstrap backend for OS X, v2 (obsolete) — Splinter Review
Performed some general code cleanup by refactoring the printed content to not be inline.

More importantly, I added a check for the OS X command line tools version. Previously, we allowed older, non-functional compilers in /usr/bin to exist. Now, we prompt to upgrade the Xcode command line tools version.

I've still only tested this on 10.8. But, I'll soon have my hands on a 10.5, 10.6, and 10.7 machine. If the instructions need refined on older OS versions, I'm fine with these being follow-up bugs.
Attachment #660328 - Attachment is obsolete: true
Attachment #660328 - Flags: review?(rnewman)
Attachment #660536 - Flags: review?(rnewman)
Attached patch Bootstrap backend for OS X, v3 (obsolete) — Splinter Review
I /think/ this will work with OS X 10.6. Although, I haven't actually tested. Apply on top of m-c (merge finally happened, yay) and let me know if it works.
Attachment #660536 - Attachment is obsolete: true
Attachment #660536 - Flags: review?(rnewman)
Attachment #660655 - Flags: review?(rnewman)
(In reply to Gregory Szorc [:gps] from comment #8)

> I /think/ this will work with OS X 10.6. Although, I haven't actually
> tested. Apply on top of m-c (merge finally happened, yay) and let me know if
> it works.

Running "python python/mozboot/mozboot/bootstrap.py" on 10.6.8 with current m-c:

---
Traceback (most recent call last):
  File "python/mozboot/mozboot/bootstrap.py", line 10, in <module>
    from mozboot.centos import CentOSBootstrapper
ImportError: No module named mozboot.centos
Run bin/bootstrap.py instead.
(In reply to Gregory Szorc [:gps] from comment #10)
> Run bin/bootstrap.py instead.

Traceback (most recent call last):
  File "python/mozboot/bin/bootstrap.py", line 134, in <module>
    sys.exit(main(sys.argv))
  File "python/mozboot/bin/bootstrap.py", line 125, in main
    dasboot.bootstrap()
  File "python/mozboot/mozboot/bootstrap.py", line 54, in bootstrap
    instance.install_system_packages()
  File "python/mozboot/mozboot/osx.py", line 15, in install_system_packages
    raise NotImplementedError('OS X bootstrap not yet implemented.')
NotImplementedError: OS X bootstrap not yet implemented.
Wait, I qpopped. I actually get:

Traceback (most recent call last):
  File "python/mozboot/bin/bootstrap.py", line 134, in <module>
    sys.exit(main(sys.argv))
  File "python/mozboot/bin/bootstrap.py", line 125, in main
    dasboot.bootstrap()
  File "python/mozboot/mozboot/bootstrap.py", line 68, in bootstrap
    instance.install_system_packages()
  File "python/mozboot/mozboot/osx.py", line 98, in install_system_packages
    self.ensure_xcode()
  File "python/mozboot/mozboot/osx.py", line 105, in ensure_xcode
    if not os.path.exsts('/Developer/Applications/Xcode.app'):
AttributeError: 'module' object has no attribute 'exsts'

Let me fix that typo…
With typo fixed I get:

Homebrew needs XQuartz installed in order to build some dependencies. Please
download and install XQuartz from the following URL:

    http://xquartz.macosforge.org/downloads/SL/XQuartz-2.7.3.dmg
When that has finished, please relaunch bootstrap.

This seems inaccurate, because I have a mozilla-central build running with a HomeBrew-installed clang *right now*. Correct?
Conditionally require XQuartz depending on whether Python is already installed.

This may not be perfect. I'm trying to sort out what Homebrew is doing...
Attachment #660655 - Attachment is obsolete: true
Attachment #660655 - Flags: review?(rnewman)
Attachment #660703 - Flags: review?(rnewman)
Comment on attachment 660703 [details] [diff] [review]
Bootstrap backend for OS X, v4

This works. It ignored the hg 1.8.2 that I had on my path, installing 2.3 from HomeBrew, and installed autoconf-2.13.
Attachment #660703 - Flags: review?(rnewman) → review+
https://hg.mozilla.org/mozilla-central/rev/e80ec1d9b1b1
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Keywords: dev-doc-needed
Resolution: --- → FIXED
Target Milestone: --- → mozilla18
Blocks: 838384
I was looking at the code you guys wrote for bootstrapping the development environment on OS X while Googling around to see if anyone had scripted a bootstrapper for Homebrew itself. I'm in the middle of writing a script that bootstraps a development stack on my OS X machine myself.

I love the Python code; I've been doing things in bash so far, but thought about using Python, and like your approach better. I think the script could be improved if you pulled the command line tools binaries off of Apple's web site, dumped them into SourceForge or something like that, and then downloaded and installed them as part of your bootstrapping script. Doing so might raise Apple's hackles, but should be legal due to the open source nature of the software, and would fully automate your script. At the same time, such an install should not interfere with an existing Xcode installation; it is the command line equivalent downloading the latest version of Xcode and installing the command line tools from within the GUI. (There could be issues if the command line tools downloaded did not match the version of an existing Xcode install, I suppose, but that scenario could be tested.)

If you guys think it's a reasonable idea, I'd be happy to contribute patches sometime over the next couple weeks, since I'm in the process of doing the same thing in my spare time anyway.
(In reply to goxberry from comment #18)
> I love the Python code; I've been doing things in bash so far, but thought
> about using Python, and like your approach better. I think the script could
> be improved if you pulled the command line tools binaries off of Apple's web
> site, dumped them into SourceForge or something like that, and then
> downloaded and installed them as part of your bootstrapping script. Doing so
> might raise Apple's hackles, but should be legal due to the open source
> nature of the software, and would fully automate your script. At the same
> time, such an install should not interfere with an existing Xcode
> installation; it is the command line equivalent downloading the latest
> version of Xcode and installing the command line tools from within the GUI.
> (There could be issues if the command line tools downloaded did not match
> the version of an existing Xcode install, I suppose, but that scenario could
> be tested.)

Fully automating things would be desirable. Avoiding the Xcode install would be nice.

That being said, I do worry about legal implications. We shouldn't have the bootstrapper bypass ToS acceptance, etc.

> If you guys think it's a reasonable idea, I'd be happy to contribute patches
> sometime over the next couple weeks, since I'm in the process of doing the
> same thing in my spare time anyway.

If there is a way to do things that doesn't upset lawyers, we'd likely be interested.

Also, commenting on old resolved bugs is generally frowned upon. If you have additional comments or patches, please file a new bug.
The command line tools don't include the OS X SDKs, so I don't think you can build m-c just from that. Might also be worth mentioning to the user in the bootstrap process that Mozilla's OS X builds use the 10.6 SDK, so if you just install latest Xcode, you won't be building the same code as the test and release builds do.
Depends on: 952395
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.