774112 - mach: bootstrap support for Linux

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

+++ This bug was initially created as a clone of Bug #774111 +++

Mach should have support for bootstrapping the development environment on systems running the yum package manager (RedHat Linux, CentOS, Fedora, others).

1) Start with a fresh/minimal install of the OS.
2) Obtain source code for mozilla-central (either a source bundle or repository clone)
3) Run ./mach and all build prerequisites are installed automatically.

The goal is to make it super simple for people to build mozilla-central. Your terminal-literate grandmother should be able to do it.

This bug essentially involves automating https://developer.mozilla.org/En/Developer_Guide/Build_Instructions/Linux_Prerequisites. Basically, query yum and offer to install missing packages.

Ideally, bootstrap support would not require a full copy of mozilla-central. We could provide a one-liner for someone to `curl | python` and magic would ensue. This could just talk to hg.mozilla.org over HTTPS (not using Mercurial) and pull down the latest .py files needed to start performing more magic. This minimal bootstrap is beyond the scope of this bug, but it should be considered during implementation.

Comment 1

[kmm [:kmm ]]

Attachment: very very rough draft of a working script for CentOS/Fedora 16

Tested on a Fedora 16 VM successfully, CentOS Live CD or base installs may give trouble with dependencies even after the yum installs; working on that now

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 656958 [details]
very very rough draft of a working script for CentOS/Fedora 16

When requesting review, use the '?'. '+' is granted review.

Since this is an early draft, I'm switching it to the feedback field. Review is typically reserved for things that are capable of landing in the tree.

Comment 3

[kmm [:kmm ]]

Attachment: unified dependency installer including both apt and yum support

unified dependency installer with apt and yum support; tested on CentOS (not working yet), Fedora (should work on 16 at least), Ubuntu/Mint (worked test beds)

Comment 4

[Gregory Szorc [:gps]]

Attachment: mozboot core

I coded up a basic skeleton for the Python package. This patch should be applied inside mozilla-central. Or, you can patch -p2 it to strip the leading python/ directory.

It doesn't do much. It should be pretty compatible with what you coded up.

Comment 5

[kmm [:kmm ]]

Attachment: bootstrapper with working CentOS 6 dependency installer

tested on 32 and 64 bit CentOS 6.3 Live CD; doesn't install libstdc++-static, but appears to be able to build; have not completed a full build with it but it passes all the checks at the start of the build process; leaving other files intact

Comment 7

[Gregory Szorc [:gps]]

Comment on attachment 658163 [details]
bootstrapper with working CentOS 6 dependency installer

First, this is looking really good. This bug excites me.

Things to change:

* Nuke the |make -f client.mk| bit. Bootstrap doesn't need to actually build the tree, just ensure the system is in the proper state to build the tree.

* Along the same vein, Mercurial cloning should at least be factored out into a common routine. Theoretically, Mercurial cloning should be identical across all systems. So, this code can be shared. I would also be fine if the initial landing of this bug did not contain support for cloning the repository. This is because IMO setting up Mercurial should be more than just cloning the repo (we also want to write out a config file that uses Mozilla's preferred settings).

* If Mercurial cloning makes it in, we need more flexibility here. Users should be able to choose the destination directory (currently it goes in $CWD/mozilla-central). It should also be optional.

* I have mixed feelings about using '-y'. On one hand, I think it is proper to prompt users before doing something. On the other, concerned users can look at the source code before they run something.

* You don't need a "pass" after each indented block. pass is only needed if the block is empty.

* That trick of updating self.__class__ is nifty. I didn't even know you could switch the class of an object after its creation! I'm pretty sure this is considered bad Python, however. This will probably go away when you merge things on top of the mozboot core. Just keep it in mind when writing Python in the future.

Comment 8

[Gregory Szorc [:gps]]

Attachment: mozboot core, v2

I fixed the code. To run this, apply the patch inside a cloned mozilla-central repository. Then:

  cd python/mozboot
  PYTHONPATH=. python bin/bootstrap.py

Comment 9

[Gregory Szorc [:gps]]

Attachment: mozboot core, v3

Added OS X skeleton. Found a few more bugs as part of this. This should actually work and should be ready for you to merge things in.

Comment 10

[Gregory Szorc [:gps]]

Attachment: mozboot core, v4

I should add files before creating patches. Looks like I picked the wrong week to quit sniffing glue.

Comment 11

[kmm [:kmm ]]

Attachment: haven't got setup.py working yet

run "python ./bootstrap1.py" with all files unzipped to same directory
will modify sys.path to allow for imports as intended; just wanted to get things up and running quickly without adding too much code; CentOS 6.3 only has Python 2.6.6 by default, so commented out argparse

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 657440 [details]
unified dependency installer including both apt and yum support

The latest zip file makes this irrelevant.

Comment 13

[Gregory Szorc [:gps]]

Attachment: Part 1: bootstrap core, v1

Core classes to perform bootstrapping. This isn't the best code in the world. It doesn't need to be. Something (that works) is better than nothing.

Also, this is just the base classes. kmm will be providing actual code that builds on top of this.

Comment 14

[Gregory Szorc [:gps]]

kmm: I don't mind cleaning up your patch for you. Before I check things in, I'll need to know what name and email address to affiliate with the checkin. If you want to use "kmm" for you name, we can do that.

Comment 15

[Jeff Hammel]

Comment on attachment 660237 [details] [diff] [review]
Part 1: bootstrap core, v1

+#!python

#!/usr/bin/env python plz

+        raise Exception('Not sure how to handle repo type: %s' % repo_type)

I'd use NotImplementedError if you anticipate adding more

+        include = os.path.normpath(os.path.join(os.path.dirname(__file__),
+            os.path.pardir))

indentation

+            files = fetch_files(repo_url, repo_type)

I'd only call this if these aren't None, None

+    options, leftover = parser.parse_args(args)

I'd be inclined to err out/print help if leftover 

+    def __init__(self):
+        pass

Please don't do this

+    def install_system_packages(self):
+        raise NotImplemented('%s must implement install_system_packages()' %
+            __name__)

indentation

wfm, with nits addressed

Comment 16

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/544c405fb456
https://hg.mozilla.org/integration/mozilla-inbound/rev/ac75030f29a7

I cleaned up kmm's patch and checked it in \o/

If we any follow-up work, we can do that on a new bug.

We'll need to update the docs on MDN to reflect the new world for bootstrapping:

  curl https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py | python -

Thank you again, kmm, for contributing the code and taking the time to verify bootstrap on all the different OSs works. Scores of Firefox developers will have you to thank.

Comment 17

[Gregory Szorc [:gps]]

Fixed an oops I missed in the review.

https://hg.mozilla.org/integration/mozilla-inbound/rev/58338ee54795

Comment 18

[Gregory Szorc [:gps]]

Updated the following to reference the new one-line bootstrap method:

https://developer.mozilla.org/en-US/docs/Simple_Firefox_build
https://developer.mozilla.org/en-US/docs/Developer_Guide/Build_Instructions/Linux_Prerequisites

Comment 19

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/544c405fb456
https://hg.mozilla.org/mozilla-central/rev/ac75030f29a7
https://hg.mozilla.org/mozilla-central/rev/58338ee54795

Comment 20

[Gregory Szorc [:gps]]

Comment on attachment 659336 [details]
haven't got setup.py working yet

Patch is stale.