Last Comment Bug 774112 - mach: bootstrap support for Linux
: mach: bootstrap support for Linux
Status: RESOLVED FIXED
: dev-doc-complete
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: All All
: -- normal (vote)
: mozilla18
Assigned To: kmm [:kmm ]
:
Mentors:
https://etherpad.mozilla.org/devenv-b...
: 774111 (view as bug list)
Depends on: 790995
Blocks: machfeatures
  Show dependency treegraph
 
Reported: 2012-07-15 13:43 PDT by Gregory Szorc [:gps]
Modified: 2012-11-19 09:49 PST (History)
5 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
very very rough draft of a working script for CentOS/Fedora 16 (1.62 KB, text/plain)
2012-08-30 11:15 PDT, kmm [:kmm ]
no flags Details
unified dependency installer including both apt and yum support (3.98 KB, text/plain)
2012-08-31 14:02 PDT, kmm [:kmm ]
no flags Details
mozboot core (4.53 KB, patch)
2012-08-31 14:22 PDT, Gregory Szorc [:gps]
no flags Details | Diff | Splinter Review
bootstrapper with working CentOS 6 dependency installer (4.49 KB, text/plain)
2012-09-04 11:28 PDT, kmm [:kmm ]
gps: feedback+
Details
mozboot core, v2 (4.52 KB, patch)
2012-09-05 14:58 PDT, Gregory Szorc [:gps]
no flags Details | Diff | Splinter Review
mozboot core, v3 (4.90 KB, patch)
2012-09-05 15:15 PDT, Gregory Szorc [:gps]
no flags Details | Diff | Splinter Review
mozboot core, v4 (5.64 KB, patch)
2012-09-05 15:27 PDT, Gregory Szorc [:gps]
no flags Details | Diff | Splinter Review
haven't got setup.py working yet (5.75 KB, application/zip)
2012-09-07 13:16 PDT, kmm [:kmm ]
no flags Details
Part 1: bootstrap core, v1 (9.69 KB, patch)
2012-09-11 15:38 PDT, Gregory Szorc [:gps]
k0scist: review+
Details | Diff | Splinter Review

Description Gregory Szorc [:gps] 2012-07-15 13:43:46 PDT
+++ This bug was initially created as a clone of Bug #774111 +++

Mach should have support for bootstrapping the development environment on systems running the yum package manager (RedHat Linux, CentOS, Fedora, others).

1) Start with a fresh/minimal install of the OS.
2) Obtain source code for mozilla-central (either a source bundle or repository clone)
3) Run ./mach and all build prerequisites are installed automatically.

The goal is to make it super simple for people to build mozilla-central. Your terminal-literate grandmother should be able to do it.

This bug essentially involves automating https://developer.mozilla.org/En/Developer_Guide/Build_Instructions/Linux_Prerequisites. Basically, query yum and offer to install missing packages.

Ideally, bootstrap support would not require a full copy of mozilla-central. We could provide a one-liner for someone to `curl | python` and magic would ensue. This could just talk to hg.mozilla.org over HTTPS (not using Mercurial) and pull down the latest .py files needed to start performing more magic. This minimal bootstrap is beyond the scope of this bug, but it should be considered during implementation.
Comment 1 kmm [:kmm ] 2012-08-30 11:15:21 PDT
Created attachment 656958 [details]
very very rough draft of a working script for CentOS/Fedora 16

Tested on a Fedora 16 VM successfully, CentOS Live CD or base installs may give trouble with dependencies even after the yum installs; working on that now
Comment 2 Gregory Szorc [:gps] 2012-08-30 11:17:05 PDT
Comment on attachment 656958 [details]
very very rough draft of a working script for CentOS/Fedora 16

When requesting review, use the '?'. '+' is granted review.

Since this is an early draft, I'm switching it to the feedback field. Review is typically reserved for things that are capable of landing in the tree.
Comment 3 kmm [:kmm ] 2012-08-31 14:02:18 PDT
Created attachment 657440 [details]
unified dependency installer including both apt and yum support

unified dependency installer with apt and yum support; tested on CentOS (not working yet), Fedora (should work on 16 at least), Ubuntu/Mint (worked test beds)
Comment 4 Gregory Szorc [:gps] 2012-08-31 14:22:03 PDT
Created attachment 657444 [details] [diff] [review]
mozboot core

I coded up a basic skeleton for the Python package. This patch should be applied inside mozilla-central. Or, you can patch -p2 it to strip the leading python/ directory.

It doesn't do much. It should be pretty compatible with what you coded up.
Comment 5 kmm [:kmm ] 2012-09-04 11:28:50 PDT
Created attachment 658163 [details]
bootstrapper with working CentOS 6 dependency installer

tested on 32 and 64 bit CentOS 6.3 Live CD; doesn't install libstdc++-static, but appears to be able to build; have not completed a full build with it but it passes all the checks at the start of the build process; leaving other files intact
Comment 6 Gregory Szorc [:gps] 2012-09-05 13:59:29 PDT
*** Bug 774111 has been marked as a duplicate of this bug. ***
Comment 7 Gregory Szorc [:gps] 2012-09-05 14:28:52 PDT
Comment on attachment 658163 [details]
bootstrapper with working CentOS 6 dependency installer

First, this is looking really good. This bug excites me.

Things to change:

* Nuke the |make -f client.mk| bit. Bootstrap doesn't need to actually build the tree, just ensure the system is in the proper state to build the tree.

* Along the same vein, Mercurial cloning should at least be factored out into a common routine. Theoretically, Mercurial cloning should be identical across all systems. So, this code can be shared. I would also be fine if the initial landing of this bug did not contain support for cloning the repository. This is because IMO setting up Mercurial should be more than just cloning the repo (we also want to write out a config file that uses Mozilla's preferred settings).

* If Mercurial cloning makes it in, we need more flexibility here. Users should be able to choose the destination directory (currently it goes in $CWD/mozilla-central). It should also be optional.

* I have mixed feelings about using '-y'. On one hand, I think it is proper to prompt users before doing something. On the other, concerned users can look at the source code before they run something.

* You don't need a "pass" after each indented block. pass is only needed if the block is empty.

* That trick of updating self.__class__ is nifty. I didn't even know you could switch the class of an object after its creation! I'm pretty sure this is considered bad Python, however. This will probably go away when you merge things on top of the mozboot core. Just keep it in mind when writing Python in the future.
Comment 8 Gregory Szorc [:gps] 2012-09-05 14:58:26 PDT
Created attachment 658637 [details] [diff] [review]
mozboot core, v2

I fixed the code. To run this, apply the patch inside a cloned mozilla-central repository. Then:

  cd python/mozboot
  PYTHONPATH=. python bin/bootstrap.py
Comment 9 Gregory Szorc [:gps] 2012-09-05 15:15:11 PDT
Created attachment 658652 [details] [diff] [review]
mozboot core, v3

Added OS X skeleton. Found a few more bugs as part of this. This should actually work and should be ready for you to merge things in.
Comment 10 Gregory Szorc [:gps] 2012-09-05 15:27:15 PDT
Created attachment 658663 [details] [diff] [review]
mozboot core, v4

I should add files before creating patches. Looks like I picked the wrong week to quit sniffing glue.
Comment 11 kmm [:kmm ] 2012-09-07 13:16:58 PDT
Created attachment 659336 [details]
haven't got setup.py working yet

run "python ./bootstrap1.py" with all files unzipped to same directory
will modify sys.path to allow for imports as intended; just wanted to get things up and running quickly without adding too much code; CentOS 6.3 only has Python 2.6.6 by default, so commented out argparse
Comment 12 Gregory Szorc [:gps] 2012-09-07 13:33:31 PDT
Comment on attachment 657440 [details]
unified dependency installer including both apt and yum support

The latest zip file makes this irrelevant.
Comment 13 Gregory Szorc [:gps] 2012-09-11 15:38:01 PDT
Created attachment 660237 [details] [diff] [review]
Part 1: bootstrap core, v1

Core classes to perform bootstrapping. This isn't the best code in the world. It doesn't need to be. Something (that works) is better than nothing.

Also, this is just the base classes. kmm will be providing actual code that builds on top of this.
Comment 14 Gregory Szorc [:gps] 2012-09-11 15:43:52 PDT
kmm: I don't mind cleaning up your patch for you. Before I check things in, I'll need to know what name and email address to affiliate with the checkin. If you want to use "kmm" for you name, we can do that.
Comment 15 Jeff Hammel 2012-09-11 15:47:03 PDT
Comment on attachment 660237 [details] [diff] [review]
Part 1: bootstrap core, v1

+#!python

#!/usr/bin/env python plz

+        raise Exception('Not sure how to handle repo type: %s' % repo_type)

I'd use NotImplementedError if you anticipate adding more

+        include = os.path.normpath(os.path.join(os.path.dirname(__file__),
+            os.path.pardir))

indentation

+            files = fetch_files(repo_url, repo_type)

I'd only call this if these aren't None, None

+    options, leftover = parser.parse_args(args)

I'd be inclined to err out/print help if leftover 

+    def __init__(self):
+        pass

Please don't do this

+    def install_system_packages(self):
+        raise NotImplemented('%s must implement install_system_packages()' %
+            __name__)

indentation

wfm, with nits addressed
Comment 16 Gregory Szorc [:gps] 2012-09-11 16:38:11 PDT
https://hg.mozilla.org/integration/mozilla-inbound/rev/544c405fb456
https://hg.mozilla.org/integration/mozilla-inbound/rev/ac75030f29a7

I cleaned up kmm's patch and checked it in \o/

If we any follow-up work, we can do that on a new bug.

We'll need to update the docs on MDN to reflect the new world for bootstrapping:

  curl https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py | python -

Thank you again, kmm, for contributing the code and taking the time to verify bootstrap on all the different OSs works. Scores of Firefox developers will have you to thank.
Comment 17 Gregory Szorc [:gps] 2012-09-11 16:43:58 PDT
Fixed an oops I missed in the review.

https://hg.mozilla.org/integration/mozilla-inbound/rev/58338ee54795
Comment 18 Gregory Szorc [:gps] 2012-09-11 18:22:26 PDT
Updated the following to reference the new one-line bootstrap method:

https://developer.mozilla.org/en-US/docs/Simple_Firefox_build
https://developer.mozilla.org/en-US/docs/Developer_Guide/Build_Instructions/Linux_Prerequisites
Comment 20 Gregory Szorc [:gps] 2012-11-19 09:49:48 PST
Comment on attachment 659336 [details]
haven't got setup.py working yet

Patch is stale.

Note You need to log in before you can comment on or make changes to this bug.