803990 - Bootstrap support for Gentoo

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Freddie F. Haddad]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.7) Gecko/20100101 Firefox/10.0.7
Build ID: 20120929235736

Steps to reproduce:

wget https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py && python bootstrap.py


Actual results:

--2012-10-21 09:06:29--  https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py
Resolving hg.mozilla.org... 63.245.215.25
Connecting to hg.mozilla.org|63.245.215.25|:443... connected.
HTTP request sent, awaiting response... 200 Script output follows
Length: 4300 (4.2K) [text/x-python]
Saving to: `bootstrap.py'

100%[=====================================================================================================================>] 4,300       --.-K/s   in 0.001s  

2012-10-21 09:06:30 (7.68 MB/s) - `bootstrap.py' saved [4300/4300]

Traceback (most recent call last):
  File "bootstrap.py", line 137, in <module>
    sys.exit(main(sys.argv))
  File "bootstrap.py", line 128, in main
    dasboot.bootstrap()
  File "/tmp/tmptX_6wg/mozboot/bootstrap.py", line 50, in bootstrap
    for path in REPOSITORY_PATHS:
NotImplementedError: Bootstrap support for this Linux distro not yet available.


Expected results:

All the tools and source code for FireFox should have downloaded.

--
In Gentoo, I was able to switch to Python2.7 using the eselect tool. I think all the needed tools to build firefox are available or are already installed.

Comment 1

[Freddie F. Haddad]

In Gentoo, I was able to switch to Python2.7 using the eselect tool. I think all the needed tools to build firefox are available or are already installed which means Gentoo should support the bootstrap method of retrieving the source.

Comment 2

[Gregory Szorc [:gps]]

This should be pretty simple to implement. We just need a Gentoo handler.

Relevant files in mozilla-central are located in python/mozboot. We should just need to emerge the appropriate packages.

Comment 3

[Freddie F. Haddad]

I'm new to mozilla development, but if this is something I can help with, I would be more than happy. I would just need some guidance on what needs to be done. I use Gentoo on my machine.

Comment 4

[Gregory Szorc [:gps]]

The bug is yours for the taking!

Clone mozilla-central:

  $ hg clone https://hg.mozilla.org/mozilla-central/

or

  $ git clone git://github.com/mozilla/mozilla-central.git

Relevant files are in python/mozboot/mozboot in that repository. You'll want to create a gentoo.py (similar to ubuntu.py, mint.py, etc). The BaseBootstrapper class contains a run_as_root method (see base.py). You'll probably use that to invoke the necessary emerge commands.

You'll need to hook the new Python module and class into bootstrap.py (https://hg.mozilla.org/mozilla-central/file/1c3e4cb1f754/python/mozboot/mozboot/bootstrap.py#l38). You'll also need to add gentoo.py to the bin/bootstrap.py file at https://hg.mozilla.org/mozilla-central/file/1c3e4cb1f754/python/mozboot/bin/bootstrap.py#l30.

If you are on the irc.mozilla.org IRC server, I'm gps. Or, you can continue to ask questions on this bug.

When you have a patch, just upload it to this bug. You may also want to peruse https://developer.mozilla.org/en-US/docs/Developer_Guide

Comment 5

[Freddie F. Haddad]

I'm in the process of building gentoo.py and I had a general question. Since most Gentoo packages are built and compiled from the source code, is it safe to assume that many of the needed development libraries are already included with Gentoo? They don't all appear in the portage tree, but I can successfully emerge firefox and thunderbird and compile them.

Comment 6

[Gregory Szorc [:gps]]

It's been a while since I've used Gentoo. But, I believe there is some option to emerge that basically says "install all packages required to build this package without actually building this package" (i.e. install the development dependencies). If you use that with the built-in Firefox package, that should be sufficient (at least for initial Gentoo support).

Comment 7

[Freddie F. Haddad]

Attachment: python/mozboot/bin

gentoo.py insertion into REPOSITORY_PATHS

Comment 8

[Freddie F. Haddad]

Attachment: python/mozboot/mozboot

GentooBootstrapper

Comment 9

[Freddie F. Haddad]

Attachment: python/mozboot/mozboot

Gentoo emerge flag should install/build all dependencies for Firefox, but not Firefox itself.

Comment 10

[Freddie F. Haddad]

I uploaded all the files that I was able to determine needed modifications for Gentoo. I can update the website as well if I did everything correctly.

Comment 11

[Freddie F. Haddad]

Comment on attachment 673753 [details]
python/mozboot/mozboot

># This Source Code Form is subject to the terms of the Mozilla Public
># License, v. 2.0. If a copy of the MPL was not distributed with this file,
># You can obtain one at http://mozilla.org/MPL/2.0/.
>
>import os
>
>from mozboot.base import BaseBootstrapper
>
>class GentooBootstrapper(BaseBootstrapper):
>    def __init__(self, version):
>        BaseBootstrapper.__init__(self)
>
>    def install_system_packages(self):
>        # -aoqv is short flags for  --ask --onlydeps --quiet --verbose
>        # displays USE flags and prompts for confirmation before installing.
>        self.run_as_root(['emerge', '-aoqv',
>            'firefox')

Comment 12

[Freddie F. Haddad]

Attachment: python/mozboot/mozboot

displays was spelled incorrectly in the comment

Comment 13

[Freddie F. Haddad]

An important note, Gentoo comes with Python2 and Python3 so users must issue the commands:

sudo eselect python list
sudo eselect python set x : x being pyton2

Since the bootstrap needs python2 and emerge needs python3, I'm not sure what exactly is going to happen. :)

Comment 14

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #6)
> If you use that with the built-in Firefox package, that should be sufficient (at least for initial Gentoo support).

It's an unreliable way to get firefox dependencies, like apt-get build-dep firefox on ubuntu. Especially when we add new ones.

Comment 15

[Gregory Szorc [:gps]]

(In reply to Mike Hommey [:glandium] from comment #14)
> (In reply to Gregory Szorc [:gps] from comment #6)
> > If you use that with the built-in Firefox package, that should be sufficient (at least for initial Gentoo support).
> 
> It's an unreliable way to get firefox dependencies, like apt-get build-dep
> firefox on ubuntu. Especially when we add new ones.

But it's better than nothing, which is what we have now.

Comment 16

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #15)
> But it's better than nothing, which is what we have now.

And it's even better to replace nothing with something that won't break randomly.

Comment 17

[Freddie F. Haddad]

(In reply to Mike Hommey [:glandium] from comment #14)
> (In reply to Gregory Szorc [:gps] from comment #6)
> > If you use that with the built-in Firefox package, that should be sufficient (at least for initial Gentoo support).
> 
> It's an unreliable way to get firefox dependencies, like apt-get build-dep
> firefox on ubuntu. Especially when we add new ones.

I thought about that, and there is an alternative solution. With Layman and Portage, you can get the "latest" development repo by installing a -9999 live ebuild. This is according to one of the developers that Gentoo, but I'm pretty sure that would be a snapshot of some recent build.

Comment 18

[Mike Hommey [:glandium]]

(In reply to Freddie F. Haddad from comment #17)
> I thought about that, and there is an alternative solution. With Layman and
> Portage, you can get the "latest" development repo by installing a -9999
> live ebuild. This is according to one of the developers that Gentoo, but I'm
> pretty sure that would be a snapshot of some recent build.

That would still rely on the fact that the build dependencies to build a firefox package on gentoo are what you need to build mozilla-central on gentoo, which, chances are, is not true. Here's a simple example: most linux distros have been building Firefox with gio support instead of gnomevfs for a while. Which means installing the build dependencies for firefox usually doesn't pull the gnomevfs development files, and the resulting mozilla-central build doesn't match the features that we were building by default on mozilla builds. It turns out mozilla builds do use gio now, instead of gnomevfs, but you get the idea.

Comment 19

[Freddie F. Haddad]

> That would still rely on the fact that the build dependencies to build a
> firefox package on gentoo are what you need to build mozilla-central on
> gentoo, which, chances are, is not true. Here's a simple example: most linux
> distros have been building Firefox with gio support instead of gnomevfs for
> a while. Which means installing the build dependencies for firefox usually
> doesn't pull the gnomevfs development files, and the resulting
> mozilla-central build doesn't match the features that we were building by
> default on mozilla builds. It turns out mozilla builds do use gio now,
> instead of gnomevfs, but you get the idea.

It sounds like using the 'firefox' build dependency in Gentoo will bring in *most* of the needed development libraries, and then *additional packages* may or may not be required. When this situation happens, bootstrap support would need to be updated. Notes about this could be posted at https://developer.mozilla.org/en-US/docs/Simple_Firefox_build. But I do believe that the current Firefox dependency tree in Gentoo meets all the *current* requirements.

Comment 20

[Freddie F. Haddad]

Anyway, I'm not sure how else to provide bootstrap support which was the goal of this bug. Even if it's only partial, it's still helpful to get started.

Comment 21

[Gregory Szorc [:gps]]

I think asking portage for the known dependencies is a good first step. We can then supplement that with gio and other one-offs, as necessary. The bootstrap script is very community driven. So, as we require new packages to build the tree, community members can step in and contribute additions to the bootstrapper.

Comment 22

[Gregory Szorc [:gps]]

Attachment: Gentoo support

Replacing the individual files as a patch (that's how we look at code).

I also cleaned things up a little (basically expanded the single letter arguments to the full form so it can be self-documenting).

This all looks pretty good to me.

We need to install Mercurial and Git. Is this one of the dependencies on Gentoo or should it be installed manually?

Why --quiet and --verbose? Those seem at odds with each other.

Comment 23

[Freddie F. Haddad]

(In reply to Gregory Szorc [:gps] from comment #22)
> Created attachment 673944 [details] [diff] [review]
> Gentoo support
> 
> Replacing the individual files as a patch (that's how we look at code).
> 
> I also cleaned things up a little (basically expanded the single letter
> arguments to the full form so it can be self-documenting).
> 
> This all looks pretty good to me.
> 
> We need to install Mercurial and Git. Is this one of the dependencies on
> Gentoo or should it be installed manually?
> 
> Why --quiet and --verbose? Those seem at odds with each other.

Mercurial and git need to be installed manually, perhaps, those should be added to the list.

Regarding the --quiet --verbose, it's indeed strange, but it hides all the compile-time output and only shows some output such as the USE flags that will be compiled for each package.

Comment 24

[Freddie F. Haddad]

* dependency graph for www-client/firefox-16.0.1
 `--  www-client/firefox-16.0.1  [~amd64 keyword] 
   `--  sys-devel/binutils-2.22-r1  (>=sys-devel/binutils-2.16.1) amd64 
   `--  dev-libs/nss-3.13.6  (>=dev-libs/nss-3.13.6) amd64 
   `--  dev-libs/nspr-4.9.2  (>=dev-libs/nspr-4.9.2) amd64 
   `--  dev-libs/glib-2.32.4-r1  (>=dev-libs/glib-2.26) amd64 
   `--  media-libs/mesa-8.0.4-r1  (>=media-libs/mesa-7.10) amd64 
   `--  media-libs/libpng-1.5.13-r1  (>=media-libs/libpng-1.5.9) amd64  [apng]
   `--  virtual/libffi-3.0.11  (virtual/libffi) amd64 
   `--  media-libs/gstreamer-0.10.35  (>=media-libs/gstreamer-0.10.33) amd64 
   `--  media-libs/gst-plugins-base-0.10.35  (>=media-libs/gst-plugins-base-0.10.33) amd64 
   `--  dev-db/sqlite-3.7.13  (>=dev-db/sqlite-3.7.12.1) amd64  [fts3 secure-delete threadsafe unlock-notify debug=]
   `--  media-libs/libvpx-1.0.0  (>=media-libs/libvpx-1.0.0) amd64 
   `--  media-libs/alsa-lib-1.0.25-r1  (media-libs/alsa-lib) amd64 
   `--  sec-policy/selinux-mozilla-9999  (sec-policy/selinux-mozilla) M[package.mask] 
   `--  virtual/pkgconfig-0  (virtual/pkgconfig) amd64 
   `--  dev-lang/python-2.7.3-r2  (=dev-lang/python-2*) amd64  [sqlite]
   `--  sys-devel/gcc-4.5.4  (>=sys-devel/gcc-4.5) amd64 
   `--  dev-lang/yasm-1.2.0  (>=dev-lang/yasm-1.1) amd64 
   `--  virtual/opengl-7.0  (virtual/opengl) amd64 
   `--  sys-apps/sed-4.2.1-r1  (>=sys-apps/sed-4) amd64 
   `--  x11-libs/libXrender-0.9.7  (x11-libs/libXrender) amd64 
   `--  x11-libs/libXt-1.1.3  (x11-libs/libXt) amd64 
   `--  x11-libs/libXmu-1.1.1  (x11-libs/libXmu) amd64 
   `--  sys-libs/zlib-1.2.7  (>=sys-libs/zlib-1.1.4) amd64 
   `--  app-arch/zip-3.0-r1  (app-arch/zip) amd64 
   `--  app-arch/unzip-6.0-r3  (app-arch/unzip) amd64 
   `--  app-text/hunspell-1.3.2-r3  (>=app-text/hunspell-1.2) amd64 
   `--  dev-libs/expat-2.1.0-r2  (dev-libs/expat) amd64 
   `--  dev-libs/libIDL-0.8.14  (>=dev-libs/libIDL-0.8.0) amd64 
   `--  dev-libs/libevent-2.0.19  (>=dev-libs/libevent-1.4.7) amd64 
   `--  x11-libs/cairo-1.10.2-r2  (>=x11-libs/cairo-1.8) amd64  [X]
   `--  x11-libs/gtk+-2.24.12  (>=x11-libs/gtk+-2.8.6) amd64 
   `--  x11-libs/pango-1.30.1  (>=x11-libs/pango-1.10.1) amd64  [X]
   `--  virtual/jpeg-0  (virtual/jpeg) amd64 
   `--  virtual/freedesktop-icon-theme-0  (virtual/freedesktop-icon-theme) amd64 
   `--  dev-libs/dbus-glib-0.100  (>=dev-libs/dbus-glib-0.72) amd64 
   `--  x11-libs/libnotify-0.7.5  (>=x11-libs/libnotify-0.4) amd64 
   `--  x11-libs/startup-notification-0.12  (>=x11-libs/startup-notification-0.8) amd64 
   `--  net-wireless/wireless-tools-29  (net-wireless/wireless-tools) amd64 
   `--  sys-devel/automake-1.11.6  (>=sys-devel/automake-1.11.1) amd64 
   `--  sys-devel/automake-1.12.4  (>=sys-devel/automake-1.12) [~amd64 keyword] 
   `--  sys-devel/autoconf-2.13  (=sys-devel/autoconf-2.1*) amd64 
   `--  sys-devel/libtool-2.4-r1  (sys-devel/libtool) amd64 
   `--  x11-base/xorg-server-1.12.2  (x11-base/xorg-server) amd64  [xvfb]
   `--  x11-apps/xhost-1.0.5  (x11-apps/xhost) amd64

Comment 25

[Gregory Szorc [:gps]]

Freddie: would you mind adding the Mercurial and Git packages to the list? (I'm not sure what they are because I don't run Gentoo).

If not, I could probably look it up and get this checked in for you.

Comment 26

[Freddie F. Haddad]

Gregory: Forgive me, I'm still learning, but when I see how the patches were submitted, it appears each modified file was not uploaded, but rather one file showing all the changes. I'm not sure exactly how to do that. I would like to learn the proper way to submit changes and so I can add those additional packages you requested. What would need to be added to the gentoo.py file is the following:

self.run_as_root(['emerge', '--ask', '--quiet', '--verbose', 'git', 'mercurial'])

I'm pretty sure it needs to be a second method invocation because we must remove the --onlydeps flag.

Comment 27

[Gregory Szorc [:gps]]

The concept behind the file I uploaded is called "patches" or "diffs". You may want to read https://developer.mozilla.org/en-US/docs/Mercurial_FAQ, specifically https://developer.mozilla.org/en-US/docs/Mercurial_FAQ#How_can_I_diff_and_patch_files.3F and below.

Assuming your current directory is your local Mercurial clone of mozilla-central, try running the following in your shell:

  # This will create a Mercurial patch queue. If this fails, see https://developer.mozilla.org/en-US/docs/Mercurial_FAQ#How_do_I_check_stuff_in.3F

  $ hg init --mq

  $ hg qimport -n mozboot-gentoo https://bug803990.bugzilla.mozilla.org/attachment.cgi?id=673944
  $ hg qpush mozboot-gentoo

  # Make your changes to files.
  $ hg st
  # Will show you all the files that have been changed.
  $ hg qref

  # Now attach the .hg/patches/mozboot-gentoo file to this bug!

If you get stuck, join #introduction in Mozilla's IRC network (https://wiki.mozilla.org/IRC) or ask on this bug and I'll help you.

Comment 28

[Freddie F. Haddad]

OK, I will submit this tonight, I am at school still. Thank you very much.

Comment 29

[Gregory Szorc [:gps]]

Freddie: it's been a few days and I thought I'd check in to see how things are going. If you need any help, don't hesitate to ping me.

Comment 30

[Freddie F. Haddad]

Gregory: Sorry, I have midterms all this week and I was having problems getting mercurial working. So I am stepping away until the Monday just to get through midterms and focus my energy on that. Then I will be back in full swing until finals. Thanks for checking.

Comment 31

[Gregory Szorc [:gps]]

No worries! Midterms are more important. Don't worry about this.

Comment 32

[Freddie F. Haddad]

Attachment: Bootstrap support for Gentoo

Comment 33

[Freddie F. Haddad]

Patch is submitted, hopefully I did everything right. I read through the FAQ and followed your instructions. Thanks again for being so active and helpful through the entire process. (= If anything is incorrect, I will fix it ASAP.

Comment 34

[Gregory Szorc [:gps]]

Comment on attachment 676150 [details] [diff] [review]
Bootstrap support for Gentoo

I'm setting the review flag so I'm reminded to attend to this. I have a busy day, so I may not get to this for a little while. We'll see...

Comment 35

[Gregory Szorc [:gps]]

Comment on attachment 676150 [details] [diff] [review]
Bootstrap support for Gentoo

Review of attachment 676150 [details] [diff] [review]:
-----------------------------------------------------------------

Aside from the one missing argument, this looks good.

Did you actually try running this? From your mozilla-central, you should be able to run:

  python python/mozboot/bin/bootstrap.py

Please verify that works without the 'dist_id' argument and then I'll check it in.

::: python/mozboot/mozboot/gentoo.py
@@ +6,5 @@
> +
> +from mozboot.base import BaseBootstrapper
> +
> +class GentooBootstrapper(BaseBootstrapper):
> +    def __init__(self, version):

I'm pretty sure that without a 'dist_id' argument here you'll get a run-time error.

Comment 36

[Freddie F. Haddad]

OK, I had to fix a few parameters and I tested the script using the above instructions. All worked and I got the final message to use hg clone... or git clone... as expected. I did notice, however, that if the user has his/her make.conf configured in a way that forces emerge to prompt for a yes or no confirmation and they select no, then the following error occurs:

fhaddad78@botched ~/files/dev/mozilla-central $ python python/mozboot/bin/bootstrap.py 
Executing as root: sudo emerge --onlydeps --quiet firefox

Nothing to merge; would you like to auto-clean packages? [Yes/No] n

Quitting.

Traceback (most recent call last):
  File "python/mozboot/bin/bootstrap.py", line 138, in <module>
    sys.exit(main(sys.argv))
  File "python/mozboot/bin/bootstrap.py", line 129, in main
    dasboot.bootstrap()
  File "python/mozboot/mozboot/bootstrap.py", line 77, in bootstrap
    instance.install_system_packages()
  File "python/mozboot/mozboot/gentoo.py", line 18, in install_system_packages
    'firefox'])
  File "python/mozboot/mozboot/base.py", line 36, in run_as_root
    subprocess.check_call(command, stdin=sys.stdin)
  File "/usr/lib64/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '[u'sudo', 'emerge', '--onlydeps', '--quiet', 'firefox']' returned non-zero exit status 130

However, as long as they answer yes, every proceeds as expected.

Comment 37

[Freddie F. Haddad]

Attachment: Tested

Comment 38

[Gregory Szorc [:gps]]

Comment on attachment 677614 [details] [diff] [review]
Tested

Review of attachment 677614 [details] [diff] [review]:
-----------------------------------------------------------------

Excellent. We'll fix the exception stuff in a follow-up (it's more of a global change to do right).

Comment 39

[Gregory Szorc [:gps]]

https://hg.mozilla.org/mozilla-central/rev/df3cc14f0ecd

This is now in the main tree. Congratulations on your first patch! Thank you for contributing.