Closed Bug 913965 Opened 6 years ago Closed 6 years ago

Ubuntu bootstrapper should not use build-dep for finding dependencies / install modern Mercurial

Categories

(Firefox Build System :: General, defect)

x86_64
Linux
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla26

People

(Reporter: ksk.3393, Assigned: gps)

References

Details

Attachments

(1 file, 3 obsolete files)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0 (Beta/Release)
Build ID: 20130803215302

Steps to reproduce:

OS : Ubuntu 12.04.2 LTS
Followed instructions from https://developer.mozilla.org/en-US/docs/Simple_Firefox_build/Linux_and_MacOS_build_preparation 
Run the following command : 
wget --no-check-certificate https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py && python bootstrap.py


Actual results:

--2013-09-09 05:58:10--  https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py
Resolving hg.mozilla.org (hg.mozilla.org)... 63.245.215.25
Connecting to hg.mozilla.org (hg.mozilla.org)|63.245.215.25|:443... connected.
HTTP request sent, awaiting response... 200 Script output follows
Length: 4449 (4.3K) [text/x-python]
Saving to: `bootstrap.py'

100%[======================================================================================================>] 4,449       --.-K/s   in 0.006s  

2013-09-09 05:58:12 (772 KB/s) - `bootstrap.py' saved [4449/4449]

Executing as root: sudo apt-get build-dep firefox
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to find a source package for firefox
Traceback (most recent call last):
  File "bootstrap.py", line 143, in <module>
    sys.exit(main(sys.argv))
  File "bootstrap.py", line 134, in main
    dasboot.bootstrap()
  File "/tmp/tmpg0MQa2/mozboot/bootstrap.py", line 85, in bootstrap
    # The next fallback is to download the files from the source
  File "/tmp/tmpg0MQa2/mozboot/ubuntu.py", line 17, in install_system_packages
  File "/tmp/tmpg0MQa2/mozboot/base.py", line 106, in run_as_root
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '[u'sudo', 'apt-get', 'build-dep', 'firefox']' returned non-zero exit status 100
I'm pretty sure this has been reported before. The underling issue is we're relying on the user's machine to have the sources package repository available so the "build-dep" command works. As has been noted many times, we should just maintain our own explicit list of dependency packages instead of relying on the build dependencies for the distro's firefox package, which likely lags behind m-c by a few releases.
Summary: Bootstrap didn't work → Ubuntu bootstrapper should not use build-dep for finding dependencies
This seems to do the trick (at least with the base Ubuntu 13.04 VM I was
using). A build-dep firefox still wants the following packages:

cdbs dh-translations diffstat gconf-service gconf-service-backend gconf2
gconf2-common gir1.2-gconf-2.0 gir1.2-gnomekeyring-1.0 hardening-wrapper
intltool libart-2.0-2 libart-2.0-dev
  libavahi-client-dev libavahi-common-dev libavahi-glib-dev
libavahi-glib1 libbonobo2-0 libbonobo2-common libbonobo2-dev
libbonoboui2-0 libbonoboui2-common libbonoboui2-dev
  libcanberra-dev libencode-locale-perl libfile-listing-perl
libgail-common libgail-dev libgail18 libgconf-2-4 libgconf2-dev
libglade2-0 libgnome-keyring-common libgnome-keyring-dev
  libgnome-keyring0 libgnome2-0 libgnome2-bin libgnome2-common
libgnome2-dev libgnomecanvas2-0 libgnomecanvas2-common
libgnomecanvas2-dev libgnomeui-0 libgnomeui-common libgnomeui-dev
  libgnomevfs2-0 libgnomevfs2-common libgnomevfs2-dev
libhtml-parser-perl libhtml-tagset-perl libhtml-tree-perl
libhttp-cookies-perl libhttp-date-perl libhttp-message-perl
  libhttp-negotiate-perl libidl-common libidl-dev libidl0
libio-socket-ssl-perl liblwp-mediatypes-perl liblwp-protocol-https-perl
libnet-http-perl libnet-ssleay-perl liborbit2
  liborbit2-dev libpopt-dev libselinux1-dev libsepol1-dev
libstartup-notification0 libstartup-notification0-dev liburi-perl
libwww-perl libwww-robotrules-perl libx11-xcb1 libxcb-util0
  libxml-parser-perl python-scour quilt xvfb

Should I throw any of these in here or should I just defer to configure?
Attachment #801886 - Flags: review?(mh+mozilla)
Assignee: nobody → gps
I looked at the package metadata and added some missing -dev packages.
This appears to satisfy all but the Debian package development
dependency set with the following build-dep packages outstanding:

cdbs dh-translations diffstat intltool libencode-locale-perl
libfile-listing-perl libhtml-parser-perl libhtml-tagset-perl
libhtml-tree-perl libhttp-cookies-perl libhttp-date-perl
  libhttp-message-perl libhttp-negotiate-perl libio-socket-ssl-perl
liblwp-mediatypes-perl liblwp-protocol-https-perl libnet-http-perl
libnet-ssleay-perl liburi-perl libwww-perl
  libwww-robotrules-perl libxml-parser-perl python-scour quilt
Attachment #801890 - Flags: review?(mh+mozilla)
Attachment #801886 - Attachment is obsolete: true
Attachment #801886 - Flags: review?(mh+mozilla)
Verified the bootstrapper works with 12.04 as well. I assume 12.10 behaves the same. I can individually verify other releases upon request.
Comment on attachment 801890 [details] [diff] [review]
Don't use build-dep for installing Ubuntu dependencies

Review of attachment 801890 [details] [diff] [review]:
-----------------------------------------------------------------

Please also change debian.py. Or, in fact, since the only real difference between debian.py and ubuntu.py, besides the fact that debian.py has been bitrotted, is apt-get build-dep iceweasel instead of apt-get build-dep firefox, please merge both.

::: python/mozboot/mozboot/ubuntu.py
@@ +17,3 @@
>          self.apt_install(
>              'autoconf2.13',
> +            'binutils-gold',

This makes /usr/bin/ld point to gold, which may not be wanted for other things than firefox. The package is not necessary for the availability of gold itself: it is in binutils. And the build system knows how to use gold from there. So please remove binutils-gold. (and binutils is in build-essential, so no need to replace)

@@ +18,5 @@
>              'autoconf2.13',
> +            'binutils-gold',
> +            'build-essential',
> +            'ccache',
> +            'clang',

Not sure this is helpful. Until 13.04 (which i wouldn't advise anyone to use yet), the clang package is clang 3.0, which can't be used to build the tree.

@@ +19,5 @@
> +            'binutils-gold',
> +            'build-essential',
> +            'ccache',
> +            'clang',
> +            'dbus-x11',

we're not using dbus-launch, which is about the only thing in that package.

@@ +25,5 @@
>              'libcurl4-openssl-dev',
> +            'libdbus-1-dev',
> +            'libdbus-glib-1-dev',
> +            'libgconf2-dev',
> +            'libgnomeui-dev',

libgnomeui is optional and not enabled by default. We're using gio now.

@@ +33,3 @@
>              'libiw-dev',
>              'libnotify-dev',
> +            'libstartup-notification0-dev',

startup notification is disabled by default.

@@ +36,2 @@
>              'libxt-dev',
>              'mercurial',

Relatedly, i'm not sure mercurial has its place here. Because if you have that code, you obviously have got the tree. Which you may well have done with git, and don't care about mercurial.
Attachment #801890 - Flags: review?(mh+mozilla) → review-
wrt mercurial, it's possible to run bootstrap.py standalone, without a copy of the tree.
(In reply to Gregory Szorc [:gps] from comment #6)
> wrt mercurial, it's possible to run bootstrap.py standalone, without a copy
> of the tree.

Where does bootstrap.py get ubuntu.py from?
Oh wow python/mozboot/bin/bootstrap.py.
Whiteboard: kanbanzilla[Needs Investigation]
Status: UNCONFIRMED → NEW
Ever confirmed: true
Whiteboard: kanbanzilla[Needs Investigation]
Removed "bad" packages and consolidated debian and ubuntu package lists.
Attachment #802683 - Flags: review?(mh+mozilla)
Attachment #801890 - Attachment is obsolete: true
Comment on attachment 802683 [details] [diff] [review]
Don't use build-dep for installing Ubuntu dependencies

Review of attachment 802683 [details] [diff] [review]:
-----------------------------------------------------------------

I'd prefer ubuntu.py to go away, the _update_package_manager and upgrade_mercurial functions to move to debian.py, and the ubuntu and elementary distros to use DebianBootstrapper.
Attachment #802683 - Flags: review?(mh+mozilla) → review-
For upgrade_mercurial(), since Ubuntu is currently shipping an ancient Mercurial, I was thinking that Ubuntu's implementation would hook up a PPA. This would necessitate a different class from Debian.

Ubuntu could definitely inherit from the Debian class, however.
Ubuntu now inherits from Debian. While I was here, I hooked up a
Mercurial PPA to Ubuntu because I don't feel like dealing with the
overhead of another bug and patch.
Attachment #803188 - Flags: review?(mh+mozilla)
Attachment #802683 - Attachment is obsolete: true
Comment on attachment 803188 [details] [diff] [review]
Don't use build-dep for installing Ubuntu dependencies

Review of attachment 803188 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozboot/mozboot/ubuntu.py
@@ +14,5 @@
>  
> +# Ubuntu shares much logic with Debian, so it inherits from it.
> +class UbuntuBootstrapper(DebianBootstrapper):
> +    DISTRO_PACKAGES = [
> +        'software-properties-common',

Please add a comment saying this contains add-apt-repository.

@@ +33,5 @@
> +        for source in os.listdir('/etc/apt/sources.list.d'):
> +            if source.startswith(list_file) and source.endswith('.list'):
> +                return
> +
> +        print(MERCURIAL_PPA)

debugging leftover?
Attachment #803188 - Flags: review?(mh+mozilla) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/6d30f146c472

With both feedback points addressed.
Status: NEW → ASSIGNED
Flags: in-testsuite-
Summary: Ubuntu bootstrapper should not use build-dep for finding dependencies → Ubuntu bootstrapper should not use build-dep for finding dependencies / install modern Mercurial
https://hg.mozilla.org/mozilla-central/rev/6d30f146c472
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla26
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.