1174037 - Linux and MacOS build preparation script bootstrap.py fails on Ubuntu Vivid (15.04)

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[Renzo Canepa [:rcanepa]]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36

Steps to reproduce:

In order to setup my environment to contribute to the Firefox project, I followed the instructions provided here http://codefirefox.com/video/linux-build-env, where I was asked to execute the next command:

wget -q https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py && python bootstrap.py

Check this link also: https://developer.mozilla.org/en-US/docs/Simple_Firefox_build/Linux_and_MacOS_build_preparation


Actual results:

The process failed throwing the following error:

Hit http://cl.archive.ubuntu.com vivid-updates/universe Translation-en                                                                                                                                   
Hit http://cl.archive.ubuntu.com vivid-backports/universe Sources                                                                                                                                        
Hit http://cl.archive.ubuntu.com vivid-backports/universe amd64 Packages                                                                                                                                 
Hit http://cl.archive.ubuntu.com vivid-backports/universe i386 Packages                                                                                                                                  
Hit http://cl.archive.ubuntu.com vivid-backports/universe Translation-en                                                                                                                                 
Fetched 1.433 B in 35s (39 B/s)                                                                                                                                                                          
W: Failed to fetch http://ppa.launchpad.net/mercurial-ppa/releases/ubuntu/dists/vivid/main/binary-amd64/Packages  404  Not Found

W: Failed to fetch http://ppa.launchpad.net/mercurial-ppa/releases/ubuntu/dists/vivid/main/binary-i386/Packages  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.
Traceback (most recent call last):
  File "bootstrap.py", line 163, in <module>
    sys.exit(main(sys.argv))
  File "bootstrap.py", line 154, in main
    dasboot.bootstrap()
  File "/tmp/user/1000/tmpl8a1BE/mozboot/bootstrap.py", line 136, in bootstrap
    parser.add_option('--application-choice', dest='application_choice',
  File "/tmp/user/1000/tmpl8a1BE/mozboot/base.py", line 293, in ensure_mercurial_modern
  File "/tmp/user/1000/tmpl8a1BE/mozboot/base.py", line 248, in _ensure_package_manager_updated
  File "/tmp/user/1000/tmpl8a1BE/mozboot/debian.py", line 133, in _update_package_manager
  File "/tmp/user/1000/tmpl8a1BE/mozboot/base.py", line 196, in apt_update
  File "/tmp/user/1000/tmpl8a1BE/mozboot/base.py", line 159, in run_as_root
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '[u'sudo', u'apt-get', u'update']' returned non-zero exit status 100

After doing a little research, I found that the ppa doesn't have a release for Ubuntu Vivid (15.04). You can see that they have the package for this version under the staging release phase (http://ppa.launchpad.net/mercurial-ppa/staging-releases/ubuntu/dists/vivid/main/binary-amd64/Packages).


Expected results:

It should have downloaded the mercurial package from the ppa.

Comment 1

[Mike Hoye [:mhoye]]

Thank you, I'll try to replicate shortly.

Comment 2

[Antonio de Luna [:A-deLuna]]

I had the same problem and found this ppa https://launchpad.net/~jacob/+archive/ubuntu/hg which has a release for Ubuntu Vivid (15.04).

I think a simple fix would be to switch the ppa in this line https://dxr.mozilla.org/mozilla-central/source/python/mozboot/mozboot/ubuntu.py#27.

I'd also appreciate being assigned to this bug.

Comment 3

[Mike Hommey [:glandium]]

I'm not too thrilled recommending random people's ppa for mercurial. Greg, what do you think?

Comment 4

[Gregory Szorc [:gps]]

I'm also not thrilled. It's tempting to throw up a .deb on people.mozilla.org or something. The reasons for why Mercurial doesn't distribute a rpm or deb are... complicated IIRC.

Comment 6

[Gregory Szorc [:gps]]

Mercurial is producing RPMs as part of their release process. See some of the directories at https://mercurial.selenic.com/release/.

I imagine we could convince them to start producing .deb files as well. If not, we can always produce one and upload it to S3 or something.

glandium: would you be comfortable having `mach bootstrap` install a .deb directly (as opposed to going through a package repository).

Comment 7

[Mike Hommey [:glandium]]

What's the reason we're trying the PPA in the first place?

Comment 8

[Gregory Szorc [:gps]]

I believe I was the one who added the PPA. At the time, the PPA seemed well supported and I perceived it as the easiest way to install a modern Mercurial version. Many instructions online referenced this PPA as the best way to get a modern Mercurial on Ubuntu. And PPAs seem to be how Ubuntu recommends installing after-market packages.

Comment 10

[Mike Hommey [:glandium]]

How about we just install mercurial from the ubuntu repo if it's above some version, and fallback to the PPA if it's not?

Comment 11

[Mike Hommey [:glandium]]

Mmmm vivid only has 3.1...

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It may have been discussed somewhere else but why cannot we use the Python package: https://pypi.python.org/pypi/Mercurial?

Comment 13

[Gregory Szorc [:gps]]

So, I thought `sudo /usr/bin/pip install Mercurial` would install things under /usr/lib/python2.7/site-packages and /usr/bin, which would almost certainly make people upset as files under /usr are typically managed by the distro's package manager. However, I see that on fresh 14.04 and 15.04 installs, `sudo pip install Mercurial` actually installs to /usr/local. I think I was confusing the behavior of OS X with Ubuntu (it's very easy to install Python packages under /usr on OS X).

Anyway, given Debian and Ubuntu's historically laggy updating of Mercurial packages, we should switch the bootstrapper to `pip install --upgrade Mercurial`.

FWIW, some patches were recently submitted to upstream Mercurial to change its .deb package generation to produce "mercurial" and "mercurial-common" debs to match Debian/Ubuntu. Presumably the release process for Mercurial 3.6 (release November 1) will produce .deb files which we can download and dpkg install if we ever wish to switch away from `pip` (which I imagine some would appreciate since .deb packages are preferred over Python packages).

Comment 14

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 1174037 - Support installing Mercurial from pip; r?glandium

Bug 1174037 - Install Mercurial from pip; r?glandium

Previously, we used a PPA on Ubuntu to install Mercurial. The PPA has
proved to be unreliable. Furthermore, we didn't have a mechanism for
installing a modern Mercurial on Debian.

In this commit, we remove the PPA from Ubuntu and use `pip` to install
Mercurial on both Debian and Ubuntu distros. A nice side effect is that
the Ubuntu bootstrapper is now equivalent to the Debian one!

Comment 15

[Mike Hommey [:glandium]]

Comment on attachment 8655083 [details]
MozReview Request: Bug 1174037 - Support installing Mercurial from pip; r?glandium

https://reviewboard.mozilla.org/r/17811/#review15919

::: python/mozboot/mozboot/debian.py:138
(Diff revision 1)
> +        self.run_as_root(['/usr/bin/pip', 'install', '--upgrade', 'Mercurial'])

Is there a specific reason to use /usr/bin/pip instead of pip?

I think there should be a prompt here. I, for one, would not like this script installing software that is not handled by the packaging system, even if it's in /usr/local. (especially when, in fact, mercurial is not mandatory)

::: python/mozboot/mozboot/ubuntu.py:10
(Diff revision 1)
> -        self.run_as_root(['add-apt-repository', 'ppa:%s' % ppa])
> +    We have previously had to make one-off changes to Ubuntu. Rather than
> +    delete this class, keep it around to make future one-off changes easier to
> +    implement.
> +    """

Considering how long this has been around,
considering the only difference was about mercurial,
and considering the expected future for mercurial installation won't differ between debian and ubuntu,
I'm not convinced of the usefulness to keep this.

Comment 16

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/17811/#review15919

> Is there a specific reason to use /usr/bin/pip instead of pip?
> 
> I think there should be a prompt here. I, for one, would not like this script installing software that is not handled by the packaging system, even if it's in /usr/local. (especially when, in fact, mercurial is not mandatory)

I'll change it to "pip" and let PATH deal with it.

I agree that a prompt is warranted here.

Comment 17

[Gregory Szorc [:gps]]

Comment on attachment 8655083 [details]
MozReview Request: Bug 1174037 - Support installing Mercurial from pip; r?glandium

Bug 1174037 - Support installing Mercurial from pip; r?glandium

Previously, we used a PPA on Ubuntu to install Mercurial. The PPA has
proved to be unreliable. Furthermore, we didn't have a mechanism for
installing a modern Mercurial on Debian and other derived distros.

In this commit, we remove the PPA from Ubuntu. We add the ability to
install Mercurial from pip to the Debian bootstrapper. However, since
some people may not want <not apt> installing package-like things,
we add a prompt explaining the situation and giving users a choice.
We recommend installing a modern Mercurial via pip. But we also given
the option to install a likely legacy version via apt. And, for Git
users, we give the option to not install Mercurial at all.

Since the new version of the Ubuntu bootstrapper is empty, it doesn't
need to exist, so it has been removed. Yes, the logic in the distro to
bootstrapper class mapping could be consolidated to a single iterable.
However, I feel the inline comments and the separation of each distro
into discrete blocks aids readability.

Comment 18

[Mike Hommey [:glandium]]

Comment on attachment 8655083 [details]
MozReview Request: Bug 1174037 - Support installing Mercurial from pip; r?glandium

https://reviewboard.mozilla.org/r/17811/#review16251

::: python/mozboot/mozboot/debian.py:13
(Diff revisions 1 - 2)
> +quickly becomes out of date. This may cause incompatibility with some

s/becomes/may become/

::: python/mozboot/mozboot/debian.py:19
(Diff revisions 1 - 2)
> +in files being placed in /usr/local/bin and /usr/local/lib.

Strikes me this shouldn't be debian specific. This very much applies to the other distros as well, and we can detect that the mercurial package is old there too. But perfect is enemy of done, so followup bug?

::: python/mozboot/mozboot/debian.py:158
(Diff revisions 1 - 2)
> -        self.run_as_root(['/usr/bin/pip', 'install', '--upgrade', 'Mercurial'])
> +        if not self.no_interactive:

Mmmm should we still install from pip if not in interactive mode?

::: python/mozboot/mozboot/base.py:318
(Diff revision 2)
> +        if self._ignore_hg_version_check:

Why not just act on the upgrade_mercurial return value? (which conveniently, if there is no return, is None, so you can check for an explicit False)

::: python/mozboot/mozboot/bootstrap.py:74
(Diff revision 2)
>              elif distro in ('Mint', 'LinuxMint'):

seems to me there could be one big if distro in ('Debian', 'debian', 'Ubuntu', 'Mint', 'LinuxMint', 'Elementary', '"elementary OS"')
Then you can add a comment that if we ever re-require separate classes for debian and ubuntu, the difference should be made based on dist_id for the different flavors of Mint.

Comment 19

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/dc6af89505c8

Comment 20

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/dc6af89505c8