Closed Bug 1630345 Opened 5 years ago Closed 5 years ago

./mach bootstrap fails with python-pip dependency issue: python-pip : Depends: python-pip-whl (= 18.1-5) but 20.0.2-4 is to be installed

Categories

(Firefox Build System :: Bootstrap Configuration, defect, P3)

Product:
Firefox Build System
Component:
Bootstrap Configuration
Platform:
x86_64
Linux
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1637386

People

(Reporter: ishikawa, Assigned: rstewart)

References

Details

This is on Debian GNU/Linux, but I think the newer version of Ubuntu has the same problem.

The dependency bug is also reported in Ubuntu bugzilla.
https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1869117

It is suggested that python3-pip be used (but that installs pip3 instead of pip?).

I am afraid that I can't update build infrastructure on my local PC
to install new "fix-stacks" python(?) script.

TIA

Assignee: nobody → rstewart
Priority: -- → P3

Which version of Debian are you running and which version of the code are you using? We haven't required installing python-pip on newer versions of Debian and Ubuntu since bug 1626098. So I'm wondering whether you're running an older version of the code from before then or if we need to augment the existing check.

Flags: needinfo?(ishikawa)

Hi, Ricky.

This is what my Debian installation tells me. I am using testing repository since Debian is too conservative and I need to obtain a few more up-to-date development tools.

ishikawa@ip030:/home/ishikawa$ uname -a
Linux ip030 5.3.0-3-amd64 #1 SMP Debian 5.3.15-1 (2019-12-07) x86_64 GNU/Linux
ishikawa@ip030:/home/ishikawa$ cat /etc/debian_version
bullseye/sid
ishikawa@ip030:/home/ishikawa$

Oh, maybe I should reboot. I have not done so for a while even after package upgrade.
Let me do it now.

After rebooting.:

ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ uname -a
Linux ip030 5.4.0-4-amd64 #1 SMP Debian 5.4.19-1 (2020-02-13) x86_64 GNU/Linux
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ cat /etc/debian_version
bullseye/sid
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$

I tried |mach bootstrap| and still got the same error as before.

Your current directory is
++ pwd
/NREF-COMM-CENTRAL/mozilla
++ /NREF-COMM-CENTRAL/mozilla/mach --log-no-times bootstrap
kind=None
environ.get('TERM', 'unknown')=xterm-256color
environ.get('TERM', 'linux')=xterm-256color

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:
  1. Firefox for Desktop Artifact Mode
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
Your choice: 2
Executing as root: sudo apt-get install autoconf2.13 build-essential nodejs python-setuptools unzip uuid zip python-pip python-dev
[sudo] password for ishikawa: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
autoconf2.13 is already the newest version (2.13-68).
zip is already the newest version (3.0-11+b1).
build-essential is already the newest version (12.8).
nodejs is already the newest version (10.19.0~dfsg-3).
uuid is already the newest version (1.6.2-1.5+b8).
python-dev is already the newest version (2.7.17-2).
python-setuptools is already the newest version (44.0.0-1).
unzip is already the newest version (6.0-25).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python-pip : Depends: python-pip-whl (= 18.1-5) but 20.0.2-4 is to be installed
E: Unable to correct problems, you have held broken packages.
Error running mach:

    ['--log-no-times', 'bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['sudo', 'apt-get', 'install', 'autoconf2.13', 'build-essential', 'nodejs', 'python-setuptools', 'unzip', 'uuid', 'zip', 'python-pip', 'python-dev']' returned non-zero exit status 100.

  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozboot/mozboot/mach_commands.py", line 44, in bootstrap
    bootstrapper.bootstrap()
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozboot/mozboot/bootstrap.py", line 443, in bootstrap
    self.instance.install_system_packages()
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozboot/mozboot/debian.py", line 145, in install_system_packages
    self.apt_install(*packages)
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozboot/mozboot/base.py", line 422, in apt_install
    self.run_as_root(command)
  File "/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozboot/mozboot/base.py", line 378, in run_as_root
    subprocess.check_call(command, stdin=sys.stdin)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)

real	0m8.275s
user	0m1.508s
sys	0m0.544s

TIA

Flags: needinfo?(ishikawa)
See Also: → 1364543

I am not sure where to write this, but I think the issue in bug 1630345 definitely will hit linux developers who use Debian (and possibly Ubuntu which is derived from it, but I am not sure) and import packages from the testing repo as soon as they upgrade the local source tree.
(Why non-standard "testing" repo? I put my perspective as a Debian user at the end.)

So maybe in a couple of weeks, more and more developers who build binaries locally notice the issue in bug 1630345 and maybe we have a pressing need to move the priority(?)

I would be happy if there is a non-standard local fix, meaning a kludge which one has to apply manually would be fine for a while.
I won't mind living with such a local fix which I can apply every time I fetch the source and
mach bootstrap would fail. (Such a kludge may not be applicable to non-Debian and non-Ubuntu users.)

Problem now is that there does not seem to be a "pip" binary on my local linux installation anymore and "pip3" exists instead.
A possible kludge fix?: I wonder if the replacement of "pip" to "pip3" in the script(s) would get me proceed.
(Or maybe I can create a sympolic link to pip3 from pip in my search path, etc. But then |mach bootstrap| will still fail with the bugzilla discussed here.)

BTW, I just realized that I cannot use "fix stack" script any more which renders the stacktrace unusable. I used to run the fix_linux_stack.py script from within my own external script. I used to invoke this script AFTER the whole |make mozmill| test is run to obtain symbolic stack trace.

Now, when I ran the script (I even tried to re-use the old copy in a archived tree.) against locally recorded mochitest log,
the numeric stacktrace is not transformed into symbolic ones.
Maybe the tangled python package dependency on my PC has nullified the effect of some tools essentially. (Or the testing script internal hid the warning/error message that would have told me that a script was not found or something to the effect.)

This is tough. The stacktrace is printed by NS_ASSERTION or something for DEBUG build and is quite useful debugging info.

TIA

PS: A perspective a long-time Debian user.

  • I use Debian GNU/Linux for its stability.

  • OTOH, Debian is so conservative to keep it stable that I need to fetch some newer versions of development tools from its
    so-called testing repository.

  • I don't think I have used Debian's APT "pinning" feature aggressively. I don't think I have used it before IIRC. I have read that not using it can get an unsuspecting user in trouble sometimes, but this time using it would have put some users in trouble (?)
    I have no pressing need for "pinning". All I need is a newer development tools for building TB locally.

  • I USED TO create tool binaries since Debian's tools could be outdated by half a year or longer. (Not any more).
    For example, for many years, I used to install GCC on my own to compile and use GNU Emacs.
    (BTW, I still build GNU Emacs from source on my own.)
    Due to some bugs, I also built and used the following binary tools from time to time.
    -- binutils package (I wanted to use GOLD linker for its speed and the use of incremental symbol loading to reduce gdb startup time),
    -- ccache to support the above feature (to create .dwo auxiliary file). It now supports the feature officially,
    -- valgrind (occasionally I hit a snug with valgrind and either added a support for not-yet-supported sysctl or feature).
    -- hg. (There was a time when Debian's hg was way behind the official release. I need to keep up with the version which mozilla recommended.)

  • But there are simply too many tools for an individual to keep track of and build locally.
    For example, I used to built and installed Facebook's |watchman| tool since HG (mercurial) required it. I built and installed from source.
    But it no longer builds after the latest Debian kernel upgrade due to some mysterious build-time errors.
    (Ouch, now I realize that there are some python header issues due to the tangled python package dependency issues discussed in this bugzilla. That is why the build failed. I wondered why so many headers were missing for |watchman| compilation. Hmm...)
    So I simply decided to import Debian package |watchman| instead of building the binary locally as I did until December.

I am afraid that there are Ubuntu users in a similar situation.

I have found out that on my particular PC running Debian GNU/Linux the following patch has made me run ./mach bootstrap successfully.
But I suppose this depends on many factors, so YMMV. At least, now I believe I could upgrade the fix-stacks script. I will check.

hg diff
diff --git a/python/mozboot/mozboot/debian.py b/python/mozboot/mozboot/debian.py
--- a/python/mozboot/mozboot/debian.py
+++ b/python/mozboot/mozboot/debian.py
@@ -113,17 +113,17 @@ class DebianBootstrapper(
             self.packages.extend(['python2.7', 'python2.7-dev'])
         else:
             if (self.distro == 'ubuntu'
                 or (self.distro == 'debian' and self.codename != "bullseye")):
                 # On old Ubuntu and Debian before bullseye (11), it was called this way
                 # Note that we don't use Debian version code as the Python API doesn't provide
                 # it yet
                 # TODO: Update once bullseye is released in 2021
-                self.packages.append('python-pip')
+                self.packages.append('python3-pip')
 
             self.packages.append('python-dev')
         self.browser_packages = self.BROWSER_COMMON_PACKAGES + self.BROWSER_DISTRO_PACKAGES
         self.mobile_android_packages = self.MOBILE_ANDROID_COMMON_PACKAGES + \
             self.MOBILE_ANDROID_DISTRO_PACKAGES
 
     def install_system_packages(self):
         # Python 3 may not be present on all distros. Search for it and

But this works on my PC with the following setup.

ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ uname -a
Linux ip030 5.4.0-4-amd64 #1 SMP Debian 5.4.19-1 (2020-02-13) x86_64 GNU/Linux

I am using testing repository of Debian.

ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ cat /etc/apt/sources.list
# 

# deb cdrom:[Debian GNU/Linux 10.0.0 _Buster_ - Official amd64 NETINST 20190706-10:23]/ buster main

#deb cdrom:[Debian GNU/Linux 10.0.0 _Buster_ - Official amd64 NETINST 20190706-10:23]/ buster main

deb http://debian-mirror.sakura.ne.jp/debian/ buster main
deb-src http://debian-mirror.sakura.ne.jp/debian/ buster main

deb http://security.debian.org/debian-security buster/updates main
deb-src http://security.debian.org/debian-security buster/updates main

# buster-updates, previously known as 'volatile'
deb http://debian-mirror.sakura.ne.jp/debian/ buster-updates main
deb-src http://debian-mirror.sakura.ne.jp/debian/ buster-updates main

deb http://debian-mirror.sakura.ne.jp/debian buster contrib non-free
deb-src http://debian-mirror.sakura.ne.jp/debian buster contrib non-free

deb http://debian-mirror.sakura.ne.jp/debian testing main
deb-src http://debian-mirror.sakura.ne.jp/debian testing main
deb http://security.debian.org/ buster/updates main contrib non-free

# This system was installed using small removable media
# (e.g. netinst, live or single CD). The matching "deb cdrom"
# entries were disabled at the end of the installation process.
# For information about how to configure apt package sources,
# see the sources.list(5) manual.

Because of the setup of apt/sources.list, my Debian version is shown as:

ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ cat /etc/debian_version 
bullseye/sid

Under this particular setup, the aforementioned patch worked. YMMV.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → DUPLICATE

Closing this as a duplicate because I think removing Py2 from bootstrap entirely is ultimately what's going to solve the underlying problem here.

Just for completeness sake from the perspective of Debian user, I quote this from
Bug 1626098 comment 1
Workaround:

sudo apt install python3-pip python3-setuptools python3-dev
./mach bootstrap --no-system-changes

|--no-system-changes| is something I have not known and would be useful.

I updated the tree yesterday (M-C/C-C), and found that python/mozboot/mozboot/debian.py has been modified with a finer
check on the version before updating python-pip tool.
(I am on Debian GNU/Linux amd64, /etc/debian_version contains "bullseye/sid" and I am using testing repository.)

I could run |make bootstrap| successfully.

I am not sure if Debian repository fixed any issues related to this, but I wanted to let people know that |make bootstrap| works as is without my local patch any more. (I substituted "python-pip" with "python3-pip" (comment 5). With the latest update, I no longer do and |make bootstrap| works.

You need to log in before you can comment on or make changes to this bug.