1666883 - running "mach install-moz-phab" fails with missing RECORD file

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

I ran mach install-moz-phab just now and it failed like this:

 ./mach install-moz-phab
 0:00.54 Installing moz-phab
Collecting MozPhab
  Downloading https://files.pythonhosted.org/packages/e1/03/f4af0c331881c82b371363b32973b654080da44ad3ac57858c04bd53c174/MozPhab-0.1.90.tar.gz (69kB)
     |████████████████████████████████| 71kB 2.0MB/s 
Requirement already satisfied, skipping upgrade: distro in /home/willkg/.pyenv/versions/3.7.5/lib/python3.7/site-packages (from MozPhab) (1.4.0)
Collecting glean-sdk>=31.0.0 (from MozPhab)
  Downloading https://files.pythonhosted.org/packages/e7/37/f2ac2989896a6707b38af442d033453aa8fc9b9fbfbc4b5f88a57960e070/glean_sdk-33.0.0-cp36-abi3-manylinux1_x86_64.whl (1.5MB)
     |████████████████████████████████| 1.5MB 6.0MB/s 
Collecting sentry-sdk>=0.14.3 (from MozPhab)
  Downloading https://files.pythonhosted.org/packages/6f/0f/fabe13834484d499064198f8ecf6904a4ce569246ce41d2d747432a8f3c8/sentry_sdk-0.17.8-py2.py3-none-any.whl (120kB)
     |████████████████████████████████| 122kB 8.9MB/s 
Requirement already satisfied, skipping upgrade: setuptools in /home/willkg/.pyenv/versions/3.7.5/lib/python3.7/site-packages (from MozPhab) (41.2.0)
Requirement already satisfied, skipping upgrade: cffi>=1 in /home/willkg/.local/lib/python3.7/site-packages (from glean-sdk>=31.0.0->MozPhab) (1.13.2)
Collecting glean-parser==1.28.5 (from glean-sdk>=31.0.0->MozPhab)
  Downloading https://files.pythonhosted.org/packages/7a/fe/edec408443174fa1a43008a22ad4f899e4d8c3e49276982561c607764d41/glean_parser-1.28.5-py3-none-any.whl (57kB)
     |████████████████████████████████| 61kB 11.0MB/s 
Requirement already satisfied, skipping upgrade: certifi in /home/willkg/.local/lib/python3.7/site-packages (from sentry-sdk>=0.14.3->MozPhab) (2019.11.28)
Requirement already satisfied, skipping upgrade: urllib3>=1.10.0 in /home/willkg/.local/lib/python3.7/site-packages (from sentry-sdk>=0.14.3->MozPhab) (1.25.7)
Requirement already satisfied, skipping upgrade: pycparser in /home/willkg/.local/lib/python3.7/site-packages (from cffi>=1->glean-sdk>=31.0.0->MozPhab) (2.19)
Collecting diskcache>=4 (from glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
  Using cached https://files.pythonhosted.org/packages/fc/d6/7b8fa70e57f7b54e01da2664a6ee21afc315c29906bd46e9905fb246c5ac/diskcache-5.0.3-py3-none-any.whl
Collecting appdirs>=1.4 (from glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
  Using cached https://files.pythonhosted.org/packages/3b/00/2344469e2084fb287c2e0b57b72910309874c3245463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: Click>=7 in /home/willkg/.pyenv/versions/3.7.5/lib/python3.7/site-packages (from glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab) (7.0)
Collecting jsonschema>=3.0.2 (from glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
  Using cached https://files.pythonhosted.org/packages/c5/8f/51e89ce52a085483359217bc72cdbf6e75ee595d5b1d4b5ade40c7e018b8/jsonschema-3.2.0-py2.py3-none-any.whl
Collecting yamllint>=1.18.0 (from glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
  Using cached https://files.pythonhosted.org/packages/3d/cc/a68c1a8191a7d58188060bc0b1f8af38a5403772feb4ba8244059fc9eedb/yamllint-1.24.2-py2.py3-none-any.whl
Collecting PyYAML>=3.13 (from glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
Collecting Jinja2>=2.10.1 (from glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
  Using cached https://files.pythonhosted.org/packages/30/9e/f663a2aa66a09d838042ae1a2c5659828bb9b41ea3a6efa20a20fd92b121/Jinja2-2.11.2-py2.py3-none-any.whl
Collecting attrs>=17.4.0 (from jsonschema>=3.0.2->glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
  Using cached https://files.pythonhosted.org/packages/14/df/479736ae1ef59842f512548bacefad1abed705e400212acba43f9b0fa556/attrs-20.2.0-py2.py3-none-any.whl
Collecting importlib-metadata; python_version < "3.8" (from jsonschema>=3.0.2->glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
  Using cached https://files.pythonhosted.org/packages/6d/6d/f4bb28424bc677bce1210bc19f69a43efe823e294325606ead595211f93e/importlib_metadata-2.0.0-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: six>=1.11.0 in /home/willkg/.pyenv/versions/3.7.5/lib/python3.7/site-packages (from jsonschema>=3.0.2->glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab) (1.13.0)
Collecting pyrsistent>=0.14.0 (from jsonschema>=3.0.2->glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
Collecting pathspec>=0.5.3 (from yamllint>=1.18.0->glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
  Using cached https://files.pythonhosted.org/packages/5d/d0/887c58853bd4b6ffc7aa9cdba4fc57d7b979b45888a6bd47e4568e1cf868/pathspec-0.8.0-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10.1->glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
  Using cached https://files.pythonhosted.org/packages/98/7b/ff284bd8c80654e471b769062a9b43cc5d03e7a615048d96f4619df8d420/MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting zipp>=0.5 (from importlib-metadata; python_version < "3.8"->jsonschema>=3.0.2->glean-parser==1.28.5->glean-sdk>=31.0.0->MozPhab)
  Using cached https://files.pythonhosted.org/packages/c4/79/3b770d51254a31bb85ba56ea70d7428d0c2c659a233cc9722352e028b539/zipp-3.2.0-py3-none-any.whl
Installing collected packages: diskcache, appdirs, attrs, zipp, importlib-metadata, pyrsistent, jsonschema, PyYAML, pathspec, yamllint, MarkupSafe, Jinja2, glean-parser, glean-sdk, sentry-sdk, MozPhab
  Running setup.py install for MozPhab ... done
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 MozPhab-0.1.90 PyYAML-5.3.1 appdirs-1.4.4 attrs-20.2.0 diskcache-5.0.3 glean-parser-1.28.5 glean-sdk-33.0.0 importlib-metadata-2.0.0 jsonschema-3.2.0 pathspec-0.8.0 pyrsistent-0.17.3 sentry-sdk-0.17.8 yamllint-1.24.2 zipp-3.2.0
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Error running mach:

    ['install-moz-phab']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file install-moz-phab| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

FileNotFoundError: [Errno 2] No such file or directory: '/home/willkg/.local/lib/python3.7/site-packages/MozPhab-0.1.90-py3.7.egg-info/RECORD'

  File "/home/willkg/mozilla/src/mozilla-unified/tools/phabricator/mach_commands.py", line 99, in install_moz_phab
    package_files = [file.split(',')[0] for file in dist.get_metadata_lines('RECORD')]
  File "/home/willkg/.pyenv/versions/3.7.5/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1432, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/home/willkg/.pyenv/versions/3.7.5/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1420, in get_metadata
    value = self._get(path)
  File "/home/willkg/.pyenv/versions/3.7.5/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1616, in _get
    with open(path, 'rb') as stream:
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

The contents of that directory:

> ls ~/.local/lib/python3.7/site-packages/MozPhab-0.1.90-py3.7.egg-info/
dependency_links.txt  entry_points.txt  installed-files.txt  PKG-INFO  requires.txt  SOURCES.txt  top_level.txt

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Agh, pip with (edit: a fresh install of) python 3.7 doesn't add the RECORD file, huh. I'll dig in here. Thanks for the report!
The only work that happens after that error is that it runs moz-phab install-certificate. If you run that manually, then you should be good.

EDIT: looks like pip 19.2.3 puts the list of the package's files in installed-files.txt, while pip 20+ puts it in RECORD.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Thank you!

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1666883: install-moz-phab should use pip CLI, not internal file

Programmatically detecting the moz-phab location by looking at pip
egg/dist files is easily subject to breakage if pip changes its internal
file structure, as it did from pip 19 to pip 20.

So, instead, call the (more stable) pip CLI directory and parse the
output.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/92f78f7866e2
install-moz-phab should use pip CLI, not internal file r=firefox-build-system-reviewers,rstewart

Comment 5

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/92f78f7866e2