Closed Bug 1481612 Opened 2 years ago Closed Last year

Figure out a solution for collecting system info on Windows (psutil is not reliably available)

Categories

(Firefox Build System :: General, enhancement)

enhancement
Not set

Tracking

(firefox64 fixed)

RESOLVED FIXED
mozilla64
Tracking Status
firefox64 --- fixed

People

(Reporter: ted, Assigned: ted)

References

Details

Attachments

(4 files)

I'm punting on this in bug 1237610 for now. I'm using psutil to collect system information since it provides most of the things I need. Unfortunately we don't reliably have psutil available on Windows, since it has a binary component and building binary Python components on Windows is hard. We'll either need to write a non-psutil implementation for Windows or figure out a way to reliably get psutil working on Windows (vendoring a wheel or something like that).
How do we handle that currently on test machines? I can remember some troubles when working on bug 1436857 for the recent psutil upgrade. For now we have the source code under third_party/python, but without any compiled binary. So where do we handle downloading/installing the appropriate psutil wheel?
It looks like mozharness installs psutil very early, and it installs a wheel. From an arbitrary Windows test job on inbound:
https://treeherder.mozilla.org/logviewer.html#?job_id=192732758&repo=mozilla-inbound

08:56:25     INFO - Installing psutil>=3.1.1 into virtualenv Z:\task_1533716509\build\venv
08:56:25     INFO - retry: Calling run_command with args: [['Z:\\task_1533716509\\build\\venv\\Scripts\\pip', 'install', '--timeout', '120', '--no-index', '--find-links', 'https://pypi.pub.build.mozilla.org/pub', 'psutil>=3.1.1']], kwargs: {'error_level': 'warning', 'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x013CA750>, 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x014C9598>, 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0x01569F78>, 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x013F2800>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x01345E60>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x015BA020>, 'level': 'critical'}], 'cwd': 'Z:\\task_1533716509\\build', 'env': {'TMP': 'C:\\Users\\task_1533716509\\AppData\\Local\\Temp', 'USERNAME': 'task_1533716509', 'COMPUTERNAME': 'I-0FD1AC54743FD', 'TASKCLUSTER_INSTANCE_TYPE': 'c4.2xlarge', 'MOZ_AUTOMATION': '1', 'KTS_VERSION': '1.19c', 'PSMODULEPATH': 'C:\\windows\\system32\\WindowsPowerShell\\v1.0\\Modules\\;C:\\Program Files\\AWS Tools\\PowerShell\\', 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files', 'PROCESSOR_IDENTIFIER': 'x86 Family 6 Model 63 Stepping 2, GenuineIntel', 'TOOLTOOL_CACHE': 'y:\\tooltool-cache', 'PROCESSOR_REVISION': '3f02', 'DNSSUFFIX': 'use1.mozilla.com', 'SYSTEMROOT': 'C:\\windows', 'HOMEDRIVE': 'C:', 'WINDOWS_TRACING_FLAGS': '3', 'TEMP': 'C:\\Users\\task_1533716509\\AppData\\Local\\Temp', 'USERDOMAIN': 'I-0FD1AC54743FD', 'PROCESSOR_ARCHITECTURE': 'x86', 'ALLUSERSPROFILE': 'C:\\ProgramData', 'LOCALAPPDATA': 'Z:\\task_1533716509\\AppData\\Local', 'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/integration/mozilla-inbound', 'MOZILLABUILD': 'C:\\mozilla-build', 'SCCACHE_DISABLE': '1', 'LOGONSERVER': '\\\\I-0FD1AC54743FD', 'PROMPT': '$P$G', 'COMSPEC': 'C:\\windows\\system32\\cmd.exe', 'PROGRAMDATA': 'C:\\ProgramData', 'GECKO_HEAD_REV': '3e8c83bddae68cc637af82c1dbe8d792f2e287a8', 'PATH': 'C:\\Program Files\\Mercurial;C:\\mozilla-build\\7zip;C:\\mozilla-build\\info-zip;C:\\mozilla-build\\kdiff3;C:\\mozilla-build\\moztools-x64\\bin;C:\\mozilla-build\\mozmake;C:\\mozilla-build\\msys\\bin;C:\\mozilla-build\\msys\\local\\bin;C:\\mozilla-build\\nsis-3.0b3;C:\\mozilla-build\\nsis-2.46u;C:\\mozilla-build\\python;C:\\mozilla-build\\python\\Scripts;C:\\mozilla-build\\upx391w;C:\\mozilla-build\\wget;C:\\mozilla-build\\yasm;C:\\windows\\system32;C:\\windows;C:\\windows\\System32\\Wbem;C:\\windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\mozilla-build\\python27;C:\\mozilla-build\\python27\\Scripts;C:\\mozilla-build\\vim\\vim72;C:\\CoreUtils\\bin;C:\\mozilla-build\\buildbotve\\scripts;c:\\Program Files\\Microsoft Windows Performance Toolkit\\;C:\\Users\\cltbld\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files\\Puppet Labs\\Puppet\\bin;C:\\mozilla-build\\hg;C:\\Program Files\\GNU\\GnuPG\\pub;C:\\Program Files\\Mercurial\\', 'HOMEPATH': '\\Users\\task_1533716509', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC', 'FP_NO_HOST_CHECK': 'NO', 'DCLOCATION': 'SCL3', 'WINDOWS_TRACING_LOGFILE': 'C:\\BVTBin\\Tests\\installpackage\\csilogfile.log', 'TASK_ID': 'F47h50LxRf2-LRm8eM2bbw', 'PROGRAMFILES': 'C:\\Program Files', 'SYSTEMDRIVE': 'C:', 'KTS_HOME': 'C:\\Program Files\\KTS', 'NUMBER_OF_PROCESSORS': '8', 'APPDATA': 'Z:\\task_1533716509\\AppData\\Roaming', 'PIP_DOWNLOAD_CACHE': 'y:\\pip-cache', 'PROCESSOR_LEVEL': '6', 'USERPROFILE': 'C:\\Users\\task_1533716509', 'OS': 'Windows_NT', 'PUBLIC': 'C:\\Users\\Public', 'WINDIR': 'C:\\windows'}}, attempt #1
08:56:25     INFO - Running command: ['Z:\\task_1533716509\\build\\venv\\Scripts\\pip', 'install', '--timeout', '120', '--no-index', '--find-links', 'https://pypi.pub.build.mozilla.org/pub', 'psutil>=3.1.1'] in Z:\task_1533716509\build
08:56:25     INFO - Copy/paste: Z:\task_1533716509\build\venv\Scripts\pip install --timeout 120 --no-index --find-links https://pypi.pub.build.mozilla.org/pub psutil>=3.1.1
08:56:25     INFO - Using env: (same as previous command)
08:56:25     INFO -  Ignoring indexes: https://pypi.python.org/simple
08:56:25     INFO -  Collecting psutil>=3.1.1
08:56:27     INFO -    Downloading https://pypi.pub.build.mozilla.org/pub/psutil-5.4.3-cp27-none-win32.whl (220kB)
08:56:27     INFO -  Installing collected packages: psutil
08:56:28     INFO -  Successfully installed psutil-5.4.3

I'm guessing this is from here:
https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/testing/mozharness/mozharness/base/python.py#473

Presumably we have the wheel available on our pypi mirror. The non-automation case is a bit trickier because we don't want to fetch anything from the network, so we'd have to have it vendored.
Do we currently fail for a local command, because I removed the compiled binary for Windows with my patch on bug 1436857?
I don't think you made things any worse than they were before, but no, it doesn't work by default on Windows:
ted@DESKTOP-32V9ND8 /c/build/mozilla-central
$ ./mach python
Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\build\mozilla-central\build\mach_bootstrap.py", line 366, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "c:\build\mozilla-central\third_party\python\psutil\psutil\__init__.py", line 146, in <module>
    from . import _pswindows as _psplatform
  File "c:\build\mozilla-central\build\mach_bootstrap.py", line 366, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "c:\build\mozilla-central\third_party\python\psutil\psutil\_pswindows.py", line 16, in <module>
    from . import _psutil_windows as cext
ImportError: cannot import name _psutil_windows

The virtualenv that we set up in the objdir is populated from virtualenv_packages.txt, which lists psutil as needing to have setup.py run:
https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/build/virtualenv_packages.txt#24

The actions supported in that file are listed here:
https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/python/mozbuild/mozbuild/virtualenv.py#227

I suspect we'd need to extend that to support installing wheels via pip somehow, but leave ourselves a fallback to installing from source when we don't have a wheel.
After doing some research, I have a proposal:
1) Vendor a win64 psutil wheel and the matching sdist somewhere (maybe third_party/python_wheels, or we could just put the archives directly in third_party/python as long as we ensure that `mach vendor python` doesn't mess things up).
2) Extend virtualenv_packages.txt to allow pip installing a package, so that we wind up running `pip install --no-index --find-links=third_party/python psutil`. This will install psutil using a wheel if one is available for the platform, otherwise it will fall back to the sdist. AFAICT there's no way to convince pip to install a wheel but fall back to installing from a source directory, we have to give it an sdist.

We might want to extend `mach vendor python` to support #1 to make life easier. I don't know what that would look like, exactly, since we'd probably need to run `pip download` once per wheel and once for the sdist. `mach vendor python` currently runs `pip download` on a generated requirements.txt and passes `--no-binary :all:` to force sdist downloads:
https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/python/mozbuild/mozbuild/vendor_python.py#46

We can use `pip download` to fetch a win64 wheel on any platform like so:
pip download --only-binary :all: --platform win_amd64 --implementation cp --python-version 27 --abi none -d /tmp/wheels psutil==5.4.3

If we also want to vendor a wheel for Python 3 that looks like:
pip download --only-binary :all: --platform win_amd64 --implementation cp --python-version 36 --abi cp36m -d /tmp/wheels psutil==5.4.3
how much of the psutil api are we using, and can we write a compatible API that uses ffi?
Not much, but I looked into it and it'd be a bunch of really annoying verbose ctypes code to write. I'd rather just vendor a wheel and move on in life.
psutil on Windows has been a thorn in my side for too long.

I think we should bite the bullet and vendor binary wheels for Windows. I'm not crazy about that and wish there were a better way. But I think on the effort-success rate-pain curve, it is at a maxima.

Let's create a central directory for wheels - say third_party/python/wheelhouse - and change the package installation code to look for packages there. If we need to vendor any other binary wheels, we can deposit them there as well.

What I'm not sure about is if we'll get lucky with pip. If we did `pip install --no-index --find-links /path/to/wheelhouse <package>`, I'm pretty sure we'd need a source wheel or .tar.gz in that directory for psutil to get installed. i.e. I don't think there's a way to tell pip "install from a wheel if available or fall back to the setup.py in this path." So the logic for installing packages may have to be taught how to recognize when a wheel is present. This may entail calling into pip's internals for identifying wheel compatibility. It could be slightly ugly. But I still think it is less ugly than other solutions.
fwiw `pip install path-to-a.whl` works.
I did some more testing and have some more info.

(In reply to Gregory Szorc [:gps] from comment #8)
> What I'm not sure about is if we'll get lucky with pip. If we did `pip
> install --no-index --find-links /path/to/wheelhouse <package>`, I'm pretty
> sure we'd need a source wheel or .tar.gz in that directory for psutil to get
> installed. i.e. I don't think there's a way to tell pip "install from a
> wheel if available or fall back to the setup.py in this path." So the logic
> for installing packages may have to be taught how to recognize when a wheel
> is present. This may entail calling into pip's internals for identifying
> wheel compatibility. It could be slightly ugly. But I still think it is less
> ugly than other solutions.

Yeah, I mentioned this in comment 5 under bullet point 2. It's unfortunate but I don't know of a better way.

(In reply to Mike Hommey [:glandium] from comment #9)
> fwiw `pip install path-to-a.whl` works.

The downside to this is that we'd either have to rename the wheels to have more-guessable names, or hardcode the wheel filenames somewhere. I guess if we're only going to have the one win64 wheel it's not so bad, but if we wanted to have a Python 3 wheel as well it might start to get ugly.

So the other issue I found is that while I can `pip install` a wheel into the objdir virtualenv just fine, it doesn't actually help me with the task I'm trying to do, because the Python there is part of `mach`, and mach does not use the in-objdir virtualenv (because it gets created during configure, as part of the build). I don't know that we want to fight that chicken-and-egg battle really, which means our best bet would probably be vendoring unpacked wheels for the platforms we care about (maybe just py2-win64?), and then changing the mach bootstrap code to have some smarter behavior:
https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/build/mach_bootstrap.py#123-146

We could probably get away with like "put the unpacked win64 wheel in sys.path on windows, otherwise run setup.py like we currently do".
Individual `mach` commands can activate a virtualenv. We could change `mach build` to activate the build system virtualenv, which would have the side-effect of installing/activating the wheel.

For the wheel name, if we vendor wheels, we should stick to the official naming strategy. We /should/ be able to call into pip's internals to find candidate filenames for a given package then cross-reference that with the set of available files. Or we can just teach our package installation code that bare minimum logic necessary to deduce a wheel name for the platforms we care about (notably 64-bit Python on Windows).
Assignee: nobody → ted
psutil was previously vendored manually, so re-vendor it using our new tooling
to make for smaller diffs in follow-up patches. This mostly just winds up
adding some extra files that must have been left out of the manual vendoring,
but it also updates the root Pipfile + Pipfile.lock.
This option is very single-purpose: it's intended to let us vendor an unpacked
wheel for psutil on Windows. To that end the mach command will error if you
try to use it for anything but vendoring a single package. The mach command
will vendor source packages as it currently does, and then run `pip download`
again with some hardcoded parameters to fetch the right wheel for Python 2.7
on win64 and unpack it to a `package-platform` directory under
`third_party/python`.

I don't expect this to be used for anything but psutil, but it should make life
simpler for anyone that wants to update our vendored copy of psutil in the
future.
This change uses the previously added `--with-windows-wheel` option to vendor
a win64 wheel of psutil. This patch was produced by running:
`mach vendor python --with-windows-wheel psutil==5.4.3`.

We vendor this wheel unpacked so that we can insert it into sys.path in mach
without needing a virtualenv in which to install a wheel with pip, since we
have a chicken-and-egg problem there where configure creates the virtualenv
but we'd like to be able to use psutil before we run configure.
This patch adds two new actions to virtualenv_packages.txt processing:
windows and !windows. The former processes the rest of the action only on
Windows, and the latter processes it only on non-Windows.

Additionally, this patch adds support for running setup.py to mach_bootstrap's
handling of these actions, so that mach can build psutil from source on other
platforms.

Finally, these new features are used in virtualenv_packages.txt to use the
path to the unpacked Windows psutil wheel when on Windows, and build psutil
from source and use that path on other platforms.

This fixes the long-standing problem of not having psutil available on most
Windows systems (since they don't have the right set of Visual C++ build tools)
as well as the first-run problem on other systems where psutil would not be
available until the first build, since mach would not run the setup.py to
build it, and we relied on the VirtualenvManager doing so while populating
the objdir virtualenv.
After much fiddling around this is the solution I settled on. I don't think there's any perfect solution here, but this works and isn't really worse than the current state of affairs.
Comment on attachment 9001371 [details]
bug 1481612 - Re-vendor psutil 5.4.3 using `mach vendor python`. r?build

Gregory Szorc [:gps] has approved the revision.
Attachment #9001371 - Flags: review+
(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #12)
> psutil was previously vendored manually, so re-vendor it using our new
> tooling
> to make for smaller diffs in follow-up patches. This mostly just winds up
> adding some extra files that must have been left out of the manual vendoring,
> but it also updates the root Pipfile + Pipfile.lock.

We are going to vendor in only releases, right? So when I did the last update of psutils I picked a tar.gz and updated everything. So I wonder where those differences are coming from.
(In reply to Henrik Skupin (:whimboo) from comment #18)
> We are going to vendor in only releases, right? So when I did the last
> update of psutils I picked a tar.gz and updated everything. So I wonder
> where those differences are coming from.

Correct. I don't know! `mach vendor python` first uses pipenv to generate a full locked requirements.txt, then runs `pip download` on that requirements.txt to fetch source packages for everything, then unpacks them into third_party/python:
https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/python/mozbuild/mozbuild/vendor_python.py#45

I would assume that if you downloaded and untarred the source .tar.gz you would get the same result. Do you remember what file you used to vendor it previously?
Comment on attachment 9001372 [details]
bug 1481612 - Add a --with-windows-wheel option to mach vendor python. r=gps

Gregory Szorc [:gps] has approved the revision.
Attachment #9001372 - Flags: review+
Comment on attachment 9001373 [details]
bug 1481612 - vendor an unpacked win64 wheel of psutil 5.4.3. r=gps

Gregory Szorc [:gps] has approved the revision.
Attachment #9001373 - Flags: review+
Comment on attachment 9001374 [details]
bug 1481612 - Add more actions to virtualenv_packages.txt and use them to include the unpacked Windows psutil wheel. r=gps

Gregory Szorc [:gps] has approved the revision.
Attachment #9001374 - Flags: review+
This failed on try, unfortunately. The root cause is making mach's faux-virtualenv code run setup.py, because that means we'll be running setup.py every time a mach command executes. The reason for the breakage is that some mozharness command was trying to run something and capture its output, and it got confused by the extra output from setup.py building things. The other downside here is that we're probably wasting a bit of time doing this for every mach invocation. I'll have to think about this a bit.
From a Windows opt build on that latest try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=173456e1bcf289fa16570667b8987b9b564f2a68&selectedJob=204238964

15:13:58     INFO -  Overall system resources - Wall time: 2242s; CPU: 61%; Read bytes: 108540416; Write bytes: 30342580736; Read time: 8; Write time: 1212

Compared to an arbitrarily-chosen Windows opt build on inbound:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&selectedJob=204245060

15:53:47     INFO -  Overall system resources - Wall time: 2218s; CPU: 0%; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
I let this sit for a while because I was trying to figure out the right path to address comment 24. I revamped the topmost patch in this series to remove the "run setup.py in mach_bootstrap" part, which was what broke things on try. The successful try push in comment 25 shows that this fixes things. This does mean that it no longer fixes the first-run problem for non-Windows platforms, but I think we can live with that for now. I'll get the new version of that patch up for review and we can get this all landed.
(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #28)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=b7a8aec7d11e185d60b72b5f96afab6806809170

Just checking that everything still works OK after a rebase.
https://hg.mozilla.org/integration/mozilla-inbound/rev/c8f12706c4218440a04cbcb822f032306360d0cb
bug 1481612 - Re-vendor psutil 5.4.3 using `mach vendor python`. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/1ecd5ddcdd796533dc7d4f057280ae04da660a5a
bug 1481612 - Add a --with-windows-wheel option to mach vendor python. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/eb440bc9fed13e4ad4a1fe2ab1a1ed445b269300
bug 1481612 - vendor an unpacked win64 wheel of psutil 5.4.3. r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/de7e35a459e30fd5f68e0f1d085f67ec1bef3a88
bug 1481612 - Add more actions to virtualenv_packages.txt and use them to include the unpacked Windows psutil wheel. r=gps
This had to be backed out due to merge conflicts with bug 1490253. Feel free to rebase and re-land whenever.
Flags: needinfo?(ted)
Backout by ryanvm@gmail.com:
https://hg.mozilla.org/mozilla-central/rev/641b4a378923
Backed out 5 changesets (bug 1481612, bug 1483651) for merge conflicts with bug 1490253.
Attachment #9001371 - Attachment description: bug 1481612 - Re-vendor psutil 5.4.3 using `mach vendor python`. r?gps → bug 1481612 - Re-vendor psutil 5.4.3 using `mach vendor python`. r?build
Attachment #9001372 - Attachment description: bug 1481612 - Add a --with-windows-wheel option to mach vendor python. r?gps → bug 1481612 - Add a --with-windows-wheel option to mach vendor python. r=gps
Attachment #9001373 - Attachment description: bug 1481612 - vendor an unpacked win64 wheel of psutil 5.4.3. r?gps → bug 1481612 - vendor an unpacked win64 wheel of psutil 5.4.3. r=gps
Attachment #9001374 - Attachment description: bug 1481612 - Add more actions to virtualenv_packages.txt and use them to include the unpacked Windows psutil wheel. r?gps → bug 1481612 - Add more actions to virtualenv_packages.txt and use them to include the unpacked Windows psutil wheel. r=gps
I thought I had rebased this over davehunt's changes but apparently not. Only the first patch had conflicts, I've fixed that and will get re-review on it. The rest rebased cleanly without changes.
Flags: needinfo?(ted)
Pushed by tmielczarek@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/aafdbf2213ec
Re-vendor psutil 5.4.3 using `mach vendor python`. r=chmanchester,gps
https://hg.mozilla.org/integration/autoland/rev/60407ad13922
Add a --with-windows-wheel option to mach vendor python. r=gps
https://hg.mozilla.org/integration/autoland/rev/724343258461
vendor an unpacked win64 wheel of psutil 5.4.3. r=gps
https://hg.mozilla.org/integration/autoland/rev/ec133330b1ec
Add more actions to virtualenv_packages.txt and use them to include the unpacked Windows psutil wheel. r=gps
Duplicate of this bug: 1233251
You need to log in before you can comment on or make changes to this bug.