1689175 - Intermittent [tier2] minGW IOError: [Errno 2] No such file or directory: '/builds/worker/workspace/venv/pyvenv.cfg'

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: abutkovits [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=328001291&repo=mozilla-esr78
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/asdUOQqrRoi0eSdICRWfOQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-01-27T17:49:15.303Z] + /builds/worker/checkouts/gecko/mach python /builds/worker/checkouts/gecko/testing/mozharness/scripts/fx_desktop_build.py --config builds/releng_base_firefox.py --config builds/releng_base_windows_32_mingw_builds.py --config builds/releng_sub_windows_configs/32_mingwclang.py --build --log-level=debug --work-dir=/builds/worker/workspace --branch=mozilla-esr78 --build-pool=taskcluster
[task 2021-01-27T17:49:16.261Z] created virtual environment CPython2.7.13.final.0-64 in 251ms
[task 2021-01-27T17:49:16.261Z]   creator CPython2Posix(dest=/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init, clear=False, global=False)
[task 2021-01-27T17:49:16.261Z]   seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/builds/worker/.local/share/virtualenv)
[task 2021-01-27T17:49:16.261Z]     added seed packages: pip==20.2.2, setuptools==44.1.1, wheel==0.35.1
[task 2021-01-27T17:49:16.261Z]   activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
[task 2021-01-27T17:49:16.564Z] WARNING: Python.h not found. Install Python development headers.
[task 2021-01-27T17:49:16.564Z] Error processing command. Ignoring because optional. (optional:setup.py:third_party/python/psutil:build_ext:--inplace)
[task 2021-01-27T17:49:16.567Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2021-01-27T17:49:16.687Z] 17:49:16     INFO - ConsoleLogger online at 20210127 17:49:16Z in /builds/worker
[task 2021-01-27T17:49:16.687Z] 17:49:16     INFO - Script was run with '--build-pool taskcluster' and 'taskcluster' matches a key in 'builds/build_pool_specifics.py'. Updating self.config with items from that key's value.
[task 2021-01-27T17:49:16.687Z] 17:49:16     INFO - To generate a config file based upon options passed and config files used, run script as before but extend options with "--dump-config"
[task 2021-01-27T17:49:16.687Z] 17:49:16     INFO - For a diff of where self.config got its items, run the script again as before but extend options with: "--dump-config-hierarchy"
[task 2021-01-27T17:49:16.687Z] 17:49:16     INFO - Both --dump-config and --dump-config-hierarchy don't actually run any actions.
[task 2021-01-27T17:49:16.687Z] 17:49:16     INFO - Run as /builds/worker/checkouts/gecko/testing/mozharness/scripts/fx_desktop_build.py --config builds/releng_base_firefox.py --config builds/releng_base_windows_32_mingw_builds.py --config builds/releng_sub_windows_configs/32_mingwclang.py --build --log-level=debug --work-dir=/builds/worker/workspace --branch=mozilla-esr78 --build-pool=taskcluster
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - ENV: HG_SHARE_BASE_DIR is now /builds/hg-shared
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - ENV: LC_ALL is now C
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - ENV: MOZ_OBJDIR is now /builds/worker/workspace/obj-build
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - ENV: MOZBUILD_STATE_PATH is now /builds/worker/.mozbuild
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - ENV: TINDERBOX_OUTPUT is now 1
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - ENV: TOOLTOOL_CACHE is now /builds/worker/tooltool-cache
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - ENV: PATH is now /usr/local/bin:/bin:/usr/bin
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - ENV: MOZ_CRASHREPORTER_NO_REPORT is now 1
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - ENV: TOOLTOOL_HOME is now /builds
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - ENV: DISPLAY is now :2
[task 2021-01-27T17:49:16.688Z] 17:49:16     INFO - Creating virtualenv /builds/worker/workspace/venv
[task 2021-01-27T17:49:16.688Z] 17:49:16    DEBUG - mkdir_p: /builds/worker/workspace Already exists.
[task 2021-01-27T17:49:16.688Z] 17:49:16     INFO - Running command: ['/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', '/builds/worker/checkouts/gecko/testing/mozharness/external_tools/virtualenv/virtualenv.py', '--always-copy', '/builds/worker/workspace/venv'] in /builds/worker/workspace
[task 2021-01-27T17:49:16.688Z] 17:49:16     INFO - Copy/paste: /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python /builds/worker/checkouts/gecko/testing/mozharness/external_tools/virtualenv/virtualenv.py --always-copy /builds/worker/workspace/venv
[task 2021-01-27T17:49:16.689Z] 17:49:16     INFO - Using partial env: {'VIRTUALENV_NO_DOWNLOAD': '1'}
[task 2021-01-27T17:49:16.689Z] 17:49:16    DEBUG - ENV: VIRTUALENV_NO_DOWNLOAD is now 1
[task 2021-01-27T17:49:16.740Z] 17:49:16     INFO -  Using real prefix '/usr'
[task 2021-01-27T17:49:16.766Z] 17:49:16     INFO -  New python executable in /builds/worker/workspace/venv/bin/python
[task 2021-01-27T17:49:16.778Z] 17:49:16    ERROR -  Traceback (most recent call last):
[task 2021-01-27T17:49:16.778Z] 17:49:16     INFO -    File "/builds/worker/workspace/venv/lib/python2.7/site.py", line 165, in <module>
[task 2021-01-27T17:49:16.778Z] 17:49:16     INFO -      try:
[task 2021-01-27T17:49:16.778Z] 17:49:16     INFO -    File "/builds/worker/workspace/venv/lib/python2.7/site.py", line 12, in main
[task 2021-01-27T17:49:16.779Z] 17:49:16     INFO -    File "/builds/worker/workspace/venv/lib/python2.7/site.py", line 65, in read_pyvenv
[task 2021-01-27T17:49:16.779Z] 17:49:16     INFO -  IOError: [Errno 2] No such file or directory: '/builds/worker/workspace/venv/pyvenv.cfg'
[task 2021-01-27T17:49:16.779Z] 17:49:16     INFO -  ERROR: The executable /builds/worker/workspace/venv/bin/python is not functioning
[task 2021-01-27T17:49:16.779Z] 17:49:16     INFO -  ERROR: It thinks sys.prefix is u'/builds/worker/workspace' (should be u'/builds/worker/workspace/venv')
[task 2021-01-27T17:49:16.779Z] 17:49:16     INFO -  ERROR: virtualenv is not compatible with this system or executable
[task 2021-01-27T17:49:16.784Z] 17:49:16    ERROR - Return code: 100
[task 2021-01-27T17:49:16.784Z] 17:49:16    ERROR - 100 not in success codes: [0]
[task 2021-01-27T17:49:16.784Z] 17:49:16    ERROR - failures found while parsing output
[task 2021-01-27T17:49:16.784Z] 17:49:16  WARNING - setting return code to 2
[task 2021-01-27T17:49:16.784Z] 17:49:16    FATAL - Halting on failure while running ['/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', '/builds/worker/checkouts/gecko/testing/mozharness/external_tools/virtualenv/virtualenv.py', '--always-copy', '/builds/worker/workspace/venv']
[task 2021-01-27T17:49:16.784Z] 17:49:16    FATAL - Running post_fatal callback...
[task 2021-01-27T17:49:16.784Z] 17:49:16    FATAL - Exiting 2
[fetches 2021-01-27T17:49:16.811Z] removing /builds/worker/fetches
[fetches 2021-01-27T17:49:17.538Z] finished
[taskcluster 2021-01-27 17:49:21.705Z] === Task Finished ===
[taskcluster 2021-01-27 17:49:21.913Z] Artifact "public/build" not found at "/builds/worker/artifacts/"
[taskcluster 2021-01-27 17:49:22.023Z] Unsuccessful task run with exit code: 2 completed in 78.21 seconds```

Comment 1

[Ryan VanderMeulen [:RyanVM]]

This is fallout from all the virtualenv uplifts. It's bizarre to me that this is happening intermittently (and I've been able to rerun failed jobs to green). Are we missing an uplift somewhere?

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

This seems to consistently behave once the following are uplifted:

• Use pip list --format freeze instead of pip freeze. r=ahal
• Remove mozharness' copy of virtualenv and use the one under /third_party/python instead r=ahal,rail

I'll get a fix that works with this and 1689184, then submit uplift requests/patches.

Comment 3

[Intermittent Failures Robot]

5 failures in 3891 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr78: 5

Platform and build breakdown:

• windows-mingw32: 5
  • all: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1689175&startday=2021-01-25&endday=2021-01-31&tree=all

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Fixed by the latest round of uplifts.