1762050 - virtualenv accessed from a different path than it was created when PGO is enabled

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[matias]

When MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=none is set in the environment and ac_add_options MOZ_PGO=1 is set in .mozconfig, I get the following output from mach build:

created virtual environment CPython3.10.4.final.0-64 in 46ms
  creator CPython3Posix(dest=/home/main-builder/pkgwork/src/mozilla-unified/obj-x86_64-pc-linux-gnu/_virtualenvs/build, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
 0:01.06 Clobber not needed.
 Config object not found by mach.
"/home/main-builder/pkgwork/src/mozilla-unified/obj-x86_64-pc-linux-gnu/instrumented/_virtualenvs/build" does not exist
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file build| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: The "build" site is out-of-date, even though it has already been activated. Was it modified while this Mach process was running?

  File "/home/main-builder/pkgwork/src/mozilla-unified/python/mozbuild/mozbuild/build_commands.py", line 178, in build
    status = instr.build(
  File "/home/main-builder/pkgwork/src/mozilla-unified/python/mozbuild/mozbuild/controller/building.py", line 1296, in build
    config_rc = self.configure(
  File "/home/main-builder/pkgwork/src/mozilla-unified/python/mozbuild/mozbuild/controller/building.py", line 1668, in configure
    build_site.ensure()
  File "/home/main-builder/pkgwork/src/mozilla-unified/python/mach/mach/site.py", line 574, in ensure
    raise Exception(

it creates the virtualenv inside a directory but tries to access it from another directory, probably because after the virtualenv is created, topobjdir is changed at mozbuild/build_commands.py:

    if doing_pgo:
        if what:
            raise Exception("Cannot specify targets (%s) in MOZ_PGO=1 builds" % what)
        instr = command_context._spawn(BuildDriver)
        orig_topobjdir = instr._topobjdir
        instr._topobjdir = mozpath.join(instr._topobjdir, "instrumented")

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1762050: Use original topobjdir for PGO build virtualenv

The PGO-specific objdir is meant for the PGO build artifacts.
The python virtualenvs just exist for the host, so there's no need to
scope them. Besides, the existing virtualenv infrastructure doesn't know
how to distinguish between "main 'build' site" and "PGO 'build' site",
which is why it raised this "out-of-date" error.

Comment 2

[matias]

After applying patch D142373, build starts successfully, but fails after everything has been compiled, when trying to run build/pgo/profileserver.py:

[...]
64:29.57 We know it took a while, but your build finally finished successfully!
[...]
65:27.24 make[3]: Leaving directory '/home/main-builder/pkgwork/src/mozilla-unified/obj-x86_64-pc-linux-gnu/instrumented/browser/locales'
65:27.24 make[2]: Leaving directory '/home/main-builder/pkgwork/src/mozilla-unified/obj-x86_64-pc-linux-gnu/instrumented/browser/installer'
65:27.26 make[1]: Leaving directory '/home/main-builder/pkgwork/src/mozilla-unified/obj-x86_64-pc-linux-gnu/instrumented/browser/installer'
65:27.26 make: Leaving directory '/home/main-builder/pkgwork/src/mozilla-unified/obj-x86_64-pc-linux-gnu/instrumented'
 Config object not found by mach.
Configure complete!
Be sure to run |mach build| to pick up any changes
To view resource usage of the build, run |mach resource-usage|.
To take your build for a test drive, run: |mach run|
For more information on what to do now, see https://firefox-source-docs.mozilla.org/setup/contributing_code.html
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file build| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

FileNotFoundError: [Errno 2] No such file or directory: '/home/main-builder/pkgwork/src/mozilla-unified/obj-x86_64-pc-linux-gnu/instrumented/_virtualenvs/build/bin/python'

  File "/home/main-builder/pkgwork/src/mozilla-unified/python/mozbuild/mozbuild/build_commands.py", line 215, in build
    subprocess.check_call(pgo_cmd, cwd=instr.topobjdir, env=pgo_env)
  File "/usr/lib/python3.10/subprocess.py", line 364, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 345, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Ah, good catch, thanks!
I've got a patch update here, tested locally and now it should happily PGO.
Thanks Matias :)

Comment 4

[matias]

Thanks a lot. New patch confirmed working :D

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/18d2da487a8d
Use original topobjdir for PGO build virtualenv r=glandium

Comment 6

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/18d2da487a8d