1670130 - "too many values to unpack" inside processhandler.py when running `mach configure`

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Kagami Rosylight [:saschanaz] (they/them)]

ValueError: too many values to unpack (expected 18)

  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mozbuild/mozbuild/build_commands.py", line 125, in build
    return driver.build(
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mozbuild\mozbuild\controller\building.py", line 1220, in build
    status = self._run_client_mk(line_handler=output.on_line,
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mozbuild\mozbuild\controller\building.py", line 1568, in _run_client_mk
    return self._run_make(srcdir=True,
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mozbuild\mozbuild\base.py", line 762, in _run_make
    return fn(**params)
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mozbuild\mozbuild\base.py", line 813, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\mixin\process.py", line 135, in run_process
    p.run()
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\testing/mozbase/mozprocess\mozprocess\processhandler.py", line 842, in run
    self.proc = self.Process([self.cmd] + self.args, **args)
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\testing/mozbase/mozprocess\mozprocess\processhandler.py", line 141, in __init__
    subprocess.Popen.__init__(self, args, **kwargs)
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\third_party/python/sentry-sdk\sentry_sdk\integrations\stdlib.py", line 190, in sentry_patched_popen_init
    rv = old_popen_init(self, *a, **kw)  # type: ignore
  File "c:\mozilla-build\python39\lib\subprocess.py", line 947, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\testing/mozbase/mozprocess\mozprocess\processhandler.py", line 270, in _execute_child
    (args, executable, preexec_fn, close_fds,
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-Break to quit

The value included 22 items:

(
  ['c:\\mozilla-build\\bin\\mozmake.EXE', '-f', 'client.mk', '-s', 'configure'],
  None,
  None,
  False,
  (),
  'C:/Users/Kagami/Documents/GitHub/gecko-dev',
  {'ALLUSERSPROFILE': 'C:\\ProgramData', 'APPDATA': 'C:\\Users\\Kagami\\AppData\\Roaming', 'COMMONPROGRAMFILES(X86)': 'C:\\Program Files (x86)\\Common Files', 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files', 'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files', 'COMPUTERNAME': 'SURFINAZ-7', 'COMSPEC': 'C:\\WINDOWS\\system32\\cmd.exe', 'DRIVERDATA': 'C:\\Windows\\System32\\Drivers\\DriverData', 'EDITOR': 'emacs.exe', 'EMSDK_NODE': 'C:/Users/Kagami/Documents/GitHub/emsdk/node/12.9.1_64bit/bin/node.exe', 'EMSDK_PYTHON': 'C:/Users/Kagami/Documents/GitHub/emsdk/python/3.7.4_64bit/python.exe', 'GETTEXTCLDRDIR': 'C:\\Program Files\\gettext-iconv\\lib\\gettext', 'GIT_SSH': 'c:/Windows/System32/OpenSSH/ssh.exe', 'GSTREAMER_1_0_ROOT_X86_64': 'C:\\gstreamer\\1.0\\x86_64\\', 'HISTFILE': 'C:/mozilla-build/msys/home/Kagami/.bash_history', 'HOME': 'c:/Users/Kagami', 'HOMEDRIVE': 'C:', 'HOMEPATH': '\\', 'HOSTTYPE': 'i686', 'INPUTRC': 'C:/mozilla-build/msys/etc/inputrc', 'JAVA_HOME': 'C:/Users/Kagami/Documents/GitHub/emsdk/java/8.152_64bit', 'LOCALAPPDATA': 'C:\\Users\\Kagami\\AppData\\Local', 'LOCAL_GIT_DIRECTORY': 'C:\\Users\\Kagami\\AppData\\Local\\GitHubDesktop\\app-2.5.6\\resources\\app\\git', 'LOGNAME': 'Kagami', 'LOGONSERVER': '\\\\SURFINAZ-7', 'MACHTYPE': 'i686-pc-msys', 'MAKE_MODE': 'unix', 'MOZILLABUILD': 'C:\\mozilla-build\\', 'MOZ_NO_RESET_PATH': '1', 'MSYSTEM': 'MINGW32', 'NUMBER_OF_PROCESSORS': '8', 'ONEDRIVE': 'C:\\Users\\Kagami\\OneDrive', 'ONEDRIVECONSUMER': 'C:\\Users\\Kagami\\OneDrive', 'ORIGINAL_XDG_CURRENT_DESKTOP': 'undefined', 'OS': 'Windows_NT', 'OSTYPE': 'msys', 'PATH': 'c:\\Users\\Kagami\\.mozbuild\\_virtualenvs\\mach\\Scripts;C:\\mozilla-build\\msys\\local\\bin;c:\\mozilla-build\\bin;c:\\mozilla-build\\kdiff3;c:\\mozilla-build\\nsis-3.01;c:\\mozilla-build\\python39;c:\\mozilla-build\\python39\\Scripts;c:\\mozilla-build\\python3;c:\\mozilla-build\\python3\\Scripts;c:\\mozilla-build\\python;c:\\mozilla-build\\python\\Scripts;.;C:\\mozilla-build\\msys\\local\\bin;C:\\mozilla-build\\msys\\mingw\\bin;C:\\mozilla-build\\msys\\bin;c:\\mozilla-build\\python39\\Scripts\\;c:\\mozilla-build\\python39\\;c:\\WINDOWS\\system32;c:\\WINDOWS;c:\\WINDOWS\\System32\\Wbem;c:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;c:\\WINDOWS\\System32\\OpenSSH\\;c:\\PHP;c:\\Program Files\\CMake\\bin;c:\\Program Files\\dotnet\\;c:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\;c:\\Program Files\\gettext-iconv\\bin;c:\\Program Files\\LLVM\\bin;c:\\Program Files\\PowerShell\\7\\;c:\\Program Files\\nodejs\\;c:\\Program Files\\Git\\cmd;c:\\Users\\Kagami\\AppData\\Local\\Programs\\Python\\Python39\\Scripts\\;c:\\Users\\Kagami\\AppData\\Local\\Programs\\Python\\Python39\\;c:\\Users\\Kagami\\AppData\\Local\\Programs\\Python\\Launcher\\;c:\\mozilla-build\\python38\\Scripts\\;c:\\mozilla-build\\python38\\;c:\\Users\\Kagami\\Documents\\GitHub\\emsdk;c:\\Users\\Kagami\\Documents\\GitHub\\emsdk\\java\\8.152_64bit\\bin;c:\\Users\\Kagami\\Documents\\GitHub\\emsdk\\upstream\\emscripten;c:\\Program Files\\PowerShell\\7;c:\\WINDOWS\\system32;c:\\WINDOWS;c:\\WINDOWS\\System32\\Wbem;c:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;c:\\WINDOWS\\System32\\OpenSSH\\;c:\\Program Files\\Git\\cmd;c:\\Program Files\\PowerShell\\7\\;c:\\PHP;c:\\Program Files\\nodejs\\;c:\\Users\\Kagami\\.cargo\\bin;c:\\Users\\Kagami\\AppData\\Local\\Microsoft\\WindowsApps;c:\\Users\\Kagami\\AppData\\Local\\GitHubDesktop\\bin;c:\\Users\\Kagami\\AppData\\Local\\Programs\\Microsoft VS Code\\bin;c:\\Users\\Kagami\\.mozbuild\\sccache;c:\\Users\\Kagami\\Documents\\GitHub\\arcanist\\bin;c:\\Users\\Kagami\\.mozbuild\\git-cinnabar;c:\\ffmpeg\\bin;c:\\ninja;c:\\Users\\Kagami\\.deno\\bin;c:\\mozilla-build\\python;c:\\Program Files\\NASM;c:\\Users\\Kagami\\AppData\\Roaming\\npm;c:\\Program Files\\Git\\cmd\\', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW', 'POWERSHELL_DISTRIBUTION_CHANNEL': 'MSI:Windows 10 Pro', 'PROCESSOR_ARCHITECTURE': 'AMD64', 'PROCESSOR_IDENTIFIER': 'Intel64 Family 6 Model 126 Stepping 5, GenuineIntel', 'PROCESSOR_LEVEL': '6', 'PROCESSOR_REVISION': '7e05', 'PROGRAMDATA': 'C:\\ProgramData', 'PROGRAMFILES(X86)': 'C:\\Program Files (x86)', 'PROGRAMFILES': 'C:\\Program Files', 'PROGRAMW6432': 'C:\\Program Files', 'PROMPT': '$P$G', 'PSMODULEPATH': 'C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules', 'PUBLIC': 'C:\\Users\\Public', 'PWD': 'c:/Users/Kagami/Documents/GitHub/gecko-dev', 'PYTHON2': 'C:\\mozilla-build\\python\\python2.exe -m', 'PYTHON3': 'c:/mozilla-build/python39/python.exe', 'SESSIONNAME': 'Console', 'SHELL': 'C:/mozilla-build/msys/bin/sh', 'SHLVL': '1', 'SSH_AGENT_PID': '16304', 'SSH_AUTH_SOCK': 'C:/Users/Kagami/AppData/Local/Temp/ssh-QSFhy17312/agent.17312', 'SYSTEMDRIVE': 'C:', 'SYSTEMROOT': 'C:\\WINDOWS', 'TEMP': 'C:/Users/Kagami/AppData/Local/Temp', 'TERM': 'cygwin', 'TMP': 'C:/Users/Kagami/AppData/Local/Temp', 'USERDOMAIN': 'SURFINAZ-7', 'USERDOMAIN_ROAMINGPROFILE': 'SURFINAZ-7', 'USERNAME': 'Kagami', 'USERPROFILE': 'C:\\Users\\Kagami', 'WINDIR': 'C:\\WINDOWS', 'WSLENV': 'WT_SESSION::WT_PROFILE_ID', 'WT_PROFILE_ID': '{617cd692-4ce5-4b14-a003-7d9ee59b0dd6}', 'WT_SESSION': '43b56de3-00ce-41e8-ba20-131d1ea6ba3c', 'MACH_MAIN_PID': '11596', 'MACH_STDOUT_ISATTY': '1', 'CONFIGURE_ARGS': '', 'TOPSRCDIR': 'C:/Users/Kagami/Documents/GitHub/gecko-dev', 'CONFIG_GUESS': 'x86_64-pc-mingw32', 'OBJDIR': 'C:/Users/Kagami/Documents/GitHub/gecko-dev/obj-x86_64-pc-mingw32', 'MACH': '1', 'PYTHONUNBUFFERED': '1'},
  None,
  0,
  False,
  Handle(724),
  -1,
  4,
  Handle(396),
  -1,
  Handle(284),
  True,
  None,
  None,
  None,
  -1,
  False
)

Adding 4 placeholders ,_,_,_,_ before = args_tuple fixes the issue.

Comment 1

[Ricky Stewart]

Are you using Python 3.9?

Comment 2

[Ricky Stewart]

I assume this patch will fix the problem, please confirm :)

diff --git a/testing/mozbase/mozprocess/mozprocess/processhandler.py b/testing/mozbase/mozprocess/mozprocess/processhandler.py
index a7bd82cceb46..c190a7ec9eb2 100644
--- a/testing/mozbase/mozprocess/mozprocess/processhandler.py
+++ b/testing/mozbase/mozprocess/mozprocess/processhandler.py
@@ -266,7 +266,19 @@ class ProcessHandlerMixin(object):
         if isWin:
             # Redefine the execute child so that we can track process groups
             def _execute_child(self, *args_tuple):
-                if six.PY3:
+                # workaround for bug 1670130
+                if sys.hexversion >= 0x03090000:  # after 3.9.0
+                    (args, executable, preexec_fn, close_fds,
+                     pass_fds, cwd, env,
+                     startupinfo, creationflags, shell,
+                     p2cread, p2cwrite,
+                     c2pread, c2pwrite,
+                     errread, errwrite,
+                     restore_signals,
+                     gid, gids, uid,
+                     umask,
+                     start_new_session) = args_tuple
+                elif six.PY3:
                     (args, executable, preexec_fn, close_fds,
                      pass_fds, cwd, env,
                      startupinfo, creationflags, shell,

Comment 3

[Ricky Stewart]

Attachment: Bug 1670130 - Fix contract of `_execute_child()` in `ProcessHandlerMixin` under Python 3.9 on Windows

This updates the signature to match the actual signature which in 3.9 gained 3 additional arguments: gid, gids, uid, and umask (see the source).

Comment 4

[Kagami Rosylight [:saschanaz] (they/them)]

Yes that works 😀

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ec24a434f5ea
Fix contract of `_execute_child()` in `ProcessHandlerMixin` under Python 3.9 on Windows r=firefox-build-system-reviewers,mhentges

Comment 6

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/ec24a434f5ea