1612579 - [Windows] TypeError: environment can only contain strings when running `mach fluent-migration-test`

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, defect)

Description

[lifanfzeng]

Attachment: bug_1608199_styleeditor.py

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0

Steps to reproduce:

Note: I'm using the Windows Artifact Desktop Build.

When testing my Fluent migration script (see https://phabricator.services.mozilla.com/D59607) and testing it with

"python/l10n/fluent_migrations/bug_1608199_styleeditor.py"

(per https://firefox-source-docs.mozilla.org/l10n/migrations/testing.html)

Actual results:

$ ./mach fluent-migration-test python/l10n/fluent_migrations/bug_1608199_styleeditor.py
0:01.95 hg pull -u
Error running mach:

['fluent-migration-test', 'python/l10n/fluent_migrations/bug_1608199_styleeditor.py']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

TypeError: environment can only contain strings

File "c:\mozilla-source\mozilla-central\testing/mach_commands.py", line 844, in run_migration_tests
rv |= fmt.test_migration(self, obj_dir, **context)
File "c:\mozilla-source\mozilla-central\python/l10n\test_fluent_migrations\fmt.py", line 114, in test_migration
client.open()
File "c:\mozilla-source\mozilla-central\third_party/python/python-hglib\hglib\client.py", line 255, in open
self.server = util.popen(self._args, self._env)
File "c:\mozilla-source\mozilla-central\third_party/python/python-hglib\hglib\util.py", line 217, in popen
startupinfo=startupinfo, env=environ)
File "c:\mozilla-build\python\lib\subprocess.py", line 394, in init
errread, errwrite)
File "c:\mozilla-build\python\lib\subprocess.py", line 644, in _execute_child
startupinfo)

Expected results:

It should have ran the test.

Comment 1

[:Gijs (he/him)]

Andrew / :gbrown , any idea what happened here? Did we recently convert this code to use python3 ?

Comment 2

[Justin Wood (:Callek)]

Try pulling latest central and running again, I think this may be related to code that was landed and backed out

Comment 3

[:Gijs (he/him)]

(In reply to Justin Wood (:Callek) from comment #2)

Try pulling latest central and running again, I think this may be related to code that was landed and backed out

Lifan, can you try this (pull central and rebase your patch) ?

Comment 4

[Riley Byrd (:crownos)]

(In reply to :Gijs (he/him) from comment #3)

(In reply to Justin Wood (:Callek) from comment #2)

Try pulling latest central and running again, I think this may be related to code that was landed and backed out

Lifan, can you try this (pull central and rebase your patch) ?

Gijs, I just tried this on my bug and no dice. I even ran a full ./mach build, if that matters.
I'd add my output but its exactly the same as Lifan's minus the file name, of course.

Comment 5

[Geoff Brown [:gbrown]]

(In reply to :Gijs (he/him) from comment #1)

Andrew / :gbrown , any idea what happened here? Did we recently convert this code to use python3 ?

Sorry, I don't know of any recent related changes, and couldn't find an explanation from hg logs.

Comment 6

[:Gijs (he/him)]

Attachment: Bug 1612579 - escape environment values in use of hglib, r?gbrown

Comment 7

[:Gijs (he/him)]

I think this is basically the same issue as bug 1585258 but hglib rolls its own popen stuff, and so still has that bug.

I don't know what the process is for updating this bit of third party code. I've just copied the hack and that fixes things for me. Very open to other suggestions...

Comment 8

[Tom Prince [:tomprince]]

It looks like there is a newer version of hglib available, and https://www.mercurial-scm.org/repo/python-hglib/rev/22767a1e61ac looks at least vaguely related. We should at least try updating before patching. If we do patch, we should submit a bug report and/or patch upstream. (https://www.mercurial-scm.org/wiki/PythonHglib has some details on that)

mach vendor python python-hglib==2.6.1 should update the in-tree version (version number from https://pypi.org/project/python-hglib/)

Comment 9

[:Gijs (he/him)]

(In reply to Tom Prince [:tomprince] from comment #8)

It looks like there is a newer version of hglib available, and https://www.mercurial-scm.org/repo/python-hglib/rev/22767a1e61ac looks at least vaguely related.

The issue is with the environment passed to popen, not the args, so this doesn't look like it'll fix anything.

We should at least try updating before patching. If we do patch, we should submit a bug report and/or patch upstream. (https://www.mercurial-scm.org/wiki/PythonHglib has some details on that)

mach vendor python python-hglib==2.6.1 should update the in-tree version (version number from https://pypi.org/project/python-hglib/)

I checked tip and it looked like it had the same problems... I could be wrong, but to test it I'd need vendoring to work, and that is also broken:

 ./mach vendor python python-hglib==2.6.1
Processing .\third_party\python\pip-tools
Building wheels for collected packages: pip-tools
  Building wheel for pip-tools (setup.py) ... done
  Created wheel for pip-tools: filename=pip_tools-3.9.0-py2.py3-none-any.whl size=39969 sha256=6ba4e8ff47b0c724566c6b40436160608689465357db4071e933975a63fc8b68
  Stored in directory: $HOME\AppData\Local\pip\Cache\wheels\46\1a\9d\7eef3175b47298b8b9c37c089d75f88c381576ca914575ac7d
Successfully built pip-tools
Installing collected packages: pip-tools
Successfully installed pip-tools-3.9.0
Traceback (most recent call last):
  File "path\to\mozilla-build\python3\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "path\to\mozilla-build\python3\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "path\to\builds\frontend\_virtualenvs\init_py3\Scripts\pip-compile.exe\__main__.py", line 4, in <module>
  File "path\to\mozilla-central\build\mach_bootstrap.py", line 431, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "path\to\builds\frontend\_virtualenvs\init_py3\lib\site-packages\piptools\scripts\compile.py", line 11, in <module>
    from .._compat import install_req_from_line, parse_requirements
  File "path\to\mozilla-central\build\mach_bootstrap.py", line 431, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "path\to\builds\frontend\_virtualenvs\init_py3\lib\site-packages\piptools\_compat\__init__.py", line 7, in <module>
    from .pip_compat import (
  File "path\to\mozilla-central\build\mach_bootstrap.py", line 431, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "path\to\builds\frontend\_virtualenvs\init_py3\lib\site-packages\piptools\_compat\pip_compat.py", line 35, in <module>
    is_file_url = do_import("download", "is_file_url")
  File "path\to\builds\frontend\_virtualenvs\init_py3\lib\site-packages\piptools\_compat\pip_compat.py", line 24, in do_import
    return getattr(imported, package)
AttributeError: module 'pip._internal.download' has no attribute 'is_file_url'
Error running mach:

    ['vendor', 'python', 'python-hglib==2.6.1']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['d:/builds/frontend\\_virtualenvs\\init_py3\\Scripts\\pip-compile', '$HOME\\AppData\\Local\\Temp\\tmp0kf4z0p9', '--no-header', '--no-index', '--output-file', 'd:/mozilla-central/third_party/python\\requirements.txt', '--generate-hashes']' returned non-zero exit status 1.

  File "path\to\mozilla-central\python/mozbuild/mozbuild/mach_commands.py", line 1376, in vendor_python
    vendor_command.vendor(**kwargs)
  File "path\to\mozilla-central\python/mozbuild\mozbuild\vendor_python.py", line 51, in vendor
    '--generate-hashes'])
  File "path\to\mozilla-build\python3\lib\subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "path\to\mozilla-build\python3\lib\subprocess.py", line 418, in run
    output=stdout, stderr=stderr)

any suggestions on how to fix this?

Comment 10

[:Gijs (he/him)]

(I tried updating pip, that didn't help.)

Comment 11

[Axel Hecht [:Pike]]

The vendoring problem is bug 1602771. And yes, locally patching hglib smells bad.

Comment 12

[Axel Hecht [:Pike]]

Attachment: Bug 1612579, use python3 for fluent-migration-test

Comment 13

[Axel Hecht [:Pike]]

Reading up in bug 1585258, it seems that python2 is the problem, so I checked if we can just run in python3. Which works for me locally, and only required a small change to the json logging handler.

I'd love to see if this works on Windows, also if I broke the tree with the handler change.

Comment 14

[:Gijs (he/him)]

(In reply to Axel Hecht [:Pike] from comment #13)

Reading up in bug 1585258, it seems that python2 is the problem, so I checked if we can just run in python3. Which works for me locally, and only required a small change to the json logging handler.

I'd love to see if this works on Windows

Yes, it works. Thanks!

also if I broke the tree with the handler change.

I don't know; should the handler change be conditional on whether we're using python2 or python3 at that point (ie something like "w" if six.PY3 else "wb" or whatever) ?

Comment 15

[Axel Hecht [:Pike]]

Thanks for testing. Given that the builds are running on all three platforms now, I think this is ready for review.

Also dropping the NI on ahal, I think we're good here.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/06b22c1c196f
use python3 for fluent-migration-test r=firefox-build-system-reviewers,rstewart

Comment 17

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/06b22c1c196f