1585702 - [mozprocess] "ValueError: embedded null character" on Windows with Python 3

Status: RESOLVED FIXED

(Testing :: Mozbase, defect, P2)

Description

[Andrew Halberstadt [:ahal]]

This happens due to a regression in Python itself:
https://bugs.python.org/issue32745

This gets raised by LPCWSTR here:
https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/winprocess.py#167

However according to the winapi docs the format needs to be:

key=value\0

So in other words, CreateProcess is expecting null characters, but ctypes' LPCWSTR now prohibits them. I haven't found a workaround yet.

Comment 1

[Andrew Halberstadt [:ahal]]

Turns out we had already discovered this issue and the tests are all skipped on Python 3 under Windows because of it:
https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/tests/manifest.ini#3

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1585702 - [mozprocess] Temporarily disable 'env' default on Windows with Python 3, r?gbrown

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6050b9e662ed
[mozprocess] Temporarily disable 'env' default on Windows with Python 3, r=gbrown

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1585702 - [mozprocess] Fix "Embedded null character" error in Windows with Python 3, r?gbrown

This works around a bug in Python:
https://bugs.python.org/issue32745

Null characters aren't allowed in 'c_wchar_p' types anymore, but we can get around
the issue by allocating a buffer in memory and casting it after the fact. This was
discovered via trial and error and I'm not really sure why it works.. But it does.

This also enables the tests under Python 3 on Windows (which thankfully all
seem to pass).

Comment 5

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/6050b9e662ed

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1585702 - [lint] Remove temporary hacks now that 'null character' issue is fixed, r?gbrown

Depends on D48113

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9ea7f7d17686
[mozprocess] Fix "Embedded null character" error in Windows with Python 3, r=gbrown
https://hg.mozilla.org/integration/autoland/rev/e4085fae0f8b
[lint] Remove temporary hacks now that 'null character' issue is fixed, r=gbrown

Comment 8

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/9ea7f7d17686
https://hg.mozilla.org/mozilla-central/rev/e4085fae0f8b