1597687 - [mozprocess] "TypeError: write() argument must be str, not bytes" with Python 3

Status: RESOLVED DUPLICATE of bug 1623701

(Testing :: Mozbase, defect, P3)

Description

[Rishi Gupta (:rgpt)]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0

Steps to reproduce:

ran specific test in marionette harness to make it run on py3.
$ python3 mach test testing/marionette/harness/marionette_harness/tests/unit/test_accessibility.py

Actual results:

mozprocess throwed an error:

0:02.08 INFO Using workspace for temporary data: "/home/rgpt/Desktop/mozilla/mozilla-unified"
0:02.09 mozversion INFO application_buildid: 20191119194152
0:02.09 mozversion INFO application_changeset: 6efd0680f02a7911423b23246b293bf56152efe3
0:02.09 mozversion INFO application_display_name: Nightly
0:02.09 mozversion INFO application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
0:02.09 mozversion INFO application_name: Firefox
0:02.09 mozversion INFO application_remotingname: firefox
0:02.09 mozversion INFO application_vendor: Mozilla
0:02.09 mozversion INFO application_version: 72.0a1
0:02.09 mozversion INFO platform_buildid: 20191119194152
0:02.09 mozversion INFO platform_changeset: 6efd0680f02a7911423b23246b293bf56152efe3
0:02.09 mozversion INFO platform_version: 72.0a1
0:02.10 INFO Application command: /home/rgpt/Desktop/mozilla/mozilla-unified/objdir-frontend/dist/bin/firefox -no-remote -marionette -profile /tmp/tmpxlh10_0r.mozrunner
Exception in thread ProcessReader:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/rgpt/Desktop/mozilla/mozilla-unified/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1085, in _read
callback(line.rstrip())
File "/home/rgpt/Desktop/mozilla/mozilla-unified/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 995, in call
e(*args, **kwargs)
File "/home/rgpt/Desktop/mozilla/mozilla-unified/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1136, in call
self.stream.write(line + '\n'.encode('utf8'))
TypeError: write() argument must be str, not bytes

Expected results:

It should have worked on py3 also as it does on py2

Comment 1

[Rishi Gupta (:rgpt)]

@whimboo, encoding on https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/processhandler.py#1136 have some history behind it. As based on 991866, i am concerned that whether removing encoding will produce some issue. and if encoding is to be removed then except block will be nullified.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Please note that the second half of the first comment is not related to this bug. This is the TypeError: unorderable types: NoneType() > int() part. Please edit your comment and remove that to lower the confusion.

Is line in this call to write() actually a utf-8 encoded string or str?

I don't have that much knowledge in this area so getting feedback from Andrew, who reviewed some of those former patches, might be the best choice.

Comment 3

[Andrew Halberstadt [:ahal]]

I'm not sure why we're encoding that \n literal. It should already be bytes on Python 2 and text on Python 3. So first, get rid of that. Next we can make sure to decode if a binary string is detected:

if six.PY3 and isinstance(line, six.binary_type):
    line = line.decode('utf-8')

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hey Rishi, can you try Andrew's suggestion? Does it fix the problem for you? Also please file the other issue as a new bug if it still exists with that patch added. Thanks.

Comment 5

[Rishi Gupta (:rgpt)]

As discussed on IRC, it fixed my problem. Thank you @ahal and @whimboo

Comment 6

[Rishi Gupta (:rgpt)]

Attachment: Bug 1597687: removing encoding of new line char and decoding if binary string is there r=ahal

Comment 7

[Wolfgang Rosenauer [:wolfiR]]

The code in mozilla77 looks like this

def __call__(self, line):
    ensure = six.ensure_text if self.text else six.ensure_binary
    self.stream.write(ensure(line) + ensure('\n'))
    self.stream.flush()

But my build still fails with

[ 5178s] ERROR:root:a bytes-like object is required, not 'str'
[ 5179s] Exception in thread ProcessReader:
[ 5179s] Traceback (most recent call last):
[ 5179s] File "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
[ 5179s] self.run()
[ 5179s] File "/usr/lib64/python3.8/threading.py", line 870, in run
[ 5179s] self._target(*self._args, **self._kwargs)
[ 5179s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1116, in _read
[ 5179s] callback(line.rstrip())
[ 5179s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1026, in call
[ 5179s] e(*args, **kwargs)
[ 5179s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1168, in call
[ 5179s] self.stream.write(ensure(line) + ensure('\n'))
[ 5179s] TypeError: write() argument must be str, not bytes
[ 5184s] ERROR:root:a bytes-like object is required, not 'str'
[ 5187s] Exception in thread ProcessReader:
[ 5187s] Traceback (most recent call last):
[ 5187s] File "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
[ 5187s] self.run()
[ 5187s] File "/usr/lib64/python3.8/threading.py", line 870, in run
[ 5187s] self._target(*self._args, **self._kwargs)
[ 5187s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1116, in _read
[ 5187s] callback(line.rstrip())
[ 5187s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1026, in call
[ 5187s] e(*args, **kwargs)
[ 5187s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1168, in call
[ 5187s] self.stream.write(ensure(line) + ensure('\n'))
[ 5187s] TypeError: write() argument must be str, not bytes

The original patch attached here does not apply. Any idea what's wrong?

Comment 8

[Geoff Brown [:gbrown]]

The patch attached here never landed; it looks like it was causing some tests to fail.

In the meantime, bug 1623701 landed; it looks like this bug should have been marked as a dup of 1623701.

Unfortunately, it looks like the change in bug 1623701 is not always effective (comment 7); :rstewart, can you have a look?

Comment 9

[Ricky Stewart]

Evidently I'm missing some context here. What are you running to produce this error? I tried the mach test command from the first comment and it works fine. If I'm meant to help with anything here I need way more information than the partial stacktrace in comment 7.

If this is related to a Python 3 port, mozprocess is consistent with the rest of Python 3 in that it can produce output as bytes or text, and when you instantiate a ProcessHandler you need to make sure you open the process in the correct mode.

Comment 10

[Geoff Brown [:gbrown]]

:wolfiR - If you are still having trouble, please file a new bug and include the full command and a full log.

Comment 11

[Wolfgang Rosenauer [:wolfiR]]

filed bug 1642410