Closed Bug 1597687 Opened 3 years ago Closed 2 years ago

[mozprocess] "TypeError: write() argument must be str, not bytes" with Python 3

Categories

(Testing :: Mozbase, defect, P3)

70 Branch
defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1623701

People

(Reporter: rgpt, Assigned: rgpt)

References

Details

Attachments

(1 file)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0

Steps to reproduce:

ran specific test in marionette harness to make it run on py3.
$ python3 mach test testing/marionette/harness/marionette_harness/tests/unit/test_accessibility.py

Actual results:

mozprocess throwed an error:

0:02.08 INFO Using workspace for temporary data: "/home/rgpt/Desktop/mozilla/mozilla-unified"
0:02.09 mozversion INFO application_buildid: 20191119194152
0:02.09 mozversion INFO application_changeset: 6efd0680f02a7911423b23246b293bf56152efe3
0:02.09 mozversion INFO application_display_name: Nightly
0:02.09 mozversion INFO application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
0:02.09 mozversion INFO application_name: Firefox
0:02.09 mozversion INFO application_remotingname: firefox
0:02.09 mozversion INFO application_vendor: Mozilla
0:02.09 mozversion INFO application_version: 72.0a1
0:02.09 mozversion INFO platform_buildid: 20191119194152
0:02.09 mozversion INFO platform_changeset: 6efd0680f02a7911423b23246b293bf56152efe3
0:02.09 mozversion INFO platform_version: 72.0a1
0:02.10 INFO Application command: /home/rgpt/Desktop/mozilla/mozilla-unified/objdir-frontend/dist/bin/firefox -no-remote -marionette -profile /tmp/tmpxlh10_0r.mozrunner
Exception in thread ProcessReader:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/rgpt/Desktop/mozilla/mozilla-unified/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1085, in _read
callback(line.rstrip())
File "/home/rgpt/Desktop/mozilla/mozilla-unified/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 995, in call
e(*args, **kwargs)
File "/home/rgpt/Desktop/mozilla/mozilla-unified/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1136, in call
self.stream.write(line + '\n'.encode('utf8'))
TypeError: write() argument must be str, not bytes

Expected results:

It should have worked on py3 also as it does on py2

Blocks: 1595814
Blocks: mozbase-py3
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: write() argument must be str, not bytes → [mozprocess] "TypeError: write() argument must be str, not bytes" with Python 3
No longer blocks: 1595814
Priority: -- → P3
See Also: → 1428713

@whimboo, encoding on https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/processhandler.py#1136 have some history behind it. As based on 991866, i am concerned that whether removing encoding will produce some issue. and if encoding is to be removed then except block will be nullified.

Please note that the second half of the first comment is not related to this bug. This is the TypeError: unorderable types: NoneType() > int() part. Please edit your comment and remove that to lower the confusion.

Is line in this call to write() actually a utf-8 encoded string or str?

I don't have that much knowledge in this area so getting feedback from Andrew, who reviewed some of those former patches, might be the best choice.

Blocks: 1597773
No longer blocks: 1355537
Flags: needinfo?(ahal)

I'm not sure why we're encoding that \n literal. It should already be bytes on Python 2 and text on Python 3. So first, get rid of that. Next we can make sure to decode if a binary string is detected:

if six.PY3 and isinstance(line, six.binary_type):
    line = line.decode('utf-8')
Flags: needinfo?(ahal)

Hey Rishi, can you try Andrew's suggestion? Does it fix the problem for you? Also please file the other issue as a new bug if it still exists with that patch added. Thanks.

Flags: needinfo?(rishigpt2009)

As discussed on IRC, it fixed my problem. Thank you @ahal and @whimboo

Flags: needinfo?(rishigpt2009)
Assignee: nobody → rishigpt2009
Status: NEW → ASSIGNED
Severity: normal → S3

The code in mozilla77 looks like this

def __call__(self, line):
    ensure = six.ensure_text if self.text else six.ensure_binary
    self.stream.write(ensure(line) + ensure('\n'))
    self.stream.flush()

But my build still fails with

[ 5178s] ERROR:root:a bytes-like object is required, not 'str'
[ 5179s] Exception in thread ProcessReader:
[ 5179s] Traceback (most recent call last):
[ 5179s] File "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
[ 5179s] self.run()
[ 5179s] File "/usr/lib64/python3.8/threading.py", line 870, in run
[ 5179s] self._target(*self._args, **self._kwargs)
[ 5179s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1116, in _read
[ 5179s] callback(line.rstrip())
[ 5179s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1026, in call
[ 5179s] e(*args, **kwargs)
[ 5179s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1168, in call
[ 5179s] self.stream.write(ensure(line) + ensure('\n'))
[ 5179s] TypeError: write() argument must be str, not bytes
[ 5184s] ERROR:root:a bytes-like object is required, not 'str'
[ 5187s] Exception in thread ProcessReader:
[ 5187s] Traceback (most recent call last):
[ 5187s] File "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
[ 5187s] self.run()
[ 5187s] File "/usr/lib64/python3.8/threading.py", line 870, in run
[ 5187s] self._target(*self._args, **self._kwargs)
[ 5187s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1116, in _read
[ 5187s] callback(line.rstrip())
[ 5187s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1026, in call
[ 5187s] e(*args, **kwargs)
[ 5187s] File "/home/abuild/rpmbuild/BUILD/firefox-77.0/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1168, in call
[ 5187s] self.stream.write(ensure(line) + ensure('\n'))
[ 5187s] TypeError: write() argument must be str, not bytes

The original patch attached here does not apply. Any idea what's wrong?

The patch attached here never landed; it looks like it was causing some tests to fail.

In the meantime, bug 1623701 landed; it looks like this bug should have been marked as a dup of 1623701.

Unfortunately, it looks like the change in bug 1623701 is not always effective (comment 7); :rstewart, can you have a look?

Flags: needinfo?(rstewart)

Evidently I'm missing some context here. What are you running to produce this error? I tried the mach test command from the first comment and it works fine. If I'm meant to help with anything here I need way more information than the partial stacktrace in comment 7.

If this is related to a Python 3 port, mozprocess is consistent with the rest of Python 3 in that it can produce output as bytes or text, and when you instantiate a ProcessHandler you need to make sure you open the process in the correct mode.

Flags: needinfo?(rstewart)

:wolfiR - If you are still having trouble, please file a new bug and include the full command and a full log.

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Flags: needinfo?(mozilla)
Resolution: --- → DUPLICATE
Duplicate of bug: 1623701
Flags: needinfo?(mozilla)
You need to log in before you can comment on or make changes to this bug.