1642662 - [mozcrash] Output of minidump_stackwalk should be converted from bytes to str in _process_dump_file

Status: RESOLVED FIXED

(Testing :: Mozbase, defect, P3)

Description

[Bob Clary [:bc] (inactive)]

Hit this while testing once but haven't been able to crash again to reproduce...

TypeError: a bytes-like object is required, not 'str'

  File "/home/bclary/mozilla/builds/autoland/mozilla/layout/tools/reftest/mach_commands.py", line 208, in run_reftest
    return self._run_reftest(**kwargs)
  File "/home/bclary/mozilla/builds/autoland/mozilla/layout/tools/reftest/mach_commands.py", line 242, in _run_reftest
    return reftest.run_desktop_test(**kwargs)
  File "/home/bclary/mozilla/builds/autoland/mozilla/layout/tools/reftest/mach_commands.py", line 87, in run_desktop_test
    rv = runreftest.run_test_harness(parser, args)
  File "/home/bclary/mozilla/builds/autoland/mozilla/layout/tools/reftest/runreftest.py", line 1035, in run_test_harness
    result = reftest.runTests(options.tests, options)
  File "/home/bclary/mozilla/builds/autoland/mozilla/layout/tools/reftest/runreftest.py", line 626, in runTests
    return self.runSerialTests(manifests, options, cmdargs)
  File "/home/bclary/mozilla/builds/autoland/mozilla/layout/tools/reftest/runreftest.py", line 983, in runSerialTests
    status = run(tests=tests)
  File "/home/bclary/mozilla/builds/autoland/mozilla/layout/tools/reftest/runreftest.py", line 940, in run
    status = self.runApp(
  File "/home/bclary/mozilla/builds/autoland/mozilla/layout/tools/reftest/runreftest.py", line 879, in runApp
    crashed = mozcrash.log_crashes(self.log, os.path.join(profile.profile, 'minidumps'),
  File "/home/bclary/mozilla/builds/autoland/mozilla/testing/mozbase/mozcrash/mozcrash/mozcrash.py", line 143, in log_crashes
    for info in CrashInfo(dump_directory, symbols_path, dump_save_path=dump_save_path,
  File "/home/bclary/mozilla/builds/autoland/mozilla/testing/mozbase/mozcrash/mozcrash/mozcrash.py", line 260, in __iter__
    rv = self._process_dump_file(path, extra)
  File "/home/bclary/mozilla/builds/autoland/mozilla/testing/mozbase/mozcrash/mozcrash/mozcrash.py", line 323, in _process_dump_file
    if "(crashed)" in line:

Looks like we need to convert the output using six.ensure_str

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: Bug 1642662 - [mozcrash] Convert output of minidump_stackwalk from bytes to str in _process_dump_file in Python3, r=gbrown.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ab0fcba34420
[mozcrash] Convert output of minidump_stackwalk from bytes to str in _process_dump_file in Python3, r=gbrown.

Comment 3

[Bob Clary [:bc] (inactive)]

This caused testing/mozbase/mozcrash/tests/test_basic.py::test_dump_directory_unicode TEST-UNEXPECTED-FAIL

I failed to run the python unit tests which would have caught this.

Comment 4

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b4adccc3ebf9
Backed out changeset ab0fcba34420 for causing Bug 1642773. CLOSED TREE

Comment 5

[Bob Clary [:bc] (inactive)]

The error that was reported in Bug 1642773 is

[task 2020-06-02T19:36:54.975Z] 0:45.79 platform win32 -- Python 2.7.15, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- Z:\task_1591125419\build\src\obj-x86_64-pc-mingw32_virtualenvs\src-JBWGhTZ--2\Scripts\python.exe
[task 2020-06-02T19:36:54.975Z] 0:45.79 rootdir: Z:\task_1591125419\build\src, inifile: Z:\task_1591125419\build\src\config\mozunit\mozunit\pytest.ini
[task 2020-06-02T19:36:54.975Z] 0:45.79 collecting ... collected 4 items
[task 2020-06-02T19:36:54.975Z] 0:45.79
[task 2020-06-02T19:36:54.975Z] 0:45.79 testing/mozbase/mozcrash/tests/test_basic.py::test_no_dump_files PASSED
[task 2020-06-02T19:36:54.975Z] 0:45.79 testing/mozbase/mozcrash/tests/test_basic.py::test_dump_count[3] PASSED
[task 2020-06-02T19:36:54.976Z] 0:45.79 testing/mozbase/mozcrash/tests/test_basic.py::test_dump_directory_unicode TEST-UNEXPECTED-FAIL
[task 2020-06-02T19:36:54.976Z] 0:45.79 testing/mozbase/mozcrash/tests/test_basic.py::test_test_name_unicode PASSED
[task 2020-06-02T19:36:54.976Z] 0:45.79
[task 2020-06-02T19:36:54.976Z] 0:45.79 ================================== FAILURES ===================================
[task 2020-06-02T19:36:54.976Z] 0:45.79 _________________________ test_dump_directory_unicode _________________________
[task 2020-06-02T19:36:54.976Z] 0:45.79
[task 2020-06-02T19:36:54.976Z] 0:45.79 request = <FixtureRequest for <Function 'test_dump_directory_unicode'>>
[task 2020-06-02T19:36:54.976Z] 0:45.79 check_for_crashes = <function wrapper at 0x0000000004734AC8>
[task 2020-06-02T19:36:54.976Z] 0:45.79 tmpdir = local(u'c:\users\task_1591125419\appdata\local\temp\pytest-of-task_1591125419\pytest-0\test_dump_directory_unicode0\\U0001f36a')
[task 2020-06-02T19:36:54.977Z] 0:45.79 capsys = <_pytest.capture.CaptureFixture object at 0x0000000004830470>
[task 2020-06-02T19:36:54.977Z] 0:45.79
[task 2020-06-02T19:36:54.977Z] 0:45.79 def test_dump_directory_unicode(request, check_for_crashes, tmpdir, capsys):
[task 2020-06-02T19:36:54.977Z] 0:45.79 """Test that check_for_crashes can handle unicode in dump_directory."""
[task 2020-06-02T19:36:54.977Z] 0:45.79 from conftest import minidump_files
[task 2020-06-02T19:36:54.977Z] 0:45.79
[task 2020-06-02T19:36:54.979Z] 0:45.80 minidump_files = minidump_files(request, tmpdir)
[task 2020-06-02T19:36:54.979Z] 0:45.80
[task 2020-06-02T19:36:54.979Z] 0:45.80 > assert 1 == check_for_crashes(dump_directory=fspath(tmpdir),
[task 2020-06-02T19:36:54.979Z] 0:45.80 quiet=False)
[task 2020-06-02T19:36:54.980Z] 0:45.80
[task 2020-06-02T19:36:54.980Z] 0:45.80 testing\mozbase\mozcrash\tests\test_basic.py:30:

In attempting to work through this I've run into python3 issues with python-test and mozprocess. I think this needs to wait until those are resolved.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:gbrown, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 7

[Geoff Brown [:gbrown]]

Attachment: Bug 1642662 - [mozcrash] Convert output of minidump_stackwalk from bytes to str in _process_dump_file in Python3

bc's original patch, but restricted to py3 (py2 already works - no need to change it!). I also updated
the six imports to suit my preference.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/928211400dc7
[mozcrash] Convert output of minidump_stackwalk from bytes to str in _process_dump_file in Python3 r=bc

Comment 9

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/928211400dc7