Closed Bug 1452864 Opened 7 years ago Closed 7 years ago

[mozfile] tree() broken with Unicode characters in path

Categories

(Testing :: Mozbase, defect, P2)

Product:
Testing
Component:
Mozbase
Type:
defect

Tracking

(firefox61 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla61
Tracking Flags:
Tracking Status
firefox61 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

Bug 1452864 - [mozfile] tree() broken with Unicode characters in path.
59 bytes, text/x-review-board-request
gbrown
: review+
Details
Bug 1452864 - [mozfile] Always use Unicode delimiters for tree() output.
59 bytes, text/x-review-board-request
gbrown
: review+
Details
A recent try push for bug 918097 has shown that the mozprofile unittests unexpectedly fail due to a Unicode char in the profile name: https://treeherder.mozilla.org/#/jobs?repo=try&revision=453a50bc94e144403cded0c6e274e09b2d29422d&selectedJob=172807337 > [task 2018-04-10T05:33:32.413Z] 2:04.09 testing/mozbase/mozprofile/tests/test_profile_view.py::TestProfilePrint::test_profile_diff TEST-UNEXPECTED-FAIL > [task 2018-04-10T05:33:32.437Z] 2:04.11 testing/mozbase/mozprofile/tests/test_profile_view.py::TestProfilePrint::test_strcast TEST-UNEXPECTED-FAIL The problem here is actually located in `mozfile.tree()`: > [task 2018-04-10T05:33:32.438Z] 2:04.11 ______________________ TestProfilePrint.test_profile_diff ______________________ > [task 2018-04-10T05:33:32.438Z] 2:04.11 > [task 2018-04-10T05:33:32.438Z] 2:04.11 self = <test_profile_view.TestProfilePrint testMethod=test_profile_diff> > [task 2018-04-10T05:33:32.438Z] 2:04.11 > [task 2018-04-10T05:33:32.438Z] 2:04.11 def test_profile_diff(self): > [task 2018-04-10T05:33:32.438Z] 2:04.11 profile1 = mozprofile.Profile() > [task 2018-04-10T05:33:32.439Z] 2:04.11 profile2 = mozprofile.Profile(preferences=dict(foo='bar')) > [task 2018-04-10T05:33:32.440Z] 2:04.11 > [task 2018-04-10T05:33:32.440Z] 2:04.11 # diff a profile against itself; no difference > [task 2018-04-10T05:33:32.441Z] 2:04.11 > self.assertEqual([], mozprofile.diff(profile1, profile1)) > [task 2018-04-10T05:33:32.441Z] 2:04.11 > [task 2018-04-10T05:33:32.441Z] 2:04.11 testing/mozbase/mozprofile/tests/test_profile_view.py:60: > [task 2018-04-10T05:33:32.442Z] 2:04.11 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > [task 2018-04-10T05:33:32.442Z] 2:04.11 testing/mozbase/mozprofile/mozprofile/diff.py:27: in diff > [task 2018-04-10T05:33:32.442Z] 2:04.11 parts[index] = prof.summary(return_parts=True)[1:] > [task 2018-04-10T05:33:32.443Z] 2:04.11 testing/mozbase/mozprofile/mozprofile/profile.py:300: in summary > [task 2018-04-10T05:33:32.443Z] 2:04.11 parts.append(('Files', '\n%s' % mozfile.tree(self.profile))) > [task 2018-04-10T05:33:32.443Z] 2:04.11 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > [task 2018-04-10T05:33:32.443Z] 2:04.12 > [task 2018-04-10T05:33:32.444Z] 2:04.12 directory = '/tmp/tmpXmNLDa.mozrunner\U0001f36a', item_marker = '\\xe2\\x94\\x9c' > [task 2018-04-10T05:33:32.444Z] 2:04.12 vertical_line = '\xe2\x94\x82', last_child = '\xe2\x94\x94' > [task 2018-04-10T05:33:32.444Z] 2:04.12 sort_key = <function <lambda> at 0x7f934b7289b0> > [task 2018-04-10T05:33:32.445Z] 2:04.12 > [task 2018-04-10T05:33:32.445Z] 2:04.12 def tree(directory, > [task 2018-04-10T05:33:32.445Z] 2:04.12 item_marker=unicode_delimeters['item_marker'], > [task 2018-04-10T05:33:32.445Z] 2:04.12 vertical_line=unicode_delimeters['vertical_line'], > [task 2018-04-10T05:33:32.446Z] 2:04.12 last_child=unicode_delimeters['last_child'], > [task 2018-04-10T05:33:32.446Z] 2:04.12 sort_key=lambda x: x.lower()): > [task 2018-04-10T05:33:32.446Z] 2:04.12 """ > [task 2018-04-10T05:33:32.446Z] 2:04.12 display tree directory structure for `directory` > [task 2018-04-10T05:33:32.446Z] 2:04.12 """ > [task 2018-04-10T05:33:32.447Z] 2:04.12 > [task 2018-04-10T05:33:32.447Z] 2:04.12 retval = [] > [task 2018-04-10T05:33:32.447Z] 2:04.12 indent = [] > [task 2018-04-10T05:33:32.447Z] 2:04.12 last = {} > [task 2018-04-10T05:33:32.448Z] 2:04.12 top = depth(directory) > [task 2018-04-10T05:33:32.449Z] 2:04.12 > [task 2018-04-10T05:33:32.450Z] 2:04.12 for dirpath, dirnames, filenames in os.walk(directory, topdown=True): > [task 2018-04-10T05:33:32.450Z] 2:04.12 > [task 2018-04-10T05:33:32.450Z] 2:04.12 abspath = os.path.abspath(dirpath) > [task 2018-04-10T05:33:32.450Z] 2:04.12 basename = os.path.basename(abspath) > [task 2018-04-10T05:33:32.451Z] 2:04.12 parent = os.path.dirname(abspath) > [task 2018-04-10T05:33:32.451Z] 2:04.12 level = depth(abspath) - top > [task 2018-04-10T05:33:32.451Z] 2:04.12 > [task 2018-04-10T05:33:32.451Z] 2:04.12 # sort articles of interest > [task 2018-04-10T05:33:32.452Z] 2:04.12 for resource in (dirnames, filenames): > [task 2018-04-10T05:33:32.452Z] 2:04.12 resource[:] = sorted(resource, key=sort_key) > [task 2018-04-10T05:33:32.452Z] 2:04.12 > [task 2018-04-10T05:33:32.452Z] 2:04.12 if level > len(indent): > [task 2018-04-10T05:33:32.453Z] 2:04.12 indent.append(vertical_line) > [task 2018-04-10T05:33:32.453Z] 2:04.12 indent = indent[:level] > [task 2018-04-10T05:33:32.453Z] 2:04.13 > [task 2018-04-10T05:33:32.454Z] 2:04.13 if dirnames: > [task 2018-04-10T05:33:32.454Z] 2:04.13 files_end = item_marker > [task 2018-04-10T05:33:32.454Z] 2:04.13 last[abspath] = dirnames[-1] > [task 2018-04-10T05:33:32.454Z] 2:04.13 else: > [task 2018-04-10T05:33:32.455Z] 2:04.13 files_end = last_child > [task 2018-04-10T05:33:32.455Z] 2:04.13 > [task 2018-04-10T05:33:32.456Z] 2:04.13 if last.get(parent) == os.path.basename(abspath): > [task 2018-04-10T05:33:32.456Z] 2:04.13 # last directory of parent > [task 2018-04-10T05:33:32.457Z] 2:04.13 dirpath_mark = last_child > [task 2018-04-10T05:33:32.457Z] 2:04.13 indent[-1] = ' ' > [task 2018-04-10T05:33:32.457Z] 2:04.13 elif not indent: > [task 2018-04-10T05:33:32.457Z] 2:04.13 dirpath_mark = '' > [task 2018-04-10T05:33:32.458Z] 2:04.13 else: > [task 2018-04-10T05:33:32.458Z] 2:04.13 dirpath_mark = item_marker > [task 2018-04-10T05:33:32.458Z] 2:04.13 > [task 2018-04-10T05:33:32.459Z] 2:04.13 # append the directory and piece of tree structure > [task 2018-04-10T05:33:32.459Z] 2:04.13 # if the top-level entry directory, print as passed > [task 2018-04-10T05:33:32.459Z] 2:04.13 retval.append('%s%s%s' % (''.join(indent[:-1]), > [task 2018-04-10T05:33:32.460Z] 2:04.13 dirpath_mark, > [task 2018-04-10T05:33:32.460Z] 2:04.13 basename if retval else directory)) > [task 2018-04-10T05:33:32.460Z] 2:04.13 # add the files > [task 2018-04-10T05:33:32.460Z] 2:04.13 if filenames: > [task 2018-04-10T05:33:32.461Z] 2:04.13 last_file = filenames[-1] > [task 2018-04-10T05:33:32.461Z] 2:04.13 retval.extend([('%s%s%s' % (''.join(indent), > [task 2018-04-10T05:33:32.462Z] 2:04.13 files_end if filename == last_file else item_marker, > [task 2018-04-10T05:33:32.462Z] 2:04.13 filename)) > [task 2018-04-10T05:33:32.462Z] 2:04.13 > for index, filename in enumerate(filenames)]) > [task 2018-04-10T05:33:32.464Z] 2:04.13 E UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128) > [task 2018-04-10T05:33:32.464Z] 2:04.13 > [task 2018-04-10T05:33:32.464Z] 2:04.13 testing/mozbase/mozfile/mozfile/mozfile.py:325: UnicodeDecodeError
Summary: [mozfile] → [mozfile] tree() broken due to Unicode characters in path
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Priority: -- → P2
Summary: [mozfile] tree() broken due to Unicode characters in path → [mozfile] tree() broken with Unicode characters in path
Attachment #8966493 - Flags: review?(gbrown)
Attachment #8966504 - Flags: review?(gbrown)
Comment on attachment 8966493 [details] Bug 1452864 - [mozfile] tree() broken with Unicode characters in path. https://reviewboard.mozilla.org/r/235202/#review241002 I'm surprised by how many unicode problems you are finding. Good stuff! ::: testing/mozbase/mozprofile/tests/test_profile_view.py:50 (Diff revision 3) > mozfile.rmtree(tempdir) > > - def test_strcast(self): > - """ > - test casting to a string > - """ > + def test_str_cast(self): > + """Test casting to a string.""" > + profile = mozprofile.Profile() > + self.assertEqual(str(profile), profile.summary().encode("UTF-8")) "UTF-8" or "utf-8" -- does it matter?
Attachment #8966493 - Flags: review?(gbrown) → review+
Comment on attachment 8966504 [details] Bug 1452864 - [mozfile] Always use Unicode delimiters for tree() output. https://reviewboard.mozilla.org/r/235226/#review241014
Attachment #8966504 - Flags: review?(gbrown) → review+
Comment on attachment 8966493 [details] Bug 1452864 - [mozfile] tree() broken with Unicode characters in path. https://reviewboard.mozilla.org/r/235202/#review241002 It's not the end yet! > "UTF-8" or "utf-8" -- does it matter? Shouldn't, but I will update.
Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/62fb6111910a [mozfile] tree() broken with Unicode characters in path. r=gbrown https://hg.mozilla.org/integration/autoland/rev/4c4719645fd9 [mozfile] Always use Unicode delimiters for tree() output. r=gbrown
https://hg.mozilla.org/mozilla-central/rev/62fb6111910a https://hg.mozilla.org/mozilla-central/rev/4c4719645fd9
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
status-firefox61: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: