having a non-ASCII character in the tip changeset's commit message causes builds to fail

RESOLVED FIXED

Status

--
critical
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: dbaron, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

Having a non-ASCII character in the tip commit causes builds to fail in automation.  I suspect this is a recent regression, and it's quite serious.

13:10:58 INFO - command: START
13:10:58 INFO - command: hg parent --template {node|short}
13:10:58 INFO - command: cwd: /builds/slave/m-in-lx-0000000000000000000000/build/src
13:10:58 INFO - command: env: {'HGPLAIN': '1'}
13:10:58 INFO - command: END (0.50s elapsed)
13:10:58 INFO - command: output:
13:10:58 INFO - 2cf8496afd5d
13:10:58 INFO - Got revision 2cf8496afd5d
13:10:58 INFO - Return code: 0
13:10:58 INFO - Setting buildbot property comments to Bug 1138157 - Change ScriptedDirectProxyHandler to inherit from BaseProxyHandler. r=efaust
13:10:58 INFO -
13:10:58 INFO - This avoids the fallback behavior from DirectProxyHandler, which can't handle revoked proxies correctly.
13:10:58 INFO -
13:10:58 INFO - Test created by André Bargull and me.
13:10:58 INFO - Writing buildbot properties ['comments'] to /builds/slave/m-in-lx-0000000000000000000000/properties/comments
13:10:58 INFO - Writing to file /builds/slave/m-in-lx-0000000000000000000000/properties/comments
13:10:58 INFO - Contents:
13:10:58 INFO - comments:Bug 1138157 - Change ScriptedDirectProxyHandler to inherit from BaseProxyHandler. r=efaust
13:10:58 INFO -
13:10:58 INFO - This avoids the fallback behavior from DirectProxyHandler, which can't handle revoked proxies correctly.
13:10:58 INFO -
13:10:58 INFO - Test created by André Bargull and me.
13:10:58 FATAL - Uncaught exception: Traceback (most recent call last):
13:10:58 FATAL - File "/tools/checkouts/mozharness/mozharness/base/script.py", line 1285, in run
13:10:58 FATAL - self.run_action(action)
13:10:58 FATAL - File "/tools/checkouts/mozharness/mozharness/base/script.py", line 1226, in run_action
13:10:58 FATAL - self._possibly_run_method("preflight_%s" % method_name)
13:10:58 FATAL - File "/tools/checkouts/mozharness/mozharness/base/script.py", line 1168, in _possibly_run_method
13:10:58 FATAL - return getattr(self, method_name)()
13:10:58 FATAL - File "/tools/checkouts/mozharness/mozharness/mozilla/building/buildbase.py", line 1406, in preflight_build
13:10:58 FATAL - self._checkout_source()
13:10:58 FATAL - File "/tools/checkouts/mozharness/mozharness/mozilla/building/buildbase.py", line 1045, in _checkout_source
13:10:58 FATAL - write_to_file=True)
13:10:58 FATAL - File "/tools/checkouts/mozharness/mozharness/mozilla/buildbot.py", line 100, in set_buildbot_property
13:10:58 FATAL - return self.dump_buildbot_properties(prop_list=[prop_name], file_name=prop_name)
13:10:58 FATAL - File "/tools/checkouts/mozharness/mozharness/mozilla/buildbot.py", line 140, in dump_buildbot_properties
13:10:58 FATAL - return self.write_to_file(file_name, contents)
13:10:58 FATAL - File "/tools/checkouts/mozharness/mozharness/base/script.py", line 437, in write_to_file
13:10:58 FATAL - fh.write(contents)
13:10:58 FATAL - UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 227: ordinal not in range(128)
13:10:58 FATAL - Running post_fatal callback...
13:10:58 ERROR - setting return code to 2 because fatal was called
13:10:58 FATAL - Exiting -1
13:10:58 INFO - Running post-run listener: _summarize
13:10:58 ERROR - # TBPL FAILURE #
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Note that it doesn't cause all builds to fail; just some of them:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=2cf8496afd5d
Summary: having a non-ASCII character in the tip commit causes builds to fail → having a non-ASCII character in the tip changeset's commit message causes builds to fail
Bug 782312 tracks using unicode everywhere in mozharness, and contains a workaround patch that may help solve this specific symptom.
philor says this is a duplicate of bug 1106342
Created attachment 8581194 [details] [diff] [review]
bug1146035

I think this will help resolve this bug, but I'd love another test pass if possible.

I don't like the nested try/except blocks; if that bothers you, maybe something like

    try:
        fh = open(file_path, open_mode)
    except IOError:
        self.log("%s can't be opened for writing!" % file_path,
                 level=error_level)
        return None  # ?
    try:
        fh.write(contents)
    except UnicodeEncodeError:
        fh.write(contents.encode('utf-8', 'replace'))
    fh.close()
    return file_path

?
Attachment #8581194 - Flags: review?(jlund)
Comment on attachment 8581194 [details] [diff] [review]
bug1146035

Review of attachment 8581194 [details] [diff] [review]:
-----------------------------------------------------------------

nested doesn't really bother me if it's only one deep. as for testing, mh is pretty self contained now thanks to pinning in tree.

Once landed, I'll pin it on a branch and pass a commit with unicode on it.
Attachment #8581194 - Flags: review?(jlund) → review+
I'm guessing this is resolved; please comment and reopen if we still hit this problem.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
actually comment 23 just means this is on our releng production env but not necessarily any c-i build/test jobs as we pin the mh rev to each branch. I have updated the mozharness rev on cedar to point to your commit and triggered a build with a unicode char in the commit message:

https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=32e5083241f5

when this build ^ completes, we should have a good idea if we can resolve this.
Status: RESOLVED → UNCONFIRMED
Ever confirmed: false
Resolution: FIXED → ---
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Duplicate of this bug: 1106342

Updated

3 years ago
Duplicate of this bug: 1121132
looks good:

11:41:10     INFO - Setting buildbot property comments to testing latest mozharness prod rev - × <- unicode char CLOSED TREE r=me
11:41:10     INFO - Writing buildbot properties ['comments'] to /builds/slave/ced-l64-d-00000000000000000000/properties/comments
11:41:10     INFO - Writing to file /builds/slave/ced-l64-d-00000000000000000000/properties/comments
11:41:10     INFO - Contents:
11:41:10     INFO -  comments:testing latest mozharness prod rev - × <- unicode char CLOSED TREE r=me

let's land this in prod branches
this was picked up with mshal's mh json bump
Status: UNCONFIRMED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.