Closed Bug 1566979 Opened 4 months ago Closed Last month

non-ASCII characters in eslint messages get lost

Categories

(Firefox Build System :: Lint and Formatting, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: heycam, Assigned: ahal)

References

(Blocks 2 open bugs)

Details

(Whiteboard: [fixed by bug 1580280])

I'm unsure which layer of code is causing this problem. Please help me redirect to an appropriate component!

I did a try run and got some eslint errors. In the Raw Log on treeheder, I see:

[task 2019-07-17T08:37:55.182Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/layout/tools/layout-debug/ui/content/layoutdebug-helper.js:12:58 | Replace `???.createInstance(Ci.nsILayoutDebuggingTools` with `.createInstance(???Ci.nsILayoutDebuggingTools?` (prettier/prettier)

When I run eslint locally, the output I get is:

/z/moz/g/layout/tools/layout-debug/ui/content/layoutdebug-helper.js
  12:58  error  Replace `⏎··.createInstance(Ci.nsILayoutDebuggingTools` with `.createInstance(⏎··Ci.nsILayoutDebuggingTools⏎`  prettier/prettier (eslint)

so it looks like those non-ASCII characters are getting folded down into "?"s.

Component: General → Lint and Formatting
Product: Testing → Firefox Build System
Version: Version 3 → unspecified

Given that ./mach lint and ./mach eslint give the right output, I'm suspecting this is something to do with the Taskcluster setup or the builders not handling unicode text.

Hence, moving it across to Taskcluster where we can hopefully get some hints.

Blocks: 1427845
Component: Lint and Formatting → General
Product: Firefox Build System → Taskcluster

Taskcluster's able to display unicode just fine:

https://tools.taskcluster.net/groups/QDWivZvaQc6ow7QK8Lq-XQ/tasks/QDWivZvaQc6ow7QK8Lq-XQ/runs/0/logs/public%2Flogs%2Flive.log

[taskcluster 2019-07-29 12:54:10.845Z] === Task Starting ===
⏎
[taskcluster 2019-07-29 12:54:11.418Z] === Task Finished ===
[taskcluster 2019-07-29 12:54:11.418Z] Successful task run with exit code: 0 completed in 2.627 seconds

It even looks fine in the raw log.

TC just treats this as an encoded bytestream that it feeds to the browser. Something in between mach and TC is causing the issue..

Forgot to follow this so I didn't see the response.

I've just done a test, and it looks like the encoding is not getting set for python. Locally, I see sys.stdout.encoding as 'UTF-8' but on the builder it is 'None'. The 'None' makes it fallback to outputting ASCII, hence the issue here.

Anyone have any ideas where we need to set this?

Duplicate of this bug: 1563806

I think this will automatically be fixed once bug 1580280 lands (which is getting closer).
But in the meantime changing this to:

out = formatter(result).encode("mbcs" if sys.platform == "win32" else "utf-8", 'replace')

would probably do the trick.

Component: General → Lint and Formatting
Product: Taskcluster → Firefox Build System

Now that bug 1580280 landed, I've just tested this here:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=24125194b7ab39fe6409bdc41dd4c301133e78cf&selectedJob=270039276
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=270039276&repo=try&lineNumber=292

We're now displaying the correctly. Hence calling this as fixed by that bug (as it was the most likely one).

Assignee: nobody → ahal
Status: NEW → RESOLVED
Closed: Last month
Depends on: 1580280
Resolution: --- → FIXED
Whiteboard: [fixed by bug 1580280]
You need to log in before you can comment on or make changes to this bug.