1605106 - `mach lint -n third_party/python/lldbutils/` hangs forever

Alexandru, would you be able to investigate what's happening here? It looks like perfdocs is hanging. For context, passing -n tells mozlint to ignore the include and exclude rules in the .yml file. So it's possible your linter isn't handling unexpected paths. Maybe we just need to add a quick check at the start of the lint function to return early if none of the passed in paths are ones we care about.

Flags: needinfo?(aionescu)

Alexandru Ionescu (needinfo me) [:alexandrui]

Comment 2

•

4 years ago

Greg, could you look into this, please?

Flags: needinfo?(aionescu) → needinfo?(gmierz2)

Jonathan Watt [:jwatt]

Reporter

Comment 3

•

4 years ago

Why is the yaml linter even being triggered? There are only .py files in that directory.

Alexandru Ionescu (needinfo me) [:alexandrui]

Comment 4

•

4 years ago

(In reply to Andrew Halberstadt [:ahal] from comment #1)

I don't get all that spew nor the yamlllint problem.. but I do see the "hang". It looks like it's the perfdocs linter that is causing the problem.

Alexandru, would you be able to investigate what's happening here? It looks like perfdocs is hanging. For context, passing -n tells mozlint to ignore the include and exclude rules in the .yml file. So it's possible your linter isn't handling unexpected paths. Maybe we just need to add a quick check at the start of the lint function to return early if none of the passed in paths are ones we care about.

I am looking into it.

Flags: needinfo?(gmierz2)

Alexandru Ionescu (needinfo me) [:alexandrui]

Comment 5

•

4 years ago

•

Edited

Jonathan, I am not getting the error you get (I use ubuntu not macos) but I do get an error (see below). I am getting an error, though, and I tried to get rid of perfdocs to see if it is the cause. I am still getting the same error:

./mach lint -n third_party/python/lldbutils/
with perfdocs

error: problem with lint setup, skipping android-api-lint, android-checkstyle, android-javadoc, android-lint, android-test
INFO perfdocs: args: (), kwargs: {}
description Performance Documentation linter
include ['testing/raptor']
extensions ['rst', 'ini', 'yml']
support-files ['/home/alexandru.ionescu/workspace/mozilla-central/tools/rewriting/Generated.txt', '/home/alexandru.ionescu/workspace/mozilla-central/tools/rewriting/ThirdPartyPaths.txt', '/home/alexandru.ionescu/workspace/mozilla-central/tools/lint/perfdocs.yml']
type structured_log
payload perfdocs:lint
setup perfdocs:setup
name perfdocs
path /home/alexandru.ionescu/workspace/mozilla-central/tools/lint/perfdocs.yml
Unable to locate shellcheck, please ensure it is installed and in
your PATH or set the SHELLCHECK environment variable.

https://shellcheck.net or your system's package manager.
Traceback (most recent call last):
  File "/home/alexandru.ionescu/workspace/mozilla-central/python/mozlint/mozlint/roller.py", line 55, in _run_worker
    res = func(paths, config, **lintargs) or []
  File "/home/alexandru.ionescu/workspace/mozilla-central/python/mozlint/mozlint/types.py", line 54, in __call__
    return self._lint(paths, config, **lintargs)
  File "/home/alexandru.ionescu/workspace/mozilla-central/python/mozlint/mozlint/types.py", line 139, in _lint
    return func(files, config, **lintargs)
  File "/home/alexandru.ionescu/workspace/mozilla-central/tools/lint/rust/__init__.py", line 134, in lint
    if is_old_rustfmt(binary):
  File "/home/alexandru.ionescu/workspace/mozilla-central/tools/lint/rust/__init__.py", line 117, in is_old_rustfmt
    universal_newlines=True,
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1278, in _execute_child
    executable = os.fsencode(executable)
  File "/usr/lib/python3.6/os.py", line 800, in fsencode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not NoneType
 0:00.20 INFO Found 1 perfdocs directories in /home/alexandru.ionescu/workspace/mozilla-central/testing
warning: no files to lint (eslint)
/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/__init__.py
...

/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/content.py
...

/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/general.py
...

/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/gfx.py
...

/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/layout.py
...

/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/utils.py
...

/home/alexandru.ionescu/workspace/mozilla-central/tools/lint/wpt.yml
...

A failure occurred in the rust linter.
✖ 64 problems (63 errors, 0 warnings, 1 failure)

without perfdocs

error: problem with lint setup, skipping android-api-lint, android-checkstyle, android-javadoc, android-lint, android-test
Unable to locate shellcheck, please ensure it is installed and in
your PATH or set the SHELLCHECK environment variable.

https://shellcheck.net or your system's package manager.
Traceback (most recent call last):
  File "/home/alexandru.ionescu/workspace/mozilla-central/python/mozlint/mozlint/roller.py", line 55, in _run_worker
    res = func(paths, config, **lintargs) or []
  File "/home/alexandru.ionescu/workspace/mozilla-central/python/mozlint/mozlint/types.py", line 54, in __call__
    return self._lint(paths, config, **lintargs)
  File "/home/alexandru.ionescu/workspace/mozilla-central/python/mozlint/mozlint/types.py", line 139, in _lint
    return func(files, config, **lintargs)
  File "/home/alexandru.ionescu/workspace/mozilla-central/tools/lint/rust/__init__.py", line 134, in lint
    if is_old_rustfmt(binary):
  File "/home/alexandru.ionescu/workspace/mozilla-central/tools/lint/rust/__init__.py", line 117, in is_old_rustfmt
    universal_newlines=True,
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1278, in _execute_child
    executable = os.fsencode(executable)
  File "/usr/lib/python3.6/os.py", line 800, in fsencode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not NoneType
warning: no files to lint (eslint)
/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/__init__.py
...

/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/content.py
...

/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/general.py
...

/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/gfx.py
...

/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/layout.py
...

/home/alexandru.ionescu/workspace/mozilla-central/third_party/python/lldbutils/lldbutils/utils.py
...

/home/alexandru.ionescu/workspace/mozilla-central/tools/lint/wpt.yml
...

A failure occurred in the rust linter.
✖ 64 problems (63 errors, 0 warnings, 1 failure)

I just temporarily moved tools/lint/perfdocs.yml to one directory upper. So if you try this on your machine you probably find out if this hangs because of perfdocs or not. Please let me know of your findings.
As a notice, this is my last office day for 2019. I will be back on Jan 3rd, so if you strongly believe/find out for sure it's from perfdocs, I'll be around today until 5PM EET. I really don't want to leave an error before leaving in PTO.

Andrew, despite I am not completely sure it's from perfdocs, you idea makes sense and we should probably and that quick check... before (actually) causing the issue.

Flags: needinfo?(jwatt)

Jonathan Watt [:jwatt]

Reporter

Comment 6

•

4 years ago

I'm afraid temporarily removing tools/lint/perfdocs.yml doesn't change what happens for me. The console output appears to be identical (except for the names of some temporary directories created during the linting).

Flags: needinfo?(jwatt)

Andrew Halberstadt [:ahal]

Assignee

Comment 7

•

4 years ago

•

Edited

Ah, my apologies Alexandru. Must be another linter that's causing the hang then. The output on my run made it look like perfdocs, but I guess it was just misleading. That said, we could still implement the change to exit early if the files don't match what we expect.. not a big deal though.

Andrew Halberstadt [:ahal]

Assignee

Comment 8

•

4 years ago

I think it's the rustfmt linter. Jonhathan, can you reproduce with:

$ ./mach lint -n third_party/python/six -l rustfmt

?

Jonathan Watt [:jwatt]

Reporter

Comment 9

•

4 years ago

Looks like that's it. That linter seems to hang forever and I get no output, event after hitting ctrl-c.

Jonathan Watt [:jwatt]

Reporter

Comment 10

•

4 years ago

And the rustfmt process appears to be using 0% CPU.

Jonathan Watt [:jwatt]

Reporter

Comment 11

•

4 years ago

Looks like it's waiting for some stdin input:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGTTIN
  * frame #0: 0x00007fff741f7ef2 libsystem_kernel.dylib`read + 10
    frame #1: 0x000000010c908375 rustfmt`_$LT$std..io..buffered..BufReader$LT$R$GT$$u20$as$u20$std..io..BufRead$GT$::fill_buf::h5b79a87340cccd93 + 69
    frame #2: 0x000000010c90993a rustfmt`_$LT$std..io..stdio..StdinLock$u20$as$u20$std..io..Read$GT$::read::h54713b6e17aa3e1c + 58
    frame #3: 0x000000010c90bc4e rustfmt`std::io::read_to_end::hd34463bf687b3dc4 + 270
    frame #4: 0x000000010c9096bd rustfmt`_$LT$std..io..stdio..Stdin$u20$as$u20$std..io..Read$GT$::read_to_string::h4b3ff86d2008097c + 157
    frame #5: 0x000000010c542fd7 rustfmt`rustfmt::execute::h03e6819950c8bcf8 + 17639
    frame #6: 0x000000010c53e83c rustfmt`rustfmt::main::h03e420aab16e2d93 + 1100
    frame #7: 0x000000010c536956 rustfmt`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h1287bc832b7fb55d + 6
    frame #8: 0x000000010c9116f8 rustfmt`std::panicking::try::do_call::ha1a4587e2a4eb439 + 24
    frame #9: 0x000000010c913dcf rustfmt`__rust_maybe_catch_panic + 31
    frame #10: 0x000000010c91219e rustfmt`std::rt::lang_start_internal::hcf96e32a124891dc + 542
    frame #11: 0x000000010c546499 rustfmt`main + 41
    frame #12: 0x000000010c525e54 rustfmt`start + 52

Specifically it looks like it's waiting at this call, which has the comment "if no file argument is supplied, read from stdin":

https://github.com/rust-lang/rustfmt/blob/0a6a7d672bc59d86229a2ba1b6062827153ead0d/src/bin/main.rs#L492

In other words, rustfmt has been invoked without passing the path of a file to format.

Andrew Halberstadt [:ahal]

Assignee

Comment 12

•

4 years ago

Attached file Bug 1605106 - [lint.rustfmt] Return early if expanding exclusions results in no paths to lint, r?jwatt — Details

Phabricator Automation

Updated

•

4 years ago

Assignee: nobody → ahal

Status: NEW → ASSIGNED

Jonathan Watt [:jwatt]

Reporter

Updated

•

4 years ago

Blocks: 1605500

Andrew Halberstadt [:ahal]

Assignee

Comment 13

•

4 years ago

Actually I'll have to skip the test for now. There aren't any pre-existing ones and when I tried to add one I started hitting issues with ./mach python-test that apparently cropped up over the holidays. I'll have to punt the test for now but I'll split the change out and try to get it landed when I have more time.

Pulsebot

Comment 14

•

4 years ago

Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/42fabe68a375
[lint.rustfmt] Return early if expanding exclusions results in no paths to lint, r=jwatt

Natalia Csoregi [:nataliaCs]

Comment 15

•

4 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/42fabe68a375

Status: ASSIGNED → RESOLVED

Closed: 4 years ago

status-firefox73: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → mozilla73

BMO Automation

Updated

•

2 years ago

Product: Firefox Build System → Developer Infrastructure

terminal output 4 years ago Jonathan Watt [:jwatt] 84.69 KB, text/plain		Details
Bug 1605106 - [lint.rustfmt] Return early if expanding exclusions results in no paths to lint, r?jwatt 4 years ago Andrew Halberstadt [:ahal] 47 bytes, text/x-phabricator-request		Details \| Review