jit-test.py: usability enhancements

NEW
Unassigned

Status

()

Core
JavaScript Engine
--
minor
6 years ago
3 years ago

People

(Reporter: joey, Unassigned)

Tracking

Trunk
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
## https://tbpl.mozilla.org/?tree=Try&rev=ea5f05803598
## https://tbpl.mozilla.org/php/getParsedLog.php?id=10680798&tree=Try

TEST-UNEXPECTED-FAIL | jit_test.py -a -m -n | /builds/slave/try-lnx/build/js/src/jit-test/tests/jaeger/recompile/bug673812.js


A try submissions recently failed one of the jit test on linux so I began looking into the problem. I have since found an MDN wiki page for the jit-test.py script but some early trial-and-error attempts produced odd results.

Naive attempts on my part were to:
  o cut & paste commands out of the log
  o a few quick argument permutations to try and run the failing test suite.

Here are the results:

TEST-UNEXPECTED-FAIL | jit_test.py -a -m -n | /builds/slave/try-lnx/build/js/src/jit-test/tests/jaeger/recompile/bug673812.js

##########################################################
## Attempt #1 - invoke verbatim command line from the logs
##########################################################
% js/src/jit-test/jit_test.py -a -m -n
Usage: jit_test.py [options] JS_SHELL [TESTS]

jit_test.py: error: no such option: -a
  ALSO: jit_test.py: error: no such option: -m
  ALSO: jit_test.py: error: no such option: -n

   ** huh ?!? those arguments were logged by try.


#############################################################
## Attempt #2 - pass in the failing test suite as an argument
#############################################################
% js/src/jit-test/jit_test.py js/src/jit-test/tests/jaeger/recompile/bug673812.js

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "js/src/jit-test/jit_test.py", line 168, in th_run_cmd
    p = Popen(cmdline, stdin=PIPE, stdout=PIPE, stderr=PIPE, **options)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1239, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied

Traceback (most recent call last):
  File "js/src/jit-test/jit_test.py", line 527, in <module>
    main(sys.argv[1:])
  File "js/src/jit-test/jit_test.py", line 516, in main
    ok = run_tests(job_list, test_dir, lib_dir, shell_args)
  File "js/src/jit-test/jit_test.py", line 281, in run_tests
    ok, out, err, code, timed_out = run_test(test, lib_dir, shell_args)
  File "js/src/jit-test/jit_test.py", line 228, in run_test
    out, err, code, timed_out = run(cmd, os.environ, OPTIONS.timeout)
  File "js/src/jit-test/jit_test.py", line 197, in run_cmd
    return run_timeout_cmd(cmdline, { 'env': env }, timeout)
  File "js/src/jit-test/jit_test.py", line 193, in run_timeout_cmd
    (out, err, code) = l[1]
TypeError: 'NoneType' object is not iterable


A few items of note and usability suggestions:
==============================================
  o If the command usage for jit-test.py also displayed an MDN url it would help avoid folks having to search for the docs online.
  o attempt #1 - we seem to have a disconnect somewhere between automation/logging and actual command usage.  When people are able to simply copy verbatim logfile lines to run failing tests it may improve willingness to research failures.
  o attempt #2 - There is obviously a problem with my attempts but the error messages reported were not very friendly.  Yes someone could spend time reading code to figure out what caused "permission denied" and why the l array (test list?) is empty.  If more self-evident error messages can be reported it allows people to spend that effort researching test failures rather than on internals or usage of the testing harness.
(In reply to Joey Armstrong [:joey] from comment #0)
> ##########################################################
> ## Attempt #1 - invoke verbatim command line from the logs
> ##########################################################
> % js/src/jit-test/jit_test.py -a -m -n
> Usage: jit_test.py [options] JS_SHELL [TESTS]
> 
> jit_test.py: error: no such option: -a
>   ALSO: jit_test.py: error: no such option: -m
>   ALSO: jit_test.py: error: no such option: -n
> 
>    ** huh ?!? those arguments were logged by try.

They were logged incorrectly, it seems.  What try actually ran was `jit_test.py --args="-m -n" ./path/to/js/executable ./path/to/testfile.js`.
 
> 
> #############################################################
> ## Attempt #2 - pass in the failing test suite as an argument
> #############################################################
> % js/src/jit-test/jit_test.py
> js/src/jit-test/tests/jaeger/recompile/bug673812.js
> 

You need to specify a js binary for the harness to use.
 
> 
> A few items of note and usability suggestions:
> ==============================================

The jsreftest suite is more mature in these specific areas.  Unfortunately the jit-test suite is still more usable in the one place that counts to its primary users: creating new tests.  We are working to improve the jsreftest suite to a point where we can use it exclusively.  Perhaps you'd like to take a look at bug 735549 and help figure out why the patch there is failing on try?
(Reporter)

Comment 2

6 years ago
(In reply to Terrence Cole [:terrence] from comment #1)
> (In reply to Joey Armstrong [:joey] from comment #0)

> > #############################################################
> > ## Attempt #2 - pass in the failing test suite as an argument
> > #############################################################
> > % js/src/jit-test/jit_test.py
> > js/src/jit-test/tests/jaeger/recompile/bug673812.js
> > 
> 
> You need to specify a js binary for the harness to use.

Yes I did see that detail the usage statement
   Usage: jit_test.py [options] JS_SHELL [TESTS]

Could jit-test perform some basic argument validation and report that the shell is missing ?
The current failure mode is reporting missing arguments as a generic 'permission denied'.

> > A few items of note and usability suggestions:
> > ==============================================
> 
> The jsreftest suite is more mature in these specific areas.  Unfortunately
> the jit-test suite is still more usable in the one place that counts to its
> primary users: creating new tests.  We are working to improve the jsreftest
> suite to a point where we can use it exclusively.  Perhaps you'd like to
> take a look at bug 735549 and help figure out why the patch there is failing
> on try?

Ok will do
Joey, I've fixed the bug I mentioned, but there is lots more low-hanging fruit in the tree under Bug 745230.
Assignee: general → nobody
Component: JavaScript Engine → jemalloc
QA Contact: general → jemalloc
Target Milestone: --- → mozilla14
Version: unspecified → Trunk
Assignee: nobody → general
Component: jemalloc → JavaScript Engine
QA Contact: jemalloc → general
Target Milestone: mozilla14 → ---
(Assignee)

Updated

3 years ago
Assignee: general → nobody
You need to log in before you can comment on or make changes to this bug.