568285 - jstests.py: option to write list of failures to a file

Status: RESOLVED FIXED

(Core :: JavaScript Engine, enhancement)

Description

[Igor Bukanov]

In general jstests.py cannot have a list of known failures. Options like --timeout or using ulimit -v to limit the amount of memory available to the shell makes can make some tests to fail even if there is nothing wrong with them. For this reason it is convenient to have an option to write a list of failures to a file to establish the base line to pass it later as --exclude-file.

The attached patch adds this option under the name --failure-file.

Comment 1

[Igor Bukanov]

Attachment: v1

Comment 2

[David Mandelin [:dmandelin]]

Comment on attachment 447566 [details] [diff] [review]
v1

>diff --git a/js/src/tests/jstests.py b/js/src/tests/jstests.py
>--- a/js/src/tests/jstests.py
>+++ b/js/src/tests/jstests.py
>@@ -136,24 +136,30 @@ class ResultsSink:
>         (TestResult.FAIL,  True,  True):  ('TEST-KNOWN-FAIL (EXPECTED RANDOM)',  ''),
> 
>         (TestResult.PASS,  False, False): ('TEST-UNEXPECTED-PASS',               'FIXES'),
>         (TestResult.PASS,  False, True):  ('TEST-PASS (EXPECTED RANDOM)',        ''),
>         (TestResult.PASS,  True,  False): ('TEST-PASS',                          ''),
>         (TestResult.PASS,  True,  True):  ('TEST-PASS (EXPECTED RANDOM)',        ''),
>         }
> 
>     def list(self):
>+        failure_file = open(OPTIONS.failure_file, 'w') if OPTIONS.failure_file else None;

AFAIK, conditional expressions are Python 2.5, but the tree still only requires Python 2.4.

>         for label, paths in sorted(self.groups.items()):
>             if label == '': continue
> 
>             print label
>             for path in paths:
>                 print '    %s'%path
>+                if failure_file and label != TestResult.PASS:

Here, label is one of ('REGRESSIONS', 'FIXES', ''), while TestResult.PASS is a different kind of token (that happens to have the value 'PASS'). I think you want something like |failure_file and label == 'REGRESSIONS'|.

But I think it would be cleaner to write this output in a separate loop:

          if OPTIONS.failure_file:
              failure_file = open(OPTIONS.failure_file, 'w')
              for path in self.groups['REGRESSIONS']:
                  ...

>+                    print >> failure_file, path
>+        
>+        if failure_file:
>+            failure_file.close()
> 
>         suffix = '' if self.finished else ' (partial run -- interrupted by user)'
>         if self.all_passed():
>             print 'PASS' + suffix
>         else:
>             print 'FAIL' + suffix
> 
>     def all_passed(self):
>         return 'REGRESSIONS' not in self.groups
>@@ -212,18 +218,20 @@ if __name__ == '__main__':
>                   help='extra args to pass to the JS shell')
>     op.add_option('-g', '--debug', dest='debug', action='store_true',
>                   help='run test in debugger')
>     op.add_option('--valgrind', dest='valgrind', action='store_true',
>                   help='run tests in valgrind')
>     op.add_option('--valgrind-args', dest='valgrind_args',
>                   help='extra args to pass to valgrind')
>     op.add_option('-c', '--check-manifest', dest='check_manifest', action='store_true',
>                   help='check for test files not listed in the manifest')
>+    op.add_option('--failure-file', dest='failure_file',
>+                  help='write tests that have not passed to the given file')
>     (OPTIONS, args) = op.parse_args()
>     if len(args) < 1:
>         if not OPTIONS.check_manifest:
>             op.error('missing JS_SHELL argument')
>         JS, args = None, []
>     else:
>         JS, args = args[0], args[1:]
>     # Convert to an absolute path so we can run JS from a different directory.
>     if JS is not None:

Comment 3

[Igor Bukanov]

Attachment: v2

It is indeed simpler to do a separated loop over failures to list them to a file.

Comment 4

[Igor Bukanov]

http://hg.mozilla.org/tracemonkey/rev/d52bf0cbc073

Comment 5

[Robert Sayre]

http://hg.mozilla.org/mozilla-central/rev/d52bf0cbc073