Closed Bug 568285 Opened 11 years ago Closed 11 years ago

jstests.py: option to write list of failures to a file

Categories

(Core :: JavaScript Engine, enhancement)

Other Branch
enhancement
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: igor, Assigned: igor)

Details

(Whiteboard: fixed-in-tracemonkey)

Attachments

(1 file, 1 obsolete file)

In general jstests.py cannot have a list of known failures. Options like --timeout or using ulimit -v to limit the amount of memory available to the shell makes can make some tests to fail even if there is nothing wrong with them. For this reason it is convenient to have an option to write a list of failures to a file to establish the base line to pass it later as --exclude-file.

The attached patch adds this option under the name --failure-file.
Attached patch v1 (obsolete) — Splinter Review
Attachment #447566 - Flags: review?(dmandelin)
Assignee: general → igor
Comment on attachment 447566 [details] [diff] [review]
v1

>diff --git a/js/src/tests/jstests.py b/js/src/tests/jstests.py
>--- a/js/src/tests/jstests.py
>+++ b/js/src/tests/jstests.py
>@@ -136,24 +136,30 @@ class ResultsSink:
>         (TestResult.FAIL,  True,  True):  ('TEST-KNOWN-FAIL (EXPECTED RANDOM)',  ''),
> 
>         (TestResult.PASS,  False, False): ('TEST-UNEXPECTED-PASS',               'FIXES'),
>         (TestResult.PASS,  False, True):  ('TEST-PASS (EXPECTED RANDOM)',        ''),
>         (TestResult.PASS,  True,  False): ('TEST-PASS',                          ''),
>         (TestResult.PASS,  True,  True):  ('TEST-PASS (EXPECTED RANDOM)',        ''),
>         }
> 
>     def list(self):
>+        failure_file = open(OPTIONS.failure_file, 'w') if OPTIONS.failure_file else None;

AFAIK, conditional expressions are Python 2.5, but the tree still only requires Python 2.4.

>         for label, paths in sorted(self.groups.items()):
>             if label == '': continue
> 
>             print label
>             for path in paths:
>                 print '    %s'%path
>+                if failure_file and label != TestResult.PASS:

Here, label is one of ('REGRESSIONS', 'FIXES', ''), while TestResult.PASS is a different kind of token (that happens to have the value 'PASS'). I think you want something like |failure_file and label == 'REGRESSIONS'|.

But I think it would be cleaner to write this output in a separate loop:

          if OPTIONS.failure_file:
              failure_file = open(OPTIONS.failure_file, 'w')
              for path in self.groups['REGRESSIONS']:
                  ...

>+                    print >> failure_file, path
>+        
>+        if failure_file:
>+            failure_file.close()
> 
>         suffix = '' if self.finished else ' (partial run -- interrupted by user)'
>         if self.all_passed():
>             print 'PASS' + suffix
>         else:
>             print 'FAIL' + suffix
> 
>     def all_passed(self):
>         return 'REGRESSIONS' not in self.groups
>@@ -212,18 +218,20 @@ if __name__ == '__main__':
>                   help='extra args to pass to the JS shell')
>     op.add_option('-g', '--debug', dest='debug', action='store_true',
>                   help='run test in debugger')
>     op.add_option('--valgrind', dest='valgrind', action='store_true',
>                   help='run tests in valgrind')
>     op.add_option('--valgrind-args', dest='valgrind_args',
>                   help='extra args to pass to valgrind')
>     op.add_option('-c', '--check-manifest', dest='check_manifest', action='store_true',
>                   help='check for test files not listed in the manifest')
>+    op.add_option('--failure-file', dest='failure_file',
>+                  help='write tests that have not passed to the given file')
>     (OPTIONS, args) = op.parse_args()
>     if len(args) < 1:
>         if not OPTIONS.check_manifest:
>             op.error('missing JS_SHELL argument')
>         JS, args = None, []
>     else:
>         JS, args = args[0], args[1:]
>     # Convert to an absolute path so we can run JS from a different directory.
>     if JS is not None:
Attachment #447566 - Flags: review?(dmandelin)
Attached patch v2Splinter Review
It is indeed simpler to do a separated loop over failures to list them to a file.
Attachment #447566 - Attachment is obsolete: true
Attachment #447973 - Flags: review?(dmandelin)
Attachment #447973 - Flags: review?(dmandelin) → review+
Summary: jstests.py: option to write lits of failures to a file → jstests.py: option to write list of failures to a file
http://hg.mozilla.org/tracemonkey/rev/d52bf0cbc073
Whiteboard: fixed-in-tracemonkey
http://hg.mozilla.org/mozilla-central/rev/d52bf0cbc073
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.