JavaScript Tests - sync Sisyphus support

RESOLVED FIXED

Status

()

Core
JavaScript Engine
RESOLVED FIXED
11 years ago
11 years ago

People

(Reporter: bc, Assigned: bc)

Tracking

Trunk
Points:
---
Bug Flags:
in-testsuite -
in-litmus -

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

This bug updates the JavaScript Test Library with the latest version
I have been using to run the automated JavaScript Tests with Sisyphus
(mozilla/testing/sisyphus). This will enable the use of buildbot to
run the shell and browser tests via Sisyphus with reporting to a
Tinderbox like waterfall display.

Except for a minor change to jsDriver.pl it does not otherwise affect
the test suite as run by normal folks.

The changes to the existing tree include removal of several old
"Spider" userhook versions and the introduction of a new userhook I
have been using for some time.

The additions are a means of running the JavaScript Tests through
Sisyphus which incorporate the post processing of the test logs,
comparison against a list of known failures and the output of lists of
possible fixes and possible regressions.

Unfortunately, I haven't documented the design or use of this however
I will begin doing so at <http://wiki.mozilla.org/Sisyphus> and
<http://wiki.mozilla.org/Sisyphus/JavaScript> this weekend, however
here is an overview.

I am considering the possibility of moving all of this code into the
Sisyphus tree, but would like to go ahead and check it into js/tests
at least temporarily.

The JavaScript Tests (jsDriver.pl with the -R option) produce output
of the form:

<jstest: ecma/Array/15.4-1.js bug:  result: PASSED type: shell description: var myarr = new Array(); myarr[Math.pow(2,32)-2]='hi'; myarr[Math.pow(2,32)-2] expected: hi actual: hi reason: >

known-failures.pl (with the help of post-process-logs.pl) take the log
of the JavaScript test run, collect the test results, determine if
tests failed to complete due to time outs and crashes, adds metadata 
concerning the build, platform, rundate, etc., performs matching
against the known failures  and produces a set of log files.

<prefix>-results-all.log            - all test results
<prefix>-results-failures.log       - failed test results.
<prefix>-results-possible-fixes.log - known failures which did
                                      not match a failing test case.
<prefix>-results-possible-regressions.log - test failures which
                                      did not match a known failure.

The format of these files is of the form:

<TEST_ID=ecma/Array/15.4-1.js, TEST_BRANCH=1.9.0, TEST_RESULT=PASSED, TEST_BUILDTYPE=debug, TEST_TYPE=browser, TEST_OS=win32, TEST_MACHINE=qm-winxp03, TEST_PROCESSORTYPE=unknown, TEST_KERNEL='1.5.24(0.156/4/2)', TEST_DATE=2007-09-28-19-07-39-0400, TEST_TIMEZONE=-0400, TEST_DESCRIPTION=var myarr = new Array(); myarr[Math.pow(2,30)-2]='hi'; myarr.length expected: 1073741823 actual: 1073741823 reason:>

The known failures follow a similar format with the addition of the
ability to use Perl regular expressions to combine failures across
branches, build types, platforms, etc. into a more compact form.

<TEST_ID=e4x/Expressions/11.1.4-08.js, TEST_BRANCH=(1.8.1|1.9.0), TEST_RESULT=FAILED, TEST_BUILDTYPE=(debug|opt), TEST_TYPE=(browser|shell), TEST_OS=(linux|mac|win32), TEST_MACHINE=.*, TEST_PROCESSORTYPE=.*, TEST_KERNEL=.*, TEST_DATE=.*, TEST_TIMEZONE=.*, TEST_DESCRIPTION=Section 50 of test - 11.1.4 - XML Initializer - {} Expressions - 08 expected: true actual: false reason: Expected value 'true', Actual value 'false'>

In the fields before TEST_DESCRIPTION, normal Perl regular expressions
are used unmodified. However in order to deal with the
TEST_DESCRIPTION which may contain regular expression characters from
the output of the JavaScript test, the patterns in TEST_DESCRIPTION
use a quoted form where every individual regular expression character
is surrounded by back-tics. For example, `.``*` to represent .*

For example:

<TEST_ID=e4x/Regress/regress-369032.js, TEST_BRANCH=1.8.1, TEST_RESULT=FAILED, TEST_BUILDTYPE=debug, TEST_TYPE=browser, TEST_OS=(linux|mac), TEST_MACHINE=.*, TEST_PROCESSORTYPE=.*, TEST_KERNEL=.*, TEST_DATE=.*, TEST_TIMEZONE=.*, TEST_DESCRIPTION=EXIT STATUS: CRASHED signal `[`1`-`9`]``[`0`-`9`]``*` (`[`0`-`9`.``]``+` seconds), BUGNUMBER: 369032; STATUS: Do not assert: kid2->parent == xml || !kid2->parent; Assertion failure: kid2->parent == xml || !kid2->parent, at `.``*`jsxml.c:`[`0`-`9`]``+`>

Files included in the patch.

create-patterns.pl - combines failure logs from various branches,
platforms using regular epxressions to produce output which can be
inserted into one of the *failures.txt files.

known-failures.pl - processes raw JavaScript Test log files to produce
the all, failures, possible fixes and possible regression logs.

post-process-logs.pl - used by known-failures.pl to collect the
javascript test results from both shell and browser tests from raw log
files.

public-failures.txt - known failures of public bugs.

runtests.sh - shell script used by Sisyphus to run the JavaScript
tests.

userhookeach.js - "Spider" userhook script used in the browser tests.

Files not included in the patch.

confidential-failures.txt - known failures from security sensitive
bugs. This is not meant to be checked in, and will be created if it
doesn't exist. I'll attach it separately from the patch.

Note: the shell scripts expect bash to live at /usr/local/bin/bash. I
hope to remove this kludge in the near future, but you can easily sym
link /usr/local/bin/bash to /bin/bash to work around the issue at
least temporarily. Bash 3.x is currently required which is not
available by default on Mac OS X.
Flags: in-testsuite-
Flags: in-litmus-
Created attachment 282830 [details] [diff] [review]
sisyphus.patch

patch
Blocks: 396028
Status: NEW → ASSIGNED

Comment 3

11 years ago
(In reply to comment #0)
> The JavaScript Tests (jsDriver.pl with the -R option) produce output
> of the form:
> 
> <jstest: ecma/Array/15.4-1.js bug:  result: PASSED type: shell description: var
> myarr = new Array(); myarr[Math.pow(2,32)-2]='hi'; myarr[Math.pow(2,32)-2]
> expected: hi actual: hi reason: >

> The format of these files is of the form:
> 
> <TEST_ID=ecma/Array/15.4-1.js, TEST_BRANCH=1.9.0, TEST_RESULT=PASSED,
> TEST_BUILDTYPE=debug, TEST_TYPE=browser, TEST_OS=win32,
> TEST_MACHINE=qm-winxp03, TEST_PROCESSORTYPE=unknown,
> TEST_KERNEL='1.5.24(0.156/4/2)', TEST_DATE=2007-09-28-19-07-39-0400,
> TEST_TIMEZONE=-0400, TEST_DESCRIPTION=var myarr = new Array();
> myarr[Math.pow(2,30)-2]='hi'; myarr.length expected: 1073741823 actual:
> 1073741823 reason:>
> 
> The known failures follow a similar format with the addition of the
> ability to use Perl regular expressions to combine failures across
> branches, build types, platforms, etc. into a more compact form.
> 
> <TEST_ID=e4x/Expressions/11.1.4-08.js, TEST_BRANCH=(1.8.1|1.9.0),
> TEST_RESULT=FAILED, TEST_BUILDTYPE=(debug|opt), TEST_TYPE=(browser|shell),
> TEST_OS=(linux|mac|win32), TEST_MACHINE=.*, TEST_PROCESSORTYPE=.*,
> TEST_KERNEL=.*, TEST_DATE=.*, TEST_TIMEZONE=.*, TEST_DESCRIPTION=Section 50 of
> test - 11.1.4 - XML Initializer - {} Expressions - 08 expected: true actual:
> false reason: Expected value 'true', Actual value 'false'>
> 
> In the fields before TEST_DESCRIPTION, normal Perl regular expressions
> are used unmodified. However in order to deal with the
> TEST_DESCRIPTION which may contain regular expression characters from
> the output of the JavaScript test, the patterns in TEST_DESCRIPTION
> use a quoted form where every individual regular expression character
> is surrounded by back-tics. For example, `.``*` to represent .*
> 
> For example:
> 
> <TEST_ID=e4x/Regress/regress-369032.js, TEST_BRANCH=1.8.1, TEST_RESULT=FAILED,
> TEST_BUILDTYPE=debug, TEST_TYPE=browser, TEST_OS=(linux|mac), TEST_MACHINE=.*,
> TEST_PROCESSORTYPE=.*, TEST_KERNEL=.*, TEST_DATE=.*, TEST_TIMEZONE=.*,
> TEST_DESCRIPTION=EXIT STATUS: CRASHED signal `[`1`-`9`]``[`0`-`9`]``*`
> (`[`0`-`9`.``]``+` seconds), BUGNUMBER: 369032; STATUS: Do not assert:
> kid2->parent == xml || !kid2->parent; Assertion failure: kid2->parent == xml ||
> !kid2->parent, at `.``*`jsxml.c:`[`0`-`9`]``+`>

This output looks suspiciously like JSON, albeit with inconsistent syntax and uneven escaping; perhaps use JSON and keep everything regular and parsable in case someone wished to use this output in the future?
(In reply to comment #3)

> 
> This output looks suspiciously like JSON, albeit with inconsistent syntax and
> uneven escaping; perhaps use JSON and keep everything regular and parsable in
> case someone wished to use this output in the future?
> 

Thanks for the back handed use of the term "suspiciously". I guess that reflects your opinion better than the use of "very much like". 

Care to be specific about the inconsistencies you found? 

I never considered JSON but only thought of what would be easiest for performing matching and for reading|writing the failure lists, but I suppose JSON output might be useful to someone. Having to deal with actually making the failures be valid JSON might be a pain though. I would prefer to go with what I have at the moment and get everything checked in and let Rob keep moving on getting buildbot managing automated running and reporting of the tests. Either myself or someone else can look at replacing the output format with a JSON format or can work on replacing the entire thing with something else. A filter that replaced the current format with JSON would be easiest and my first choice.

I chose the selective bac tic quoting of the regular expression characters in the description and not the other fields to minimize the need to use the bac tics in the other fields since they are a pain. I chose quoting with bac tics after a long period of trying to use back slash escapes for the regular expression characters in the description. Using a single escaping character was very difficult and I found quoting to be more reliable in terms of turning the description into a valid regular expression.

Comment 5

11 years ago
(In reply to comment #4)
> Thanks for the back handed use of the term "suspiciously". I guess that
> reflects your opinion better than the use of "very much like". 

I'm not sure whether I offended you or not, but if I did, I didn't mean to.  I was just commenting that inventing/dealing with new escaping formats seems like avoidable pain.

> Care to be specific about the inconsistencies you found? 

Only the Perl regex escaping you mentioned, which didn't seem from a skim of the comment to be used for the other entries, that's all.

> Having to deal with actually making the failures be valid JSON might be a
> pain though.

I was assuming there's some standard-ish Perl module for creating JSON; if you'd have to hand-generate that'd indeed be a bit more painful.  In that case, yeah, doing something else for now at least makes sense.
(In reply to comment #5)
> 
> I'm not sure whether I offended you or not, but if I did, I didn't mean to.  I
> was just commenting that inventing/dealing with new escaping formats seems like
> avoidable pain.
> 

Ok. The choice of word seemed to imply lack of ethical or moral behavior. Perhaps I'm too sensitive.

> > Care to be specific about the inconsistencies you found? 
> 
> Only the Perl regex escaping you mentioned, which didn't seem from a skim of
> the comment to be used for the other entries, that's all.
> 

The problem the back tic quoting is meant to resolve is the test result output contains regular expression characters but I wanted to be able to match the test results against known failures using regular expressions. In the example above I wanted to match "Assertion failure: kid2->parent == xml ||
!kid2->parent" occuring in jsxml.c. 

The question was how to distinguish the regular expression characters that occur naturally in the test result output (e.g. |) from those used to match groups of test result messages (e.g. .* or [0-9]+). I am very open to suggestions on how to do this better.

> > Having to deal with actually making the failures be valid JSON might be a
> > pain though.
> 
> I was assuming there's some standard-ish Perl module for creating JSON; if
> you'd have to hand-generate that'd indeed be a bit more painful.  In that case,
> yeah, doing something else for now at least makes sense.
> 

The pain I am worried about is the pain of matching test results to known failures and not so much the pain of generating the JSON. Getting the matching to work reasonably well in producing lists of possible fixes and regressions is the main goal. I'm not that happy with the approach I ended up using and am open to suggestions for improvement in the future.

<http://search.cpan.org/~makamaka/JSON-1.14/lib/JSON.pm> looks like a good candidate, but if there are others that are better that would be great too.

The reason I care so much about using the test result output in matching to the known failures is that just matching solely on a PASS|FAIL basis won't catch cases where a regression causes a test which has a known failure to start failing with a different error message or output which has occurred in the past.

JSON has an RFC, but bc says this fix will help rcampbell in the short run, so it can certainly go in with a followup bug to be JSON-standard later.

/be
Filed Bug 398178 on JSON support.

Checking in Makefile;
/cvsroot/mozilla/js/tests/Makefile,v  <--  Makefile
new revision: 1.14; previous revision: 1.13
done
Removing Maketests;
/cvsroot/mozilla/js/tests/Maketests,v  <--  Maketests
new revision: delete; previous revision: 1.13
done
RCS file: /cvsroot/mozilla/js/tests/create-patterns.pl,v
done
Checking in create-patterns.pl;
/cvsroot/mozilla/js/tests/create-patterns.pl,v  <--  create-patterns.pl
initial revision: 1.1
done
Checking in jsDriver.pl;
/cvsroot/mozilla/js/tests/jsDriver.pl,v  <--  jsDriver.pl
new revision: 1.72; previous revision: 1.71
done
RCS file: /cvsroot/mozilla/js/tests/known-failures.pl,v
done
Checking in known-failures.pl;
/cvsroot/mozilla/js/tests/known-failures.pl,v  <--  known-failures.pl
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/js/tests/post-process-logs.pl,v
done
Checking in post-process-logs.pl;
/cvsroot/mozilla/js/tests/post-process-logs.pl,v  <--  post-process-logs.pl
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/js/tests/public-failures.txt,v
done
Checking in public-failures.txt;
/cvsroot/mozilla/js/tests/public-failures.txt,v  <--  public-failures.txt
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/js/tests/runtests.sh,v
done
Checking in runtests.sh;
/cvsroot/mozilla/js/tests/runtests.sh,v  <--  runtests.sh
initial revision: 1.1
done
Checking in test-browser.sh;
/cvsroot/mozilla/js/tests/test-browser.sh,v  <--  test-browser.sh
new revision: 1.4; previous revision: 1.3
done
Checking in test-shell.sh;
/cvsroot/mozilla/js/tests/test-shell.sh,v  <--  test-shell.sh
new revision: 1.4; previous revision: 1.3
done
Checking in test.sh;
/cvsroot/mozilla/js/tests/test.sh,v  <--  test.sh
new revision: 1.3; previous revision: 1.2
done
Removing userhook-e4x.js;
/cvsroot/mozilla/js/tests/userhook-e4x.js,v  <--  userhook-e4x.js
new revision: delete; previous revision: 1.8
done
Removing userhook-js.js;
/cvsroot/mozilla/js/tests/userhook-js.js,v  <--  userhook-js.js
new revision: delete; previous revision: 1.9
done
Removing userhookeach-e4x.js;
/cvsroot/mozilla/js/tests/userhookeach-e4x.js,v  <--  userhookeach-e4x.js
new revision: delete; previous revision: 1.8
done
Removing userhookeach-js.js;
/cvsroot/mozilla/js/tests/userhookeach-js.js,v  <--  userhookeach-js.js
new revision: delete; previous revision: 1.8
done
RCS file: /cvsroot/mozilla/js/tests/userhookeach.js,v
done
Checking in userhookeach.js;
/cvsroot/mozilla/js/tests/userhookeach.js,v  <--  userhookeach.js
initial revision: 1.1
done

Status: ASSIGNED → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.