462208 - Add PCRE test suite

Status: RESOLVED INACTIVE

(Core :: JavaScript Engine, defect)

Description

[David Mandelin [:dmandelin]]

We should add the PCRE test suite (used in WebKit) to test correctness of our regexes, especially with the ongoing compilation work in bug 461050.

Comment 1

[Brendan Eich [:brendan]]

Beware we deviate from ECMA-262 Edition 3 and (no surprise) Perl on a number of edge cases, for web compatibility. We don't want to consider these deviations to be bugs without more evidence than the PCRE test suite barking (if it does bark in the night).

/be

Comment 2

[David Mandelin [:dmandelin]]

I plan to just regress against current regexes to start with. We could also compare with WebKit to look for other bugs. Running PCRE directly is going to produce a lot of false positives, I think, because their spec is different in the corners, and the test suite looks like it goes after the corners pretty hard.

Just getting it to run at all is going to be some amount of work due to those differences, but so far I got to run a few hundred PCRE cases against the regex->native version and the results are identical to the standard engine.

Crowder suggested that recording regexes and their inputs from a web browsing session would be a better way of generating test cases.

Comment 3

[Mike Shaver (:shaver emeritus)]

Does WebKit not have test suite adaptations to account for the differences between PCRE and ECMA?  I thought they did, or perhaps v8.

Comment 4

[David Mandelin [:dmandelin]]

Per Crowder's suggestion, I read off all the regexps run by mochitest, which gave about 600,000 of them, and then sampled another 250,000 or so by surfing around some top sites a bit. On these, SpiderMonkey regexps, jitted regexps, and WebKit regexps give identical results.

Comment 5

[David Mandelin [:dmandelin]]

Attachment: Patch to save regexps executed

Comment 6

[David Mandelin [:dmandelin]]

Attachment: Scripts to generate and run regexp tests

These scripts aren't entirely "finished"--they require a little editing of input/output file names to run.

Comment 7

[David Mandelin [:dmandelin]]

Attachment: Extracted PCRE tests

These are the regexps executed by the WebKit PCRE suite in a format usable with the test harness in the scripts attachment, minus one that runs forever. TraceMonkey behaves the same on this suite using the -j flag. 

There are some variances between SM and WebKit in these tests. I filed one of them as bug 468000. I think many of the others are of the same kind.

Comment 8

[André Bargull [:anba]]

Closing old bug without progress. Maybe even something which makes more sense for test262 nowadays?