510443 - BOM should be treated as whitespace by the parser

Status: RESOLVED FIXED

(Rhino Graveyard :: Core, defect)

Description

[rspeyer]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
Build Identifier: Rhino 1.7 release 2 2009 03 22

As per 7.2
Implying that:
- it should be valid as a token separator
- it should cause syntax errors when in the middle of an identifier/keyword.

Reproducible: Always

Comment 1

[rspeyer]

Attachment: made BOM chars be treated as whitespace

Comment 2

[rspeyer]

BTW, the BOM char is \uFEFF.

Comment 3

[rspeyer]

Attachment: Made BOM be treated as whitespace throughout the language

Including:
- parsing
- in \s regex patterns
- in parseInt and parseFloat
- in String.prototype.trim
- in ToNumber for strings

Also generally consolidated handling of whitespace.

Comment 4

[Norris Boyd]

Checking in src/org/mozilla/javascript/NativeGlobal.java;
/cvsroot/mozilla/js/rhino/src/org/mozilla/javascript/NativeGlobal.java,v  <--  NativeGlobal.java
new revision: 1.88; previous revision: 1.87
done
Checking in src/org/mozilla/javascript/NativeString.java;
/cvsroot/mozilla/js/rhino/src/org/mozilla/javascript/NativeString.java,v  <--  NativeString.java
new revision: 1.69; previous revision: 1.68
done
Checking in src/org/mozilla/javascript/ScriptRuntime.java;
/cvsroot/mozilla/js/rhino/src/org/mozilla/javascript/ScriptRuntime.java,v  <--  ScriptRuntime.java
new revision: 1.321; previous revision: 1.320
done
Checking in src/org/mozilla/javascript/TokenStream.java;
/cvsroot/mozilla/js/rhino/src/org/mozilla/javascript/TokenStream.java,v  <--  TokenStream.java
new revision: 1.78; previous revision: 1.77
done
Checking in src/org/mozilla/javascript/regexp/NativeRegExp.java;
/cvsroot/mozilla/js/rhino/src/org/mozilla/javascript/regexp/NativeRegExp.java,v  <--  NativeRegExp.java
new revision: 1.111; previous revision: 1.110
done
RCS file: /cvsroot/mozilla/js/rhino/testsrc/doctests/controlchars.doctest,v
done
Checking in testsrc/doctests/controlchars.doctest;
/cvsroot/mozilla/js/rhino/testsrc/doctests/controlchars.doctest,v  <--  controlchars.doctest
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/js/rhino/testsrc/doctests/whitespace.doctest,v
done
Checking in testsrc/doctests/whitespace.doctest;
/cvsroot/mozilla/js/rhino/testsrc/doctests/whitespace.doctest,v  <--  whitespace.doctest
initial revision: 1.1
done