81203 - smart quote cannot be submit in html form

Status: RESOLVED FIXED

(Core :: Layout: Form Controls, defect, P2)

Description

[Oliver Seiler]

I've been seeing this for at least a wekk (since at least around May 9th),
and couldn't find an entry in Bugzilla for it.

The behaviour is pretty simple. If I type into a text or memo field in a
form, certain characters won't get displayed, but will appear as a
different character(s). If I paste those offending characters (I'm
typing this report in Notepad so I can do just that), the characters
appear just fine.

The characters that cause me the most problems:

@ becomes "
' becomes `
" by itself doesn't appear
"" becomes ``
` becomes #
# becomes /
^ becomes ?
[ by itself doesn't appear
[[ becomes ^^

It seems isolated to punctuation; I haven't seen any problems with letters and
numbers, and some punctuation seems unaffected (e.g., $, !)

This is either just a problem with displaying, and form submissions themselves
are unaffected, or this is actually changing what can be typed into the 
text field itself; either way there is a bug to report. This paragraph is
being typed directly into the browser, so as a test, here is the full set
of punctuation characters on a typical QWERTY keyboard across the top 
number keys (left-to-right, 1, 2, ..., 0): !"/$%?&*()

so, some test characters on t

Comment 1

[Jacob Kjome]

Hmmmm... what build are you using?  WFM on build 2001051604 on win2k (SP2)

I'm typing this whole comment directly into the textarea for Additional Comments
in Mozilla.

Here are all the special chars from left to right with a single space between each:

` ~ ! @ # $ % ^ & * ( ) - _ = + \ | [ { ] } ; : ' " , < . > / ?

here are some double chars that you mentioned:  "" [[

All seems to work fine.  No weirdness that I can see.

We'll see how this post turns out.  I report back if what is posted is different
than what I typed.

Jake

Comment 2

[Kathleen :Brade]

What OS are you using?  What keyboard?  Do you have any IME installed?  Are you 
seeing the same translation in Composer or Mail Compose?

In your preferences, what is your default character encoding set to?

Comment 3

[Jacob Kjome]

I reported the OS along with the Mozilla build #.  Again, it is Win2k (same as 
you) with Service Pack 2 installed.

I have a standard QUERTY keyboard US English.  

I don't use Mozilla Mail or Composer and I don't know what IME is????

default character encoding is Western ISO-8859-1 and my language of preference 
is Engish [en]

Jake

Comment 4

[Jacob Kjome]

Sorry brade,

Thought you were talkig to me and that you were the reporter...didn't look 
closely enough.  Anyway, that is my info.

Jake

Comment 5

[rods (gone)]

reassigning

Comment 6

[Oliver Seiler]

More details:

The OS is Win2K professional SP1. Standard US keyboard. No IME installed as far 
as I know. The same problem occurs in Composer and while composing mail.

Using the modern theme (changing doesn't make a difference). Preferred (only) 
language is English. Default character encoding is Western (ISO-8859-1).

I should point out that this problem also is occuring in the URL field in the 
top browser bar. No other application on the machine has exhibited this problem.

Comment 7

[rubydoo123]

assigning this to kin, setting to 9.2 and asking sujay if he can dup this -- I 
can't get it to do what the reporter states. Note, I cannot confirm this bug, 
leaving as unconfirmed

Comment 8

[sujay]

I cannot reproduce this at all...I tried 2000 and 98...composer and URL
fields...

Comment 9

[Oliver Seiler]

Well, I tried seeing if I could make this go away. In my regional settings, my
original settings were for my input locales were:

Input Language            Keyboard layout/IME
English (United States)   US
English (Canada)          US (this was default)
English (Canada)          Canadian French
English (Canada)          Canadian Multilingual Standard

I removed all of these but English (United States) and the problem went away.
Unfortunately I couldn't get the problem to reappear by reintroducing the other
entries, so *I* can't even seem to reproduce this anymore... Interesting
problem; given that I didn't have any problems in any of my other applications I
can only think that Mozilla has some subtle bug, but beyond that I couldn't
really say.

Comment 10

[Keyser Sose]

Marking WORKSFORME as per reporters comments.

Comment 11

[sujay]

verified.

Comment 12

[alastair-h]

I can still reproduce something similar to this bug nearly every time in
Mozilla0.9.1 (still to download 0.9.2 I'll check it later) at
www.nitcentral.com/discus.


Steps:
1) Open up a discussion thread.
2) Normally if I'm going to be writing a long posting I'll use Word97 to type
things up instead of the browser incase something nasty happens and a browser
window crashes taking the rest with them. Include chars such as ", ',. and so on.
3) Copy what you've typed up into the box provided at the bottom of a discussion
and hit "Preview/Post".
4) The top preview post should be OK but if you look again at the messagebox
under it alot of the words like "it's" and "wasn't" have been converted to
"it?s" and "wasn?t". However, not /all/ of the chars are converted to ?s. I've
even had three dots (...) converted into one ?

Reproducible: Almost allways (95% of the time I'd say) on two differnt PCs (same
moz build)
Platform: Win98SE
Mozilla version: 0.9.1
Mozilla Details: Mozilla/5.0 (Windows; U; Win98; en-US; rv:0.9.1) Gecko/20010607

Comment 13

[Jacob Kjome]

Alastair,

since you know how to reproduce the behavior, could you test this in the latest 
nightly build?  There have been many fixes between 0.9.1 and today, so the 
problem you are seeing very well may have been fixed.

If you still see the behavior in the most recent builds then re-open this bug.

Jake

Comment 14

[Kathleen :Brade]

Alistair--interesting...  Thanks for the comments.
If you can reproduce this in a new build, I would guess that the problem is a 
character encoding issue or similar.  I'm guessing that Word97 is doing the smart 
quotes thing and so the apostrophe and double-quote are the ascii ' and " you 
would get if you typed such characters in mozilla.  

Teruko--could you (or someone else on your team) try to reproduce this bug by 
typing some text in Word and then copy/pasting it into a textarea (such as a 
bugzilla comment)?  You might also try the url above.

Please reopen if this can be reproduced with a newer build.

Comment 15

[Jacob Kjome]

--> adding Allastair to cc list

Allastair, just thought you'd want to see brade's comments that were just 
posted.

Jake

Comment 16

[alastair-h]

Thanks for putting me on the CC list Jake.

I have a question to ask regarding the nightlies. I've read the relevant pages 
and have to admit to being a little baffled, which is why up until now I've 
stuck with the likes of 0.9.1.
 Is there an installer version like there is for the "completed" versions like 
0.9.1 and .2 or do I have to build the nightlies myself? If so I'd be greatful 
if someone could pass me a slightly revised (Read: Idiot friendly) copy of the 
building instructions for the mozilla-win32-talkback.zip as I haven't a clue 
whats going on and what I need after reading the build instructions at 
http://www.mozilla.org/build/win32.html

Cheers.

Comment 17

[Jacob Kjome]

Allastair, 

Just go here:  http://ftp.mozilla.org/pub/mozilla/nightly/latest-trunk/

Grab the build that works for your system.

If you are using Windows and want a zip build, just grab:
http://ftp.mozilla.org/pub/mozilla/nightly/latest-trunk/mozilla-win32-talkback.zip

for the installer build, grab:
http://ftp.mozilla.org/pub/mozilla/nightly/latest-trunk/mozilla-win32-installer.exe

I would just use the zip build.  Create a new directory like c:\apps\mozilla\nightly

then just unzip the zip build to that and run "mozilla.exe" in the "bin" directory.

Jake

Comment 18

[alastair-h]

Thanks for the help Jake.
Ok, I downloaded the latest nightly this morning (about 10am GMT) and unzipped 
it into the directory following Jakes instructions and then ran mozilla.exe.

Here's what happened:

Typed out the following line in Word97 - It's  wasn't  times… "text"  ain't   
they're
Copied it into the message box at http://www.nitcentral.com/discus and hit 
"preview/post message" and here was the result - It?s  wasn?t  times? "text"  
ain't   they're

As I said before, its odd how it only does it for /some/ of the characters. I've 
had the quote marks change into ?s before as well but after doing it five or six 
times I couldn't reproduce that.

Comment 19

[Jacob Kjome]

Re-opening based on the fact that Alastair tested with the newest nightly build 
and was still seeing the bahavior that he reported earlier.

Alastair, I'm assuming that you tested the "preview" functionality at 
http://www.nitcentral.com/discus with IE5.5 using exactly the same steps you 
used to produce the behavior in Mozilla.  If you haven't, could you please do 
that just to double check that this isn't something that is happening on the 
server side?

thanks,

Jake

Comment 20

[alastair-h]

Yes, I've been at that board for quite some time and have used both IE4/ 5.5 and 
NS4.7x many times and have never seen it before until I started to use Mozilla.

Comment 21

[Keyser Sose]

Marking NEW.

Comment 22

[kinmoz]

I can't reproduce this problem on my Win2k machine, but based on the reporters 
comments about the fact that form submissions are fine, I can only conclude that 
the content the editor is putting into the dom tree is correct, so this sounds 
like a rendering/font encoding problem.

ftang, your group still handles all text layout correct?

Passing this off to ftang and cc'ing bstell and myself.

Comment 23

[Frank Tang]

This seems a recycle of bug report here. 
The origional problem [e-mail] (Oliver Seiler) reported seems caused by
accidental switch to English (Canadian) keyboard. 
The new problem [e-mail] reported is a seperated issue, the
problem is we submit a ISO-8859-1 form with characters that we cannot encoded in
ISO-8859-1. 

What does IE and N4.x do ?

Comment 24

[Frank Tang]

This is regression from N4.x. We really need to fix this one.

Comment 25

[Frank Tang]

In form manager, if the document charset is "ISO-8859-1", instead of asking for
"ISO-8859-1" encoder, we should ask "windows-1252" encoder

Comment 26

[Frank Tang]

Change the summary to "smart quote cannot be submit in html form"

Comment 27

[Frank Tang]

Attachment: use windows-1252 for ISO-8859-1 when we try to submit

Comment 28

[kinmoz]

Isn't this a cross-platform change? Why would we use "windows-1252" on Mac and
Linux?

Comment 29

[Frank Tang]

move to m0.9.5

Comment 30

[Frank Tang]

mark as p2

Comment 31

[msanz]

nsbranch- since Frank moved it to 0.9.5

Comment 32

[J Luh]

Why windows-1252 and not utf-8?

See also bug 65697 and bug 70838

Comment 33

[sujay]

wrong qa_contact.

Comment 34

[nhottanscp]

In the patch, it is selecting an encoder for a generating string for the server.
The server charset is not always specified (by accept charset). It is fair to
send windows-1252 which is a superset of ISO-8859-1 to the server.
r=nhotta

Comment 35

[kinmoz]

It might be more efficient on some platforms if you used NS_LITERAL_STRING
instead of the runtime conversion methods:


+     if(charset.Equals(NS_LITERAL_STRING("ISO-8859-1")))
+       charset.Assign(NS_LITERAL_STRING("windows-1252"));


I understand that windows-1252 is a superset of ISO-8859-1, but is the
windows-1252 encoder available/active in Linux and Mac builds?

Comment 36

[Frank Tang]

>I understand that windows-1252 is a superset of ISO-8859-1, but is the
>windows-1252 encoder available/active in Linux and Mac builds?
The answer is YES.

Comment 37

[Frank Tang]

Attachment: v2 of the patch which address kin's comment

Comment 38

[Frank Tang]

nhotta- can you r= the new patch (v2)
kin can you sr it ?

Comment 39

[Kathleen :Brade]

Comment on attachment 52836 [details] [diff] [review]
v2 of the patch which address kin's comment

r=brade

Comment 40

[kinmoz]

Comment on attachment 52836 [details] [diff] [review]
v2 of the patch which address kin's comment

sr=[e-mail]

Comment 41

[Frank Tang]

fixed and check in

Comment 42

[Madhur Bhatia]

Verified fixed in the trunk build :- 2001-10-11-09trunk win2000.

It needs to be checked in the branch build. I checked in the today's branch 
build and i can still reproduce the bug there.

Adding keyword 'vbranch'

Comment 43

[Madhur Bhatia]

checked on win2000
BuildID: 2001-10-22-0.9.4 branch build -----
*** is not fixed here ****
when i copy paste the following in the  message box (url provided above):- 

It's  wasn't  times… "text"  ain't   they're

and hit preview/post message
I get the following output of the text I entered:-

It's  wasn't  times? "text"  ain't   they're

buildID: 2001-11-08-06-trunk build
***works fine here***

when is the fixed patch going to be checked iin the branch build?

Comment 44

[Simon Montagu :smontagu]

This could do with a regression test (in several different encodings)