Closed Bug 505250 Opened 15 years ago Closed 13 years ago

AJAX does not show Greek characters, response is empty, encoding with UTF-8 or ISO8859-7

Categories

(Firefox :: General, defect)

x86
macOS
defect
Not set
major

Tracking

()

VERIFIED INCOMPLETE

People

(Reporter: spyros.papantoniou, Unassigned)

Details

(Whiteboard: FIXED)

Attachments

(2 files)

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1

When submitting a form via AJAX, no response is received if there are Greek UTF-8 characters in the form. The non latin-1 characters seem not to be transmitetd correctly to the server. When using IE or Safari on teh same application there is no problem. If teh AJAX query contains only latin characters but the response conteians Greek charachers then they are correctly receipved and diisplayed. It seems FireFox cannot send properly the non english characters to the server.

Reproducible: Always

Steps to Reproduce:
1. Have a form that makes a mySQL query on teh addressbook
2. If the name includes Greek characters no result is received
3. If teh name includes latin characters (among Greek characters) a contains query on the latin characters of teh name yield steh correct results that include the Greek characters.
Actual Results:  
Have screen shots.


The application works correctly on IE and Safari. This prohibits the use of FirefOx in Greece whenever AJAX is used.

It seems that the encoding of teh form sent to our server is not performed correctly when doing an XMLhttpRequest:

code used:
function callNoResultPost(debug, url, params)
{
    if ( debug )
        alert("AJAX Method: " + url + "?" + params);
    var xmlObj = null;
    if(window.XMLHttpRequest){
        xmlObj = new XMLHttpRequest();
        xmlObj.open ('POST', url, false); // false = sync call
        //xmlObj.setRequestHeader("Content-type", "application/x-www-form-urlencoded;"); original, chnaged in next line to try enforce encoding
         xmlObj.setRequestHeader("Content-type", "text/xml;charset=UTF-8");
        xmlObj.setRequestHeader("Content-length", params.length);
        xmlObj.setRequestHeader("Connection", "close");
        xmlObj.send(params);
    } else if(window.ActiveXObject){
        xmlObj = new ActiveXObject("Microsoft.XMLHTTP");
        xmlObj.open ('POST', url, false); // false = sync call
        xmlObj.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
        xmlObj.setRequestHeader("Content-length", params.length);
        //xmlObj.setRequestHeader("Connection", "close"); // ie6 bug, may be removed when ie6 is gone
        xmlObj.send(params);
    } else {
        alert("Browser does not support AJAX");
        return;
    }
    if (debug)
        alert("AJAX Method: "+xmlObj.responseText);
    if (hasError(xmlObj))
        return false;
    return true; 
}
Use an extension such as Firebug or Live HTTP Headers and post the outgoing headers on your XHR.
"post the outgoing headers on your XHR." 
please explain, I don know FireBug, installed; but where are the headers?
When I send the form, an alert shows that it is escaped like: Κα = %CE%9A%CE%B1  
this is sent ok via Safari and IE, but not so in FireFox. Where can I enforce the encoding=UTF-8 ?
Is this on every site you visit? Or is it the site you are developing?
Do you have an example site?

The encoding used for sending data, is the same encoding of the page.
But you may set the encoding specifically for a form with 'accept-charset',
see http://www.w3.org/TR/html401/interact/forms.html

Lucas
Its not a "form" actually, it a XMLhttpRequest in AJAX of our application. Its a pity that it does not work on FireFox, s it works in Safari and IE.
I need to enforce the encoding of teh XMLhttpRequest that leaves FireFox towards the server. The escaped sequence above from Safari is not sent correctly in FireFox.
Yes, I see now. I am reading the standard.

http://www.w3.org/TR/XMLHttpRequest/#setrequestheader

If you have text/xml then it follows the XML standard. It looks like you
have to set 'encoding' instead of 'charset'. But I am not sure.

I assume your webpage is not in UTF-8? 

Lucas
Everything is set to UTF-8, I cama across the following interesting hint:

http://squio.nl/blog/2006/06/27/xmlhttprequest-and-character-encoding/


that I will try out...
The problem is that httpRequest 'escapes' the sent string. If it were unescaped in UTF-8 maybe it would be ok.
Safari accepts the escaped string and manages to send it correctly to the Tomcat.
I don't have your problem clear. I assume:
- You are making a site, it doesn't appear on existing site.
- Your site is in UTF-8 (please check with the menu).
- You send data, with XMLHttpRequest
- The data is escaped when it contains Greek characters.

Is this right?

The escaping looks like 'urlencoding'. So, that should be off.
And, remove the 'Content-length', because I think that is counted in bytes and not in characters.

Lucas
You mention 'ISO8859-7' in the title of this bug. Is it somewhere in your html page/code? If not, then it shouldn't pop up suddenly (I wouldn't be surprised when 'ISO8859-1' pops up, but for ...-7 that is unlikely.

Lucas
ISO8859-7 is the single byte Greek character set.
We need UTF-8 all teh way. I wonder how to make the AJAX stop escaping the Greek characters. The english charactrs coem unescaped. When I start typing Greek they are ascaped. Apparently Frfox does  not send the correct sequence. The escaping was discovered with alert() on safari. Firefox does not display the alert().
This is not a site its an application on Tomcat with Java.
Spyros, for investigating your problem, we need a reproducible test case.

You claim that if you do the following:

callNoResultPost(0, "", "Γειά σου κόσμος")

then Firefox will send an escaped sequence instead of an UTF-8 sequence.

Is that right?

Lucas
As in #1, install Firebug:

https://addons.mozilla.org/nl/firefox/addon/1843

Then, click on the bug on the bottom of the page.

Click on 'Net' which makes it possible to see all the traffic between browser and server.

Then click on 'XHR', to see the XMLHttpRequests.

It is really great!

Lucas
Thank you for the hint in the Firebug tool! Its great. However strange !!
It seems that Firefox sends the correct request:

method:gr.sip.prot.document.getSenderSelectionAjaxPicker,xsl:gr.sip.prot.document.getFieldSelectionAjaxPicker,field:sender,stamp:1248164852327,pattern:Καρδί

unescaped, the the pattern string. This shall get from the server all names like the string:
the response is empty:
<html><head><title>Company Picker</title><link type="text/css" rel="StyleSheet" media="screen" href="resource

?name=com.triboni.application.explorer.css" encoding="UTF-8"/><script>

          

          function focusItem(i)

          {

            var f = document.getElementById('f' + i);

            if (f)

              f.focus();

          }

          function select(id,val)

          {

            parent.document.dialog.sender.value=id;

            parent.document.dialog.sender_text.value=val;

            parent.focus();

            parent.document.dialog.sender_text.focus();

            parent.pickerClose();

            parent.pickerDone();

            parent.pickerUpdate_sender();

          }

          function highlight(td)

          {

            td.style.color='white';

            td.style.backgroundColor='#000099';

            td.style.cursor='default';

          }

          function unHighlight(td)

          {

            td.style.color='black';

            td.style.backgroundColor='white';

            td.style.cursor='default';

          }

        </script></head><body topmargin="0" text="#000000" rightmargin="0" onFocus="focusItem(1);" onClick

="parent.pickerClose();" marginwidth="0" marginheight="0" leftmargin="0" bgcolor="white"><table width

="100%" cellspacing="0" cellpadding="1" border="0"/></body></html>
The Firefox response is empty,
When running as a check the XHR in the browser I get correctly the response:
<html><head><title>Company Picker</title><link type="text/css" rel="StyleSheet" media="screen" href="resource

?name=com.triboni.application.explorer.css" encoding="UTF-8"/><script>
          function focusItem(i)
          {
            var f = document.getElementById('f' + i);

            if (f)

              f.focus();

          }

          function select(id,val)

          {

            parent.document.dialog.sender.value=id;

            parent.document.dialog.sender_text.value=val;

            parent.focus();

            parent.document.dialog.sender_text.focus();

            parent.pickerClose();

            parent.pickerDone();

            parent.pickerUpdate_sender();

          }

          function highlight(td)

          {

            td.style.color='white';

            td.style.backgroundColor='#000099';

            td.style.cursor='default';

          }

          function unHighlight(td)

          {

            td.style.color='black';

            td.style.backgroundColor='white';

            td.style.cursor='default';

          }

        </script></head><body topmargin="0" text="#000000" rightmargin="0" onFocus="focusItem(1);" onClick

="parent.pickerClose();" marginwidth="0" marginheight="0" leftmargin="0" bgcolor="white"><table width

="100%" cellspacing="0" cellpadding="1" border="0"><tr><a onFocus="focusItem(1)" href="javascript:select

('dcbsymugeuaaaaaaaalg','Καρδίτσης, Νίκος');"><td title="Καρδίτσης, Νίκος" style="color:black; font-family

: Arial, Helvetica, sans-serif; font-size: 8pt; font-weight: bold" onMouseOver="highlight(this);" onMouseOut

="unHighlight(this);" onFocus="highlight(this);" onClick="select('dcbsymugeuaaaaaaaalg','Καρδίτσης, 

Νίκος')" onBlur="unHighlight(this);" id="f1">Καρδίτσης, Νίκος</td></a></tr></table></body></html>
Shall I use WireShark to see what request really gets to the server?
Where is the problem? Firebug shows correct un-escaped XHR sequence sent to the server.
Why does the server not respond when Firefox is used? the html response from the server is empty. As if no pattern was sent.
Lucas:
From what I understand, I send a XHR:
 callTextResultPost(debug=false, url, params), where (from Firebug)
params= method:gr.sip.prot.document.getSenderSelectionAjaxPicker,xsl:gr.sip.prot.document.getFieldSelectionAjaxPicker,field:sender,stamp:1248198722163,pattern:Κα

The keyboard input is 'patten', this triggers merely a select statement "....where sender like'%pattern%' " on the server. In Firebug I see the correct POST, but an empty response whenever pattern is Greek.
The response is OK in Safari and OK in FireFox when pattern is english.
Spyros, I think you are not following standards.

http://en.wikipedia.org/wiki/POST_%28HTTP%29

You have to proper encode the request. I think you come in undefined behavior, the way you are doing.

If you have a dictionary with your parameters called 'submitParameters',
then use the following code:

            var parametersString = ""
            for (var i in submitParameters) {
                if (parametersString.length > 0) {
                    parametersString += "&";
                }
                parametersString += encodeURIComponent(i) + "=" +
                                    encodeURIComponent(submitParameters[i])
            }
            xmlObj.open("POST", url, true)
            xmlObj.setRequestHeader("Content-Type", "application/x-www-form-urlencoded")
            xmlObj.send(parametersString)


This way, you are following the standards, and it should work in any browser.
Should also work with any unicode.
Here is what we use:
function ajaxRequest(requestArg)
{
    var url = "/triboni/exec";
    var params = "request=" + requestArg;
    var debug = false;
    
    this.addParam = function (name, value)
    {
        params += ","+encodeURIComponent(name)+":"+encodeURIComponent(escapeMethodParameter(String(value)));    
    }

    this.setDebug = function ()
    {
        debug = true;
    }

    this.callNoResult = function () { return callNoResultPost(debug, url, params); }
    this.callTextResult = function () { return callTextResultPost(debug, url, params); }
    this.callXmlResult = function () { return callXmlResultPost(debug, url, params); }
    
    addParamStamp(this);
}
function callTextResultPost(debug, url, params)
{.......
 if(window.XMLHttpRequest){
        xmlObj = new XMLHttpRequest();
        xmlObj.open ('POST', url, false); // false = sync call
        xmlObj.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
        xmlObj.setRequestHeader("Content-length", params.length);
        xmlObj.setRequestHeader("Connection", "close");
        xmlObj.send(params);

The only difference I see is the "Content-length" and the 'true' in the open statement, that I commented out/changed  and still it does not work...
I suspect somewhere the UTF-8 is swallowed/distorted, therefore no response comes back...
Can you simplify the test case, by just making the 'params' a hard coded string. I get your program now with bits and pieces. A proper bug report contains a standalone testcase.

The POST method was invented before UTF-8. It is a proper unicode encoding, so it should not be a problem to send any Greek.
OK I will proceed with hardcoding a few Greek chars in the XHR.
Hardcoding in JavaScript (client side) before the XHR did not bring anything new.
Firebug shows correctly the string. However there is an empty html response from the server, ie the html response is there but without hits. As if the query did not find anything.

I will make a hardcoded query on the server, that sends greek results to the client irrespectively of the AJAX pattern chosen on the client...

If the query had latin chars like 'John' it would find 'Ανδρεας John Καρδιτσης' and display correctly. When I type a greek string [ like Ανδρ ] no result is returned.
Is it of any help to open the dev-server on the internet? This way you may try on your own.
Can you post your program with hard coded string and with as less as possible of any other code/html?

Then I will look if I can reproduce. I have a small PHP program that just returns the raw data. It is odd that you have something that behaves differently on IE/Safari and Firefox. But it is also odd, that you are the only person. Unicode and POST is rather common construction.

Lucas
Resolving incomplete, please reopen if you can help with comment 27.
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago
Resolution: --- → INCOMPLETE
When the response contains Greek chars empty response is received. Hence it seems that the sent string is not the ( only?) problem.
This has been tested with a fixed response string
Resolution: INCOMPLETE → FIXED
Sent two screen shots, one with Safari, one with FF. Both on the same application running the same form.
The FF result is empty.
Here is the xsl that calls the XHR:

                     
                       <!-- AJAX for options=ajaxPicker,multi-line: sender,recipient,cc,distribution -->
                       <!-- ajax drop down list used by sender,recipient,cc -->
                     
                     <xsl:when test="contains(property[@name='options'],'ajaxPicker') and (contains(property[@name='options'],'multi-line'))">
                     
                        <script src="resource?name=com.triboni.application.ajaxLib" language="JavaScript">var iedummy1 = null;</script>  
                        <script src="resource?name=gr.sip.prot.application.protDocs.ajaxPicker" language="JavaScript">var iedummy2 = null;</script>
                      
                          <xsl:variable name="rows" select="substring-after(property[@name='options'],':')"/>
                            <xsl:variable name="fieldTooLong" select="/output/method/method[@id='messages']/object[@type='message']/property[@name='fieldTooLong']"/>
                            <xsl:choose>
                              <xsl:when test="string-length($value)=0">
                                 <div id="{$field-name}_text_div">
                                   <textarea onFocus="" rows="{$rows}" name="{$field-name}_text" autocomplete="off" onKeyDown="pickerTab(this, event);"  onKeyUp="{$pickerScript};doKeyPicker(this, p_method, event, 266);document.dialog.{$field-name}.value=document.dialog.{$field-name}_text.value;" cols="40" id="gsmallblack">&#160;</textarea>
                                 </div>
                               </xsl:when>
                              <xsl:otherwise>
                                 <div id="{$field-name}_text_div">
                                   <textarea rows="{$rows}" name="{$field-name}_text" autocomplete="off" onKeyDown="pickerTab(this, event);"  onKeyUp="{$pickerScript};doKeyPicker(this, p_method, event, 266);document.dialog.{$field-name}.value=document.dialog.{$field-name}_text.value;" cols="40" id="gsmallblack"><xsl:value-of select="$value"/></textarea>
                                  </div>
                               </xsl:otherwise>
                            </xsl:choose>
                             <input type="hidden" name="{$field-name}" value="{$value}"/>   <!-- recipient,sender,distribution or cc field-names -->
                      </xsl:when>

This makes the displayed multi-line form filed for the recipients.
The JS called therein is:

/*
modified ajaxPicker, model=com.triboni.application.explorer.ajaxPicker.js
Used for extProtocol alignment
*/

/*
Java script support for ajaxPicker.
*/
var statusPickerDone = false;

function doKeyPicker(field, method, evt, width)
{ 
 var popup = null;
 var doExit = false;
 if (statusPickerDone)
     doExit = true;
 statusPickerDone = false;
 if (doExit)
     return;
 if(navigator.userAgent.match(/MSIE/)){
    popup = document.all['dropList'];
 }
 else {
    popup = document.getElementById('dropList');
 }
 popup.style.left = findPosX(field.name); // field must have a div with id=name_div
 popup.style.top = findPosY(field.name);
 popup.style.width = width-3;
 popup.style.visibility = 'visible';

 var m = new ajaxRequest(method);
 var leadInd = 0;
 var lead = '';
 leadInd = field.value.indexOf(',');
 if (leadInd != -1){ lead = field.value.substring(0,leadInd);  } else { lead = field.value };
 m.addParam('pattern',lead); // alert('pattern='+lead);//  field.value
// m.setDebug();  this where debug is ebabled, enable-Debug
 
 var resp = m.callTextResult();
 var respxml = m.callXmlResult();
 if (resp==null)
     return;
 window.dropList.document.open();
 window.dropList.document.write(resp);
 window.dropList.document.close();

} 

function findPosX(fldName)
{ 
    var curleft = 0;
    var obj = document.getElementById(fldName+'_div');
    if (obj==null){
        return 9; // default
    }
    if (obj.offsetParent)
    {
        while (obj.offsetParent) 
        {   //  this is for not(MSIE),
            curleft += obj.offsetLeft; 
            obj = obj.offsetParent;
        }
    }
    else if (obj.x)
        curleft += obj.x;
      if(fldName == 'extProtocol_text'){  curleft += 200;  }; // alert('curleft 250='+fldName+curleft);   make the window on right side of extProtocol
      if(fldName == 'sender_text'){  curleft += 260; };
       if(fldName == 'recipient_text'){  curleft += 260; };
        if(fldName == 'cc_text'){  curleft += 260;  };
        if(fldName == 'distribution_text'){  curleft += 260;  };
    return curleft;
}

function findPosY(fldName)
{
    var curtop = 0;
    var obj = document.getElementById(fldName+'_div');
    if (obj==null){
        return 64; // default
    }
    if (obj.offsetParent)
    {
        while (obj.offsetParent)
        {
            curtop += obj.offsetTop
            obj = obj.offsetParent;
        }
    }
    else if (obj.y)
        curtop += obj.y;
  //  return curtop+18;  for extProtocol no lowering.
   if(fldName == 'extProtocol_text'){   return curtop;} else { 
          if(fldName == 'sender_text'){    return curtop;} else { 
          if(fldName == 'recipient_text'){   return curtop; } else { 
         if(fldName == 'cc_text'){          return curtop;  }  else { 
         if(fldName == 'distribution_text'){   return curtop; } else { return curtop+18; }}}}};
}

and:
function ajaxRequest(requestArg)
{
    var url = "/triboni/exec";
    var params = "request=" + requestArg;
    var debug = false;
    
    this.addParam = function (name, value)
    {
        params += ","+encodeURIComponent(name)+":"+encodeURIComponent(escapeMethodParameter(String(value)));    
    }

    this.setDebug = function ()
    {
        debug = true;
    }

    this.callNoResult = function () { return callNoResultPost(debug, url, params); }
    this.callTextResult = function () { return callTextResultPost(debug, url, params); }
    this.callXmlResult = function () { return callXmlResultPost(debug, url, params); }
    
    addParamStamp(this);
}

function callNoResultPost(debug, url, params)
{
    if ( debug )
        alert("AJAX Method: " + url + "?" + params);
    var xmlObj = null;
    if(window.XMLHttpRequest){
        xmlObj = new XMLHttpRequest();
        xmlObj.open ('POST', url, false); // false = sync call
        xmlObj.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
        xmlObj.setRequestHeader("Content-length", params.length);
        xmlObj.setRequestHeader("Connection", "close");
        xmlObj.send(params);
    } else if(window.ActiveXObject){
        xmlObj = new ActiveXObject("Microsoft.XMLHTTP");
        xmlObj.open ('POST', url, false); // false = sync call
        xmlObj.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
        xmlObj.setRequestHeader("Content-length", params.length);
        //xmlObj.setRequestHeader("Connection", "close"); // ie6 bug, may be removed when ie6 is gone
        xmlObj.send(params);
    } else {
        alert("Browser does not support AJAX");
        return;
    }
    if (debug)
        alert("AJAX Method: "+xmlObj.responseText);
    if (hasError(xmlObj))
        return false;
    return true; 
}

function callXmlResultPost(debug, url, params)
{
    if ( debug )
        alert("AJAX Method: " + url + "?" + params);
    var xmlObj = null;
    if(window.XMLHttpRequest){
        xmlObj = new XMLHttpRequest();
        xmlObj.open ('POST', url, false); // false = sync call
        xmlObj.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
        xmlObj.setRequestHeader("Content-length", params.length);
        xmlObj.setRequestHeader("Connection", "close");
        xmlObj.send(params);
    } else if(window.ActiveXObject){
        xmlObj = new ActiveXObject("Microsoft.XMLHTTP");
        xmlObj.open ('POST', url, false); // false = sync call
        xmlObj.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
        xmlObj.setRequestHeader("Content-length", params.length);
        //xmlObj.setRequestHeader("Connection", "close"); // ie6 bug, may be removed when ie6 is gone
        xmlObj.send(params);
    } else {
        alert("Browser does not support AJAX");
        return null;
    }
    if (debug)
        alert("AJAX Method: "+xmlObj.responseText);
    if (hasError(xmlObj))
        return null;
    var xmlResp = xmlObj.responseXML;
    if (xmlResp==null)
        return null;
    return xmlResp.documentElement; 
}

function callTextResultPost(debug, url, params)
{
    if (debug)
        alert("AJAX Method: " + url + "?" + params);
    var xmlObj = null;
    if(window.XMLHttpRequest){
        xmlObj = new XMLHttpRequest();
        xmlObj.open ('POST', url, false); // false = sync call
        xmlObj.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
        xmlObj.setRequestHeader("Content-length", params.length);
        xmlObj.setRequestHeader("Connection", "close");
        xmlObj.send(params);
    } else if(window.ActiveXObject){
        xmlObj = new ActiveXObject("Microsoft.XMLHTTP");
        xmlObj.open ('POST', url, false); // false = sync call
        xmlObj.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
        xmlObj.setRequestHeader("Content-length", params.length);
        //xmlObj.setRequestHeader("Connection", "close"); // ie6 bug, may be removed when ie6 gone
        xmlObj.send(params);
    } else {
        alert("Browser does not support AJAX");
        return null;
    }
    if (debug)
        alert("AJAX Method: "+xmlObj.responseText);
    if (hasError(xmlObj))
        return null;
    return xmlObj.responseText; 
}

the ajaxRequest() is a url like: /triboni/exec?method:gr.sip.prot.document.getAjaxRecipient,xsl:getAjaxRecipient,field:distribution,pattern:Υπε

the pattern variable contains the Greek string. The method works fine when not called via AJAX.
Status: RESOLVED → UNCONFIRMED
Resolution: FIXED → ---
Summary: AJAX encoding with UTF-8 and ISO8859-7 → AJAX does not show Greek characters, response is empty, encoding with UTF-8 or ISO8859-7
Reporter, are you still seeing this issue with Firefox 3.6.13 or later in safe mode? If not, please close. These links can help you in your testing.
http://support.mozilla.com/kb/Safe+Mode
http://support.mozilla.com/kb/Managing+profiles

You can also try to reproduce in Firefox 4 Beta 8 or later, there are many improvements in the new version, http://www.mozilla.com/en-US/firefox/all-beta.html
Whiteboard: [CLOSEME 2011-1-30]
No reply, INCOMPLETE. Please retest with Firefox 3.6.13 or later and a new profile (http://support.mozilla.com/kb/Managing+profiles). If you continue to see this issue with the newest firefox and a new profile, then please comment on this bug.
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago13 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → VERIFIED
FIXED in Firefox 6 and later
Whiteboard: [CLOSEME 2011-1-30] → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: