297317 - IDN not supported (no UTF-8 support)

Status: RESOLVED FIXED

(Webtools Graveyard :: Reporter, defect)

Description

[Hixie (not reading bugmail)]

STEPS TO REPRODUCE
 1. Go to http://☺.damowmow.com/. (Or, go to http://damowmow.com/portal/ and 
    click the header at the top.)
 2. Click Help | Report Broken Web Site.
 3. Choose Problem Type: Other Problem.
 4. Click Next.
 5. Look up the report in http://reporter.mozilla.org/.

ACTUAL RESULTS
   URI on site is "http://?.damowmow.com/".
   http://reporter-test.mozilla.org/app/report/?report_id=RMO11184143084248

EXPECTED RESULTS
   URI should be "http://☺.damowmow.com/".

Comment 1

[Robert Accettura [:raccettura]]

It's being dropped somewhere before the database server.  That leaves: Mozilla,
or the server to blame.

Anyone willing to sniff packets and see how mozilla sends the request?  Is it
UTF-8 when it's sent to the server?

you can use reporter-dev.mozilla.org for testing purposes.

set extensions.reporter.serviceURL to http://reporter-dev.mozilla.org/service/

the dev server has some updated libraries, so it may be worth checking there
just to see if it's resolved on it's own.

Comment 2

[Robert Accettura [:raccettura]]

I could use some help diagnosing this.

Comment 3

[Robert Accettura [:raccettura]]

Attachment: Dump

Looks like it's client side somewhere:

<url xmlns:ns1="http://www.w3.org/1999/XMLSchema"
xsi:type="ns1:string">http://....damowmow.com/</url>

should be sending it as utf-8.

Comment 4

[Robert Accettura [:raccettura]]

Ok, it doesn't appear to be the service.  I'm looking elsewhere.

Comment 5

[Hixie (not reading bugmail)]

Uh, look closer (at the hex). What you're being sent is e2 98 ba, which looks to
me like the UTF-8 encoded form of a ☺. (It appears as "."s in the ASCII side
because those are high-bit characters.) So I would say that the client is doing
the right thing.

Comment 6

[Robert Accettura [:raccettura]]

Yes, I (in the past few hours) agree.

I think it's database related.  I'll look into it closer in coming days.

Comment 7

[Doron Rosenberg (IBM)]

Perhaps php (or whatever the sever side script is written in) is not decoding it
correctly?

But yeah, the soap/web service code is fully utf-8 aware :)  I assume you are
using a mysql db?  http://www.shawnolson.net/a/946/ might help.

Comment 8

[Robert Accettura [:raccettura]]

Thanks Doron.

Unfortuantely we are on PHP 4.3.x and mySQL 4.0.x.  I'm still researching what's
going on, and where we are dropping the ball.

I started client side, just in case something was broken there (so we could get
it fixed quick before 1.1).  Since it's server side, we can deploy after
release.  Were not bound to the restraints of the release cycle.

Comment 9

[Jungshik Shin]

(In reply to comment #8)

> Unfortuantely we are on PHP 4.3.x and mySQL 4.0.x.  I'm still researching 

  PHP 4.3.x can support multibyte encodings including UTF-8 fine with mbstring
and iconv modules. In case of mySQL 4.0.x, I'm not sure, but you can just store
UTF-8 strings. Sorting and character counting wouldn't work properly, but data
'loss' can be avoided with a little 'manual' fix-up.
 

Comment 10

[Robert Accettura [:raccettura]]

jshin: correct.  Though that's up to justdave and our super admin's to
evaluate/fix.  Were not going to be on the current server for too long.  Not
sure what the new servers will be running.

Comment 11

[Robert Accettura [:raccettura]]

renaming to be a bit more accurate.

Comment 12

[Robert Accettura [:raccettura]]

*** Bug 308598 has been marked as a duplicate of this bug. ***

Comment 13

[Robert Accettura [:raccettura]]

*** Bug 322342 has been marked as a duplicate of this bug. ***

Comment 14

[Robert Accettura [:raccettura]]

This is fixed in the new version of reporter.  The problem was on the server side, so when the new server is deployed, all clients will be functioning correctly (no client side update needed)

It's not deployed as of yet, but the bug is fixed.  Someone can verify when we upgrade.

Comment 15

[Paul Kim]

When will the upgrade be deployed?

Comment 16

[Robert Accettura [:raccettura]]

 (In reply to comment #15)
> When will the upgrade be deployed?
> 

Hopefully sooner than later.  I'm hoping to be feature complete by the end of the month, and perhaps do some testing in February.

The other option is to back-port this patch to the production instance.  I can talk to Dave Miller about that.

I'd obviously prefer to wait at this point.  We don't seem to have that many reports effected, and only a hand full of bugs after several months since reporter went into production.

This might be a discussion worth having.  Not sure.

Comment 17

[Robert Accettura [:raccettura]]

*** Bug 325166 has been marked as a duplicate of this bug. ***

Comment 18

[dynamis (Tomoya ASAI)]

It's already March but actually not resolved yet.
So, I reopen this bug.

Here is test reports with multibyte string and IDN.
http://reporter.mozilla.org/app/report/?report_id=RMO11444886628826&submit_reportID=Lookup+Report
http://reporter.mozilla.org/app/report/?report_id=RMO11444885744102&submit_reportID=Lookup+Report

Comment 19

[dynamis (Tomoya ASAI)]

Oops, it's already April, not March.

Comment 20

[Robert Accettura [:raccettura]]

Please don't re-open bugs without cause.

The bug is resolved.

The production instance has not been updated with this code.  That's another topic, and another component.  This bug is specific to the code, not instances hosted by mozilla.org.  Reopening bugs like this makes it rather hard to track my todo list.

The server will be upgraded when the code is stable.  That still is a few weeks away.  My current target is by the end of May.  Yes it's slipped, and I apologize for that that.

Thanks.

Comment 21

[Hiro]

It still reproduces though it tried today. 
http://reporter.mozilla.org/app/report/?report_id=RMO11577378288494&submit_reportID=Lookup+Report

Comment 23

[Po-chiang Chao [:bobchao]]

Is there anyone working on this? "That still is a few weeks away," and 2 YEARs passed. People need this tool to report website, or please remove the menuitem in Help menu of builds other than English.

Comment 24

[Po-chiang Chao [:bobchao]]

Test again, the description part in new reports successfully display UTF-8, but the old ones still shows "?".