1015567 - eBay searchplugin: change InputEncoding to UTF-8 to support non-ASCII characters (shows diamond character with question mark on results page)

Status: RESOLVED INVALID

(Firefox :: Search, defect, P3)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Tested on Aurora 2014-05-23 on Windows 8.1

When using the eBay search plugin in an en-US Firefox and searching for a term with a non-ascii character like "Café", that character will be replaced with a placeholder diamond with question mark in the results page. Searching the term with the eBay website works fine.

Some locales are already using UTF-8 as encoding: http://mxr.mozilla.org/l10n-mozilla-aurora/search?string=InputEncoding&find=eBay&findi=&filter=^[^\0]*%24&hitlimit=&tree=l10n-mozilla-aurora

Comment 1

[Francesco Lodolo [:flod]]

That's definitely weird. Italian is not using UTF-8, and "caffé" is showing up just fine. 

I even think I tried to switch it to UTF-8, and it didn't work (but I'm not completely sure if it was eBay or a different searchplugin).

I wonder if it's related to the last change, for now only in en-US (coming soon in l10n repos). Mike, any idea?

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Nö, the same issue happens for German (e.g. in Aurora and Beta). The Italian eBay search engine has many more differences to the English one than the German has, e.g.

- no search suggestions
- different parameter name

Comment 3

[Francesco Lodolo [:flod]]

No idea why the German searchplugin was updated in 2011, and the Italian one no (I checked French, no updates there either)
http://hg.mozilla.org/l10n-central/de/rev/3669624dcef5

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Italian is different because eBay hadn't requested an update for that locale in bug 625062.

Comment 5

[Francesco Lodolo [:flod]]

(In reply to Archaeopteryx [:aryx] from comment #4)
> Italian is different because eBay hadn't requested an update for that locale
> in bug 625062.

I'm aware of that (therefore the link to the changeset), I don't understand why only German. I was starting this update assuming that all locales should use the same general structure.

Comment 6

[Francesco Lodolo [:flod]]

I've installed the en-US searchplugin in my build. I search for "caffè"

Request is sent to
http://rover.ebay.com/rover/1/711-47294-18009-3/4?mfe=search&mpre=http://www.ebay.com/sch/i.html?_nkw=caff%E8

I get a 301 and a redirect to
http://www.ebay.com/sch/i.html?_nkw=caff%E8&clk_rvr_id=649737727332&mfe=search

Which displays caff� everywhere in the page.

Switching encoding change the first url but not the result
http://rover.ebay.com/rover/1/711-47294-18009-3/4?mfe=search&mpre=http://www.ebay.com/sch/i.html?_nkw=caff%C3%A8

Then 301 to (note that nkw value doesn't change compared to ISO-8859-1)
http://www.ebay.com/sch/i.html?_nkw=caff%E8&clk_rvr_id=649735390481&mfe=search

If I search for caff%C3%A8 instead of caff%E8 it works
http://www.ebay.com/sch/i.html?_nkw=caff%C3%A8&clk_rvr_id=649735390481&mfe=search

I think the problem is in the redirect between the first and second URL.

Comment 8

[:Paolo Amadini]

Ah, strange that this bug did not show up as a duplicate when I filed bug 1039456.

The eBay redirector works correctly if I encode the URL parameter:

http://rover.ebay.com/rover/1/711-47294-18009-3/4?mfe=search&mpre=http%3A%2F%2Fwww.ebay.com%2Fsch%2Fi.html%3F_nkw%3Dcaff%25C3%25A8

This leads me to believe that the character set used in the redirector parameter is auto-detected, but the redirector itself uses the ISO-8859-1 encoding when translating the parameter into the target URL.

The Italian plug-in uses a different syntax for the redirector URL. Both encodings actually work:

http://rover.ebay.com/rover/1/724-51951-19398-1/4?satitle=caff%E8
http://rover.ebay.com/rover/1/724-51951-19398-1/4?satitle=caff%C3%A8

In this case, the redirector correctly uses UTF-8 for the target. This works for the English affiliate code as well:

http://rover.ebay.com/rover/1/711-47294-18009-3/4?satitle=caff%E8
http://rover.ebay.com/rover/1/724-51951-19398-1/4?satitle=caff%C3%A8

We should upstream a bug about the Rover infrastructure and/or add special support for redirectors requiring a double encoding of the URL and/or switch to the "satitle" parameter. In fact, searches containing ampersands are broken in the current English engine, and this requires one of the latter two solutions to be fixed.

Who can determine the correct approach or handle the eBay bug?

Comment 9

[:Paolo Amadini]

Correct URIs for the English affiliate code example:

http://rover.ebay.com/rover/1/711-47294-18009-3/4?satitle=caff%E8
http://rover.ebay.com/rover/1/711-47294-18009-3/4?satitle=caff%C3%A8

Comment 10

[Francesco Lodolo [:flod]]

:mconnor should have sent an email to our eBay contact, not sure if we have updates (he's also ready in needinfo).

While this is annoying for en-US, it's even worse for other locales, across platforms (we have a lot of locales shipping eBay on Fennec).

Comment 11

[Florian Quèze [:florian]]

Mike is already needinfo'ed here, but maybe kev know the situation with eBay, or can give Mike a gentle ping? :-)

Comment 13

[Florian Quèze [:florian]]

Mike Kaply may know what the situation is here, and if it's worth fixing.

Comment 14

[Mike Kaply [:mkaply]]

We're in the process of renegotiating the ebay contract. We should get new engine info as a part of that.

So I don't think there's anything to do with this now, but it's on the right people's radar.

It's unfortunate this has been around so long.

Comment 15

[Mike Connor [:mconnor]]

We've now removed eBay from all locales, marking this INVALID.