415317 - Search API should accept non-ASCII characters

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: API, defect)

Description

[Atsushi Sakai]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3pre) Gecko/2008020104 Minefield/3.0b3pre Firefox/3.0b
Build Identifier: 

Search API does not accept non-ASCII characters.

Reproducible: Always

Steps to Reproduce:
1. Visit https://services.addons.mozilla.org/en-US/firefox/api/search/%E3%81%97%E3%81%84%E3%81%9F%E3%81%91

Actual Results:  
400 Bad Request


Expected Results:  
Returns search result.

https://addons.mozilla.org/en-US/firefox/search?q=%E3%81%97%E3%81%84%E3%81%9F%E3%81%91
performs search.
API should also perform search with non-ASCII characters.
It is very important for non-English users.

Comment 2

[dynamis (Tomoya ASAI)]

(In reply to comment #0)
> https://addons.mozilla.org/en-US/firefox/search?q=%E3%81%97%E3%81%84%E3%81%9F%E3%81%91
> performs search.
> API should also perform search with non-ASCII characters.
> It is very important for non-English users.

Not sure when and which patch solved this but as far as I tested now this is fixed. Correct result are shown when I search Japanese strings:
https://addons.mozilla.org/ja/firefox/search?q=%E3%83%84%E3%83%BC%E3%83%AB%E3%83%90%E3%83%BC&cat=all
https://addons.mozilla.org/en-US/firefox/search?q=%E3%81%97%E3%81%84%E3%81%9F%E3%81%91

Please confirm (especially in other languages too).

Comment 3

[Masahiro YAMADA]

https://addons.mozilla.org/en-US/firefox/search?q=%E3%81%97%E3%81%84%E3%81%9F%E3%81%91
still returns HTTP 400.
----------
Date: Fri, 13 Jun 2008 02:20:01 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.1.6
X-AMO-ServedBy: mrapp09
Content-Length: 0
Connection: close
Content-Type: text/html; charset=UTF-8

400 Bad Request

Comment 4

[Tony Mechelynck [:tonymec]]

(In reply to comment #3)
> https://addons.mozilla.org/en-US/firefox/search?q=%E3%81%97%E3%81%84%E3%81%9F%E3%81%91
> still returns HTTP 400.
> ----------
> Date: Fri, 13 Jun 2008 02:20:01 GMT
> Server: Apache/2.2.3 (Red Hat)
> X-Powered-By: PHP/5.1.6
> X-AMO-ServedBy: mrapp09
> Content-Length: 0
> Connection: close
> Content-Type: text/html; charset=UTF-8
> 
> 400 Bad Request
> 

Can you retest (and, if you still get a bad result, hit shift+control+R to clear your cache)? I get a page with しいたけ in the search box, and showing an extension named "中止ボタンがしいたけに見えて困る", by Hadakadenkyu.

Comment 5

[Masahiro YAMADA]

I tested again and now I can see result correctly.
But, when searching from Tool - Add-ons still not work.

Comment 6

[Masahiro YAMADA]

I tested searching
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9pre) Gecko/2008061008 Minefield/3.0pre

According to error console, this search does not work.
https://services.addons.mozilla.org/en-US/firefox/api/1.1/search/%E3%81%97%E3%81%84%E3%81%9F%E3%81%91/all/10/WINNT/3.0pre
--------------
Date: Fri, 13 Jun 2008 08:05:31 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.1.6
X-AMO-ServedBy: mrapp09
Content-Length: 0
Connection: close
Content-Type: text/html; charset=UTF-8

400 Bad Request

Comment 7

[Masahiro YAMADA]

Should I file issue on Comment 5 and Comment 6 as another bug?

Comment 8

[Atsushi Sakai]

(In reply to comment #2)
> (In reply to comment #0)
> > https://addons.mozilla.org/en-US/firefox/search?q=%E3%81%97%E3%81%84%E3%81%9F%E3%81%91
> > performs search.
> > API should also perform search with non-ASCII characters.
> > It is very important for non-English users.
> 
> Not sure when and which patch solved this but as far as I tested now this is
> fixed. Correct result are shown when I search Japanese strings:
> https://addons.mozilla.org/ja/firefox/search?q=%E3%83%84%E3%83%BC%E3%83%AB%E3%83%90%E3%83%BC&cat=all
> https://addons.mozilla.org/en-US/firefox/search?q=%E3%81%97%E3%81%84%E3%81%9F%E3%81%91

I said, such a query performs search, i.e. works fine. This is not problem.
The problem is mentioned in "Steps to Reproduce:."

Comment 9

[dynamis (Tomoya ASAI)]

(In reply to comment #8)
> I said, such a query performs search, i.e. works fine. This is not problem.
> The problem is mentioned in "Steps to Reproduce:."

I misunderstood sorry. API *also* have to support non-ascii.

Comment 10

[Alex Buchanan [:abuchanan]]

laura:  I think this is due to the XSS check in bootstrap.php

in bootstrap.php (line 114) there is an XSS check that throws a HTTP 400, it contains...
preg_match('/[^\w\d\/\.\-_!: ]/',$_GET['url'])

testing that with one of the faulty URLs above I got 1 match, which would cause the fail

not sure how to fix, maybe use utf characters in the preg_match?

Comment 11

[Laura Thomson :laura]

Attachment: Patch to avoid applying this test to API urls

Comment 12

[Michael Morgan [:morgamic]]

After applying the patch, I still don't get results for this:
http://morgamic.khan.mozilla.org/amo/site/en-US/firefox/api/1.1/search/%E3%81%97%E3%81%84%E3%81%9F%E3%81%91/all/10/WINNT/3.0pre

Though I do not get the 400 -- which is the patch meant to fix?

Comment 13

[Laura Thomson :laura]

Just the 400.  If you search for that term via regular AMO search, you'll get no results.

Comment 14

[Michael Morgan [:morgamic]]

Comment on attachment 328436 [details] [diff] [review]
Patch to avoid applying this test to API urls

Ok, that's the behavior I saw.

Comment 15

[Laura Thomson :laura]

In r16794.

Comment 16

[Samuel Sidler (old account; do not CC)]

Going to https://preview.addons.mozilla.org/en-US/firefox/api/search/%E3%81%97%E3%81%84%E3%81%9F%E3%81%91 returns and XML file with no search results (expected), vs https://services.addons.mozilla.org/en-US/firefox/api/search/%E3%81%97%E3%81%84%E3%81%9F%E3%81%91 which returns an HTTP error 400 (or a blank page in most browsers).

VERIFIED.

Comment 17

[Samuel Sidler (old account; do not CC)]

Filed bug 444522 as a follow up.

Comment 18

[Justin Scott [:fligtar]]

3.4.5 was pushed on Friday.