Closed Bug 730207 Opened 12 years ago Closed 12 years ago

QuickSearch doesn't return data for a search term with german umlauts like ä,ö,ü

Categories

(Bugzilla :: Query/Bug List, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED
Bugzilla 4.2

People

(Reporter: Bastian.Wassermann, Assigned: LpSolit)

References

()

Details

(Keywords: regression, Whiteboard: [blocker will fix])

User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; Tablet PC 2.0)

Steps to reproduce:

I type in a search term in the quicksearch (in top bar) with a german umlaut like ä,ü,ö for example "Übernahme" or "ALL Übernahme".

Webserver is IIS 7.5
MySql is 5.5.20 (utf8)
Perl Version is 5.12.2
SOAP::Lite is 0.714


Actual results:

I get an empty buglist


Expected results:

I should get a list with many bugs mentioning "Übernahme".
I can indeed reproduce in both 4.0.5 and 4.2, but not in 3.6.8, so this is a regression. The weird thing is that the SQL query generated by QS is exactly the same as the one returned by the Advanced Search page, and executing this query from the mysql CLI returns the correct data. So maybe the SQL query itself is fine, but the bug is excluded by QS for some other reason.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: regression
Target Milestone: --- → Bugzilla 4.2
I see the same issue but with postgresql on *NIX platform.

Postgres database is using UTF8, changing client_encoding in postgresql.conf does not help.


Software error:
DBD::Pg::st execute failed: ERROR:  invalid byte sequence for encoding "UTF8": 0xf6273a3a
... at /usr/local/www/bugtracker/buglist.cgi line 908


I changed line buglist.cgi 908 with this diff:

--- buglist.cgi.orig    2012-02-28 17:53:32.000000000 +0100
+++ buglist.cgi 2012-02-28 17:55:27.000000000 +0100
@@ -904,7 +904,7 @@
 $::SIG{PIPE} = 'DEFAULT';

 # Execute the query.
-my $buglist_sth = $dbh->prepare($query);
+my $buglist_sth = $dbh->prepare(Encode::encode ("UTF8", $query) );
 $buglist_sth->execute();


Now a single umlaut does not throw the exception, but a word containing a umlaut.


New error:
DBD::Pg::db selectcol_arrayref failed: ERROR:  invalid byte sequence for encoding "UTF8": 0xe4747a
... called at Bugzilla/Search.pm line 2069


It seems working with now with this additional diff

--- Bugzilla/Search.pm.orig     2012-02-28 18:16:12.000000000 +0100
+++ Bugzilla/Search.pm  2012-02-28 18:16:49.000000000 +0100
@@ -1271,7 +1271,7 @@
     my $q = "SELECT $inner FROM $table WHERE $cond";
     #return "$outer IN ($q)";
     my $dbh = Bugzilla->dbh;
-    my $list = $dbh->selectcol_arrayref($q);
+    my $list = $dbh->selectcol_arrayref(Encode::encode ("UTF8", $q) );
     return "1=2" unless @$list; # Could use boolean type on dbs which support it
     return $dbh->sql_in($outer, $list);}

--
Regards,
olli
Summary: Doesnt get search results for a search term with german umlauts like ä,ö,ü Doesnt get search results for a search term with german umlauts like ä,ö,ü → QuickSearch doesn't return data for a search term with german umlauts like ä,ö,ü QuickSearch doesn't return data for a search term with german umlauts like ä,ö,ü
Marc: any idea about this issue? I suppose you see this problem every day.
Since bug 314364, QuickSearch is not being used as intensely any more.

I can reproduce the bug, too. I'm curious which change caused the regression...
Does bug 659816 come into play here, perhaps?
I forgot to mention that clicking „Suchkriterien bearbeiten“ (“Edit this search”) leads to a query which does exactly what the QuickSearch should have done in the first place.
I think I know how to fix this problem nicely. So marking as blocker for now.
Flags: blocking4.2.1+
Depends on: 554819
I have a patch ready. For trunk, it will be fixed by my patch from bug 554819. IMO, we should take it for 4.2 too, despite all the heavy changes I made. If not, I will look for a reduced patch.
Assignee: query-and-buglist → LpSolit
Status: NEW → ASSIGNED
Whiteboard: [blocker will fix for 4.4]
Blocks: 720634
Fixed by bug 554819, which landed in 4.2 too.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Whiteboard: [blocker will fix for 4.4] → [blocker will fix]
Arrg, was bitten at the moment again by this bug after updating to 4.0.9 with PgSQL as backend.

Luckily my simple patch in comment2 is working good enough for me.
Please can you fix this within 4.0.10?
(In reply to olli hauer from comment #11)
> Luckily my simple patch in comment2 is working good enough for me.
> Please can you fix this within 4.0.10?

No, 4.0.x is restricted to security bugs only. You should upgrade to 4.2.4 or 4.4rc1 instead, or stop using umlauts in your password till your installation is upgraded.
The issue is *not* a password, it is the *search form*.
That's what you get if someone fires a search request with umlauts

Software error:

DBD::Pg::db selectcol_arrayref failed: ERROR:  invalid byte sequence for encoding "UTF8": 0xfc
HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding". [for Statement "SELECT products.id FROM products WHERE POSITION(LOWER('übertrag'::text) IN LOWER(products.name::text)::text) > 0"] at Bugzilla/Search.pm line 1274
	Bugzilla::Search::build_subselect('bugs.product_id', 'products.id', 'products', 'POSITION(LOWER(\'\x{fc}bertrag\'::text) IN LOWER(products.name:...') called at Bugzilla/Search.pm line 2068
	Bugzilla::Search::_product_nonchanged('Bugzilla::Search=HASH(0x29da2c58)', 'multi_fields', 'ARRAY(0x295c1e88)', 'sequence', 'SCALAR(0x295df424)', 'ff', 'SCALAR(0x295dfd20)', 'wherepart', 'ARRAY(0x295c1c94)', ...) called at Bugzilla/Search.pm line 1165
	Bugzilla::Search::do_search_function('Bugzilla::Search=HASH(0x29da2c58)', 'HASH(0x29d8e53c)') called at Bugzilla/Search.pm line 968
	Bugzilla::Search::init('Bugzilla::Search=HASH(0x29da2c58)') called at Bugzilla/Search.pm line 414
	Bugzilla::Search::new('Bugzilla::Search', 'fields', 'ARRAY(0x297db9ec)', 'params', 'Bugzilla::CGI=HASH(0x28d789d8)', 'order', 'ARRAY(0x297e2500)') called at /usr/local/www/bugtracker/buglist.cgi line 832
(In reply to olli hauer from comment #13)
> The issue is *not* a password, it is the *search form*.

Err, yes sorry. No idea why I said password. :) I meant stop using umlauts in bareword, or quote your words if you need to use umlauts, as Marc said in comment 6.
So is this fixed or isn't it? Or not for all versions? All DBs, or just some? Which dependent bugs need to be reopened, or does this bug here need to be reopened?
(In reply to Marc Schumann [:Wurblzap] from comment #15)
> So is this fixed or isn't it? Or not for all versions?

This is fixed in Bugzilla 4.2. And we won't fix it in 4.0 as it's restricted to security fixes only. No need to reopen anything.
You need to log in before you can comment on or make changes to this bug.