1172576 - Non-standard percent-encoding in URLs, produced by the "Add a Keyword for this Search..." context menu button

Status: RESOLVED DUPLICATE of bug 1121903

(Firefox :: Search, defect, P3)

Description

[Alexander Varbanov]

The "Add a Keyword for this Search..." function builds a URL template with a query string representing an HTML form.

A problem arises when the relevant markup contains characters beyond U+007F: Instead of in the format defined in RFC 3986 (http://tools.ietf.org/html/rfc3986), they are encoded in a non-standard format, using "%XX" and "%uXXXX" sequences (where each X is a hex digit).

In specific, characters from U+00A0 to U+00FF are encoded as "%XX" (where XX are the latter two characters in that code point). Characters beyond that are encoded in the UTF-16 format, each unit prefixed by "%u".

---

For an example, see the search bar on http://jisho.org/. The form has a hidden input element, named "utf8", with value "✓" (U+2713, ✓, CHECK MARK), which gets encoded as "%u2713", instead of the correct "%E2%9C%93".

Comment 1

[David]

This is also breaking "Add a Keyword for this Search..." on Github.

The ✓ utf8-forcing URL convention is being used by lots of websites due to its adoption in Ruby on Rails.

Comment 2

[Mark Banner (:standard8)]

Looks like it is likely this code that is breaking:

https://searchfox.org/mozilla-central/rev/38c88cbf4be87dfa0636d15a5d3599a8ea0d1a72/browser/actors/ContextMenuChild.jsm#173-212

Namely the use of escape() rather than encodeURIComponent() in escapeNameValuePair().

Changing all those calls makes the use cases here pass, but breaks the test: https://searchfox.org/mozilla-central/rev/38c88cbf4be87dfa0636d15a5d3599a8ea0d1a72/browser/components/places/tests/browser/browser_bookmarkProperties_addKeywordForThisSearch.js#57-61,75-79

Maybe we should just change the non-isURLEncoded instances?

Comment 3

[Mark Banner (:standard8)]

Just found the older bug 1121903, which this seems to be a duplicate of.