769844 - Special characters in TelemetryPing properties are lost when sent to the server

Status: RESOLVED FIXED

(Webtools Graveyard :: Telemetry Server, defect)

Description

[Nicolás Chaim]

Attachment: Repro

Non-ascii characters in the TelemetryPing payload are lost and replaced with a default character before the data is compressed in gzipCompressString(); UTF-8 characters are not preserved.

Comment 1

[Nicolás Chaim]

Attachment: Proposed patch

Comment 2

[Nicolás Chaim]

See related bug 767077

Comment 3

[Vladan Djeric (:vladan)]

Comment on attachment 638047 [details] [diff] [review]
Proposed patch

Looks, good but let's wait on pushing this until we can test this against Metrics' servers:

(7:53:26 PM) nchaim: proposing to update TelemetryPing to send data in UTF-8 format to the server (bug 769844)
(7:53:26 PM) nchaim: does this need any special considerations on the server side?
(7:59:28 PM) vladan: xstevens: would you know this ^^ type of thing?
(8:02:21 PM) xstevens: yeah…I've never tried sending that type of payload
(8:03:02 PM) xstevens: I use UTF-8 just about everywhere I can think of on my end
(8:03:31 PM) vladan: can we do a test ping and you can let us know if any issues come up?
(8:03:54 PM) vladan: do you have guys have a test server somewhere?
(8:05:41 PM) xstevens: not at the moment
(8:05:57 PM) xstevens: my boxes got blown away by ops this week
(8:06:12 PM) xstevens: I can ask them next week to give me some test machines
(8:17:39 PM) vladan: alright

Comment 4

[Nathan Froyd [:froydnj]]

Comment on attachment 638047 [details] [diff] [review]
Proposed patch

Review of attachment 638047 [details] [diff] [review]:
-----------------------------------------------------------------

This looks like it's on the right track, but can we avoid converting the string to UTF-8 twice?

Also, this needs a test that we can round-trip properties and/or values with non-ASCII characters.

Comment 5

[Nathan Froyd [:froydnj]]

As I understand it, we really only need this to deal with sending in plugin information, since we have no control over what encoding the plugin information is.

However, we've decided that we're not interested in general plugins at this point, just Flash.  So this bug is not worth fixing at this point.

Comment 6

[Vladan Djeric (:vladan)]

I thought this came up when reporting extension IDs, not plugins?

Comment 7

[Georg Fritzsche [:gfritzsche]]

(In reply to Vladan Djeric (:vladan) from comment #6)
> I thought this came up when reporting extension IDs, not plugins?

I originally stumbled upon this with plugins (and their basically free form text fields for names etc.).

Comment 8

[[:Cww]]

I'm doing an analysis of Search Engine usage (and the non-English search engines are full of non-English characters) and I'm getting a ton of literal \ufffd instead of properly encoded unicode codepoints. Any hope of getting this worked on soon?

Comment 9

[Mark Reid [:mreid]]

I can reproduce this - it appears to be a problem with the server validation/conversion code.

Comment 10

[Mark Reid [:mreid]]

I've sent a PR to fix this:
https://github.com/mozilla/telemetry-server/pull/95

Comment 11

[Mark Reid [:mreid]]

This fix has been deployed.