790192 - B2G SMS: support automatically replacing Latin characters with ASCII ones

Status: RESOLVED FIXED

(Core :: DOM: Device Interfaces, defect)

Description

[Vicamo Yang [:vicamo][:vyang]]

See Gaia issue https://github.com/mozilla-b2g/gaia/issues/3298.

For some countries, they want to use GSM 7-bit default alphabet encoding for characters that can be replaced with similar-in-shape ASCII ones. The sent message stored should be still consisted of Latin characters while the received message is partially or fully in ASCII.

Comment 1

[Vicamo Yang [:vicamo][:vyang]]

Characters to be replaced:
Á(\u00C1) => A(\u0041),
á(\u00E1) => a(\u0061),
Í(\u00CD) => I(\u0049),
í(\u00ED) => i(\u0069),
Ó(\u00D3) => O(\u004F),
ó(\u00F3) => o(\u006F),
Ú(\u00DA) => U(\u0055),
ú(\u00FA) => u(\u0075),
ç(\u00E7) => Ç(\u00C7)

Comment 2

[Mounir Lamouri (:mounir)]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #1)
> Characters to be replaced:
[...]
> ç(\u00E7) => Ç(\u00C7)

You probably meant:
ç(\u00E7) => c(???)

Right?

Comment 3

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Mounir Lamouri (:mounir) from comment #2)
> (In reply to Vicamo Yang [:vicamo][:vyang] from comment #1)
> > Characters to be replaced:
> [...]
> > ç(\u00E7) => Ç(\u00C7)
> 
> You probably meant:
> ç(\u00E7) => c(???)
> 
> Right?

Actually, Ç(capital cedilla) is in GSM 7-bit default but ç(lower case cedilla) is not. Beatriz and I will confirm this again.

Comment 4

[Beatriz Rodríguez [:brg]]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #3)
> (In reply to Mounir Lamouri (:mounir) from comment #2)
> > (In reply to Vicamo Yang [:vicamo][:vyang] from comment #1)
> > > Characters to be replaced:
> > [...]
> > > ç(\u00E7) => Ç(\u00C7)
> > 
> > You probably meant:
> > ç(\u00E7) => c(???)
> > 
> > Right?
> 
> Actually, Ç(capital cedilla) is in GSM 7-bit default but ç(lower case
> cedilla) is not. Beatriz and I will confirm this again.

Checked today with the help of vicamo, the suitable change is:
> > > ç(\u00E7) => Ç(\u00C7)
ç(lower case cedilla) is not in GSM 7-bit default Code Scheme.

Comment 5

[Mounir Lamouri (:mounir)]

That sounds really weird. In French, I would prefer to read "garcon" instead of "garÇon"...

Comment 6

[Beatriz Rodríguez [:brg]]

Portuguese and catalan people are used to see Ç, they prefer it because it is not a gramatical mistake only a minor mistake.

Comment 7

[Beatriz Rodríguez [:brg]]

In Portuguese and Spanish languages, the default code scheme for SMS should be 7 bit. The reason is the number of characters included in a SMS is 160. The latin characters that are not covered with the 7 bit default scheme are replaced as described in comment #1.

Please mark this issue as blocking because it is important for platform customers.

Comment 8

[Vicamo Yang [:vicamo][:vyang]]

Do we must have a Settings UI for this? Or it's fine to be a product specific, read-only flag? Or a mutable preference setting like "dom.sms.strict7bitEncoding"?

Comment 9

[Andrew Overholt [:overholt]]

Is this going to make feature freeze?  Otherwise I think we'll have to - this.

Comment 10

[Andrew Overholt [:overholt]]

Are you going to take this, Vicamo?

Comment 11

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 1: WIP

Comment 12

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 2: test cases WIP

Comment 13

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 1

Comment 14

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 2: test cases

Comment 15

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part1: v2

let the map be string to string.

Comment 16

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 2: test cases : v2

add test cases to verify the map

Comment 17

[Marshall Culpepper [:marshall_law]]

Comment on attachment 666484 [details] [diff] [review]
Part1: v2

Review of attachment 666484 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, r=me with comments addressed.

::: b2g/app/b2g.js
@@ +385,5 @@
>  pref("dom.ipc.browser_frames.oop_by_default", false);
>  
>  // Temporary permission hack for WebSMS
>  pref("dom.sms.enabled", true);
> +//pref("dom.sms.strict7BitEncoding", true); // Disabled by default.

nit: should we uncomment this and set it to false?

::: dom/system/gonk/RadioInterfaceLayer.js
@@ +1952,5 @@
>      return options;
>    },
>  
>    getNumberOfMessagesForText: function getNumberOfMessagesForText(text) {
> +    let strict7BitEncoding = Services.prefs.getBoolPref("dom.sms.strict7BitEncoding");

IIRC this will throw an exception if the pref doesn't exist. We should handle that gracefully here (and below in sendSMS)

::: dom/system/gonk/ril_consts.js
@@ +1621,5 @@
> +  map["\u00FA"] = "\u0075"; // ú(\u00FA) => u(\u0075)
> +  map["\u00E7"] = "\u00C7"; // ç(\u00E7) => Ç(\u00C7)
> +
> +  return map;
> +})();

Any reason this can't be a literal object instead of creating and discarding a function scope?

Comment 18

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 1: v3

address review comment #17:
1. Uncomment pref() call and set it to false. This does make it more clear.
2. Wrap call to getBoolPref with try..catch
3. Use literal object instead.

Thank you :)

Comment 19

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 2: test cases : v3

re-formated in hg

Comment 20

[Vicamo Yang [:vicamo][:vyang]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6f1ea4ac1aa4
https://hg.mozilla.org/integration/mozilla-inbound/rev/97d7f66e2b30

Comment 21

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/6f1ea4ac1aa4
https://hg.mozilla.org/mozilla-central/rev/97d7f66e2b30