816082 - B2G SMS: UCS2 chars mapping to 7 bits default alphabet

Status: RESOLVED FIXED

(Core :: DOM: Device Interfaces, defect, P2)

Description

[Carlos Martínez Toral [:carlosmartinez]]

Attachment: Conversion UCS2 to 7 bits

Related to: https://bugzilla.mozilla.org/show_bug.cgi?id=815564

Attached you can find a document describing the desired mapping between UCS2 and 7 bits default alphabet regarding SMS sending.

Comment 1

[Carlos Martínez Toral [:carlosmartinez]]

Attachment: Conversion UCS2 to 7 bits

Comment 2

[Carlos Martínez Toral [:carlosmartinez]]

First attached document doesn´t have the correct format, please use the 2nd one.

Comment 3

[Lawrence Mandel [:lmandel] (use needinfo)]

Chris, is this a product requirement for v1?

Comment 4

[Beatriz Rodríguez [:brg]]

This is critical for product certification in latin countries.

Comment 5

[Vicamo Yang [:vicamo][:vyang]]

Character "₤"(U+20a4) is not part of default 7-bit alphabet.

Comment 6

[Carlos Martínez Toral [:carlosmartinez]]

Hi Vicamo,

According to the documentation we have, is part of the default alphabet, you can found it in: http://www.turkcell.com.tr/downloads/hakkimizda/pdf/23038-820.pdf or attached to the bug. The table we have used to create the list is in page 18.

If this document is not correct or you have a better one, please share it with us so we can correct the list.

Thank you.

Comment 7

[Carlos Martínez Toral [:carlosmartinez]]

Attachment: 3GPP Technical Spec

Comment 8

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Carlos Martínez Toral [:carlosmartinez] from comment #6)
> If this document is not correct or you have a better one, please share it
> with us so we can correct the list.

http://www.3gpp.org/ftp/Specs/html-info/23038.htm
v11.0.0 released on 2012-09-21, "₤"(U+20a4) is not included.

Comment 9

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 1: more entries in GSM_SMS_STRICT_7BIT_CHARMAP

Won't request review because we still have problem with the list.

Comment 10

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 2: add test scripts

Comment 11

[Vicamo Yang [:vicamo][:vyang]]

Depends on bug 759529 because test script needs it.

Comment 12

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Carlos Martínez Toral [:carlosmartinez] from comment #6)
> Hi Vicamo,
> 
> According to the documentation we have, is part of the default alphabet, you
> can found it in:
> http://www.turkcell.com.tr/downloads/hakkimizda/pdf/23038-820.pdf or
> attached to the bug. The table we have used to create the list is in page 18.

Actually, I can't find "₤"(U+20a4) in that whole document, either. Are you sure that you don't mean "￡"(U+ffe1)? The updated list contains "₤" => "₤", but "₤" is not part of default alphabet, so it's actually ineffective and will still be sent with UCS2 encoding.

Comment 13

[Vicamo Yang [:vicamo][:vyang]]

Comment on attachment 686649 [details] [diff] [review]
Part 1: more entries in GSM_SMS_STRICT_7BIT_CHARMAP

Review of attachment 686649 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/system/gonk/ril_consts.js
@@ +1811,5 @@
> +  "\u00cc": "\u0049", // "Ì" => "I"
> +  "\u00cd": "\u0049", // "Í" => "I"
> +  "\u00ce": "\u0049", // "Î" => "I"
> +  "\u00cf": "\u0049", // "Ï" => "I"
> +  "\u00d1": "\u00d1", // "Ñ" => "Ñ"

already in default alphabet

@@ +1889,5 @@
> +  "\u017c": "\u007a", // "ż" => "z"
> +  "\u017d": "\u005a", // "Ž" => "Z"
> +  "\u017e": "\u007a", // "ž" => "z"
> +  "\u025b": "\u0045", // "ɛ" => "E"
> +  "\u0398": "\u0398", // "Θ" => "Θ"

ditto

Comment 14

[Beatriz Rodríguez [:brg]]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #12)
> (In reply to Carlos Martínez Toral [:carlosmartinez] from comment #6)
> 
> Actually, I can't find "₤"(U+20a4) in that whole document, either. Are you
> sure that you don't mean "￡"(U+ffe1)? The updated list contains "₤" => "₤",
> but "₤" is not part of default alphabet, so it's actually ineffective and
> will still be sent with UCS2 encoding.

If 7bit code scheme is set, the sms should be codified using it even when the user introduce a character that it is not covered in the conversion table. Please use a wildcard character when it happens to avoid the codification in any other system.

Comment 15

[Carlos Martínez Toral [:carlosmartinez]]

Hi Vicamo, 

You´re right, we have confused "₤"(U+20a4) and "￡"(U+ffe1), please use the correct one "￡"(U+ffe1), and sorry for the typo.

Thank you.

Carlos

Comment 16

[Chris Lee [:clee]]

Looks like this is required for certification in market.  Marking blocking.

Comment 17

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Carlos Martínez Toral [:carlosmartinez] from comment #15)
> You´re right, we have confused "₤"(U+20a4) and "￡"(U+ffe1), please use the
> correct one "￡"(U+ffe1), and sorry for the typo.

And there is also problem with "€"(U+20ac) => "E". "€"(U+20ac) is already in default 7-Bit shift table, which means you don't have to map it into "E" and it will still be encoded in 7-Bit alphabet. Is this really what you want?

(In reply to Beatriz Rodríguez [:brg] from comment #14)
> If 7bit code scheme is set, the sms should be codified using it even when
> the user introduce a character that it is not covered in the conversion
> table. Please use a wildcard character when it happens to avoid the
> codification in any other system.

You really have to make up your mind. What's the FINAL version of this feature? That's exactly why I don't want to enable it in Gecko by default. And maybe we shouldn't have implemented it at first place at all.

Comment 18

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #17)
> blocking-basecamp: + → ?

Oops, sorry! I don't mean it.

Comment 19

[Peter Dolanjski [:pdol]]

No problem Vicamo :)
Marked blocking again.

Comment 20

[Carlos Martínez Toral [:carlosmartinez]]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #17)
> (In reply to Carlos Martínez Toral [:carlosmartinez] from comment #15)
> > You´re right, we have confused "₤"(U+20a4) and "￡"(U+ffe1), please use the
> > correct one "￡"(U+ffe1), and sorry for the typo.
> 
> And there is also problem with "€"(U+20ac) => "E". "€"(U+20ac) is already in
> default 7-Bit shift table, which means you don't have to map it into "E" and
> it will still be encoded in 7-Bit alphabet. Is this really what you want?
> 
> (In reply to Beatriz Rodríguez [:brg] from comment #14)
> > If 7bit code scheme is set, the sms should be codified using it even when
> > the user introduce a character that it is not covered in the conversion
> > table. Please use a wildcard character when it happens to avoid the
> > codification in any other system.
> 
> You really have to make up your mind. What's the FINAL version of this
> feature? That's exactly why I don't want to enable it in Gecko by default.
> And maybe we shouldn't have implemented it at first place at all.

You´re right again, "€"(U+20ac) is included in the GSM 7 bit default alphabet extension table but I didn´t know that this is included by default, please use the € instead of E in this case, and sorry again.

Comment 21

[Beatriz Rodríguez [:brg]]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #17)
> 
> (In reply to Beatriz Rodríguez [:brg] from comment #14)
> > If 7bit code scheme is set, the sms should be codified using it even when
> > the user introduce a character that it is not covered in the conversion
> > table. Please use a wildcard character when it happens to avoid the
> > codification in any other system.
> 
> You really have to make up your mind. What's the FINAL version of this
> feature? That's exactly why I don't want to enable it in Gecko by default.
> And maybe we shouldn't have implemented it at first place at all.
We need to support 7bit code scheme when sending sms with latin languages.If code scheme is set to 7bit, the platform should use it. Platform cannot change it automatically when a character not listed is typed by the user, that is why I suggest to put a wildcard character.

Comment 22

[Alex Keybl [:akeybl]]

Mass Modify: All un-milestoned, unresolved blocking-basecamp+ bugs are being moved into the C3 milestone. Note that the target milestone does not mean that these bugs can't be resolved prior to 12/10, rather C2 bugs should be prioritized ahead of C3 bugs.

Comment 23

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 1: more entries in GSM_SMS_STRICT_7BIT_CHARMAP : v2

comment out another three already in default alphabet entries.

Comment 24

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 2/4: refactor fragmentText, pass fragemented text to ril_worker instead.

options passed to ril_worker should contain full per segment info even it contains only one segment. Overridden text strings are included now, so duplicated logic in ril_worker can be removed.

Comment 25

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 3/4: replace non-7bit-alphabet chars with '*' when strict7BitEncoding is enabled

Comment 26

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 4/4: add test scripts : v2

Add both xpcshell & marionette test cases. Note that marionette test cases depend on https://github.com/mozilla-b2g/platform_external_qemu/pull/10 .

Comment 27

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Comment on attachment 691263 [details] [diff] [review]
Part 3/4: replace non-7bit-alphabet chars with '*' when strict7BitEncoding is enabled

Review of attachment 691263 [details] [diff] [review]:
-----------------------------------------------------------------

Please see my comment below, thank you :)

::: dom/system/gonk/RadioInterfaceLayer.js
@@ +2042,5 @@
> +        c = '*';
> +        if (langTable.indexOf(c) >= 0) {
> +          length++;
> +        } else {
> +          length += 2;

Would you please comment why 'length += 2' here? Thanks!

Comment 28

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Comment on attachment 691296 [details] [diff] [review]
Part 4/4: add test scripts : v2

Review of attachment 691296 [details] [diff] [review]:
-----------------------------------------------------------------

Nice!

::: dom/sms/tests/marionette/test_strict_7bit_encoding.js
@@ +178,5 @@
> +  }
> +
> +  // When strict7BitEncoding is enabled, we should replace characters that
> +  // can't be encoded with GSM 7-Bit alphabets with '*'.
> +  add_case("\u65b0\u5e74\u5feb\u6a02", "****"); // "Happy New Year" in Chinese.

:D

Comment 29

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 1/4: more entries in GSM_SMS_STRICT_7BIT_CHARMAP : v3

Add r=hsinyi

Comment 30

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 2/4: refactor fragmentText, pass fragemented text to ril_worker instead : v2

Add r=hsinyi

Comment 31

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 3/4: replace non-7bit-alphabet chars with '*' when strict7BitEncoding is enabled : v2

1) Address review comment #27.

In previous revision, I used the fact that '*' is either in locking shift table or in single shift table. It's true for current released specifications but there is no guarantee in the future.

In this revision, I increase 'length' by 2 only when we can find '*' in single shift table and return -1 otherwise.

2) Add r=hsinyi

Comment 32

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 4/4: add test scripts : v3

Add r=hsinyi

Comment 33

[Vicamo Yang [:vicamo][:vyang]]

Attachment: Part 4/4: correct xpcshell test cases : v4

In stressed marionette runs, I find the marionette test script can fail following ones. It inserts a lot (200+) unfinished database transactions, quits successfully, and leaves unfinished tasks counted on the next test script. When the next test script happens to have a short MARIONETTE_TIMEOUT value set, it will probably starve.

This issue doesn't affect the integrity and correctness of the SMS databases, but is an implementation weakness for timed automation tests. I remove the marionette part in this revision.

Comment 34

[Vicamo Yang [:vicamo][:vyang]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/609a53b92cee
https://hg.mozilla.org/integration/mozilla-inbound/rev/07baed38f016
https://hg.mozilla.org/integration/mozilla-inbound/rev/2a616878a5b1
https://hg.mozilla.org/integration/mozilla-inbound/rev/f0d7097db421

Comment 35

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #33)
> This issue doesn't affect the integrity and correctness of the SMS
> databases, but is an implementation weakness for timed automation tests. I
> remove the marionette part in this revision.

Bug 822207 filed to address this problem.

Comment 36

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/609a53b92cee
https://hg.mozilla.org/mozilla-central/rev/07baed38f016
https://hg.mozilla.org/mozilla-central/rev/2a616878a5b1
https://hg.mozilla.org/mozilla-central/rev/f0d7097db421

Comment 37

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/176672e2e40b
https://hg.mozilla.org/releases/mozilla-aurora/rev/88c7005b6a29
https://hg.mozilla.org/releases/mozilla-aurora/rev/7cd948d04ace
https://hg.mozilla.org/releases/mozilla-aurora/rev/0adc3bb27f5b

https://hg.mozilla.org/releases/mozilla-b2g18/rev/ef9e5c5703ce
https://hg.mozilla.org/releases/mozilla-b2g18/rev/6d856daeb583
https://hg.mozilla.org/releases/mozilla-b2g18/rev/40f01f3d6860
https://hg.mozilla.org/releases/mozilla-b2g18/rev/3128ce43bf1a

Comment 38

[isabelrios]

With today build on Unagi there is a problem with the € symbol:
Gaia 73270c3
Geckonb8d7310

Using a VIVO SIM Card from Brazil, that characther '€' is sent like '?e'
According to the pdf attached, it should be sent as 'E'

Comment 39

[Carlos Martínez Toral [:carlosmartinez]]

Sorry for the confusion, € should be sent as €, as it was included in the default extension table, as per comment 17.

Comment 40

[Dietrich Ayala (:dietrich)]

Vicamo, should the problem in comment 39 be handled in a followup bug or is it a major issue with the patches that landed here?

Comment 41

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Dietrich Ayala (:dietrich) from comment #40)
> Vicamo, should the problem in comment 39 be handled in a
> followup bug or is it a major issue with the patches that
> landed here?

It's not even a problem. Please update that PDF.

Comment 42

[Vicamo Yang [:vicamo][:vyang]]

Bug 824612 is the marionette test cases for this issue. As you can see in https://tbpl.mozilla.org/?tree=Try&rev=64003f2f846f , all test cases, inclusive of '\u20ac', passed with all greens. However, this is based on emulator. I'll test it on Unagi next morning.

Comment 43

[isabelrios]

With Unagi and latest build 01/014/2012 the '€' symbol sent from a VIVO SIM Card is received as '?e'. Please correct us if we are not right but our understanding is that is should be received as '€'

Comment 44

[Vicamo Yang [:vicamo][:vyang]]

(In reply to Isabel Rios [:isabel_rios] from comment #43)
> With Unagi and latest build 01/014/2012 the '€' symbol sent
> from a VIVO SIM Card is received as '?e'. Please correct us if
> we are not right but our understanding is that is should be
> received as '€'

Could you help me to capture some debug log messages? I just solved bug 828633 today. The test cases run correctly with expected results. And last time you reported the same thing, I was then told it's actually a false alarm.

Anyway, detailed steps for enable debug messages are:
1) Connect usb cable
2) Execute `adb pull /system/b2g/defaults/pref/user.js .`
3) Edit pulled file user.js. Change value of "ril.debugging.enabled" to "true"
4) Execute `adb push user.js /system/b2g/defaults/pref`
5) reboot Unagi
6) Execute `adb logcat | tee filename.log` and send a SMS message that contains '€' in GUI. Press Ctrl-C.
7) Attach captured log.

Thanks.

Comment 45

[isabelrios]

Attachment: log file sending sms with € from VIVO SIM Card

Comment 46

[isabelrios]

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #44)
> (In reply to Isabel Rios [:isabel_rios] from comment #43)
> > With Unagi and latest build 01/014/2012 the '€' symbol sent
> > from a VIVO SIM Card is received as '?e'. Please correct us if
> > we are not right but our understanding is that is should be
> > received as '€'
> 
> Could you help me to capture some debug log messages? I just solved bug
> 828633 today. The test cases run correctly with expected results. And last
> time you reported the same thing, I was then told it's actually a false
> alarm.
> 
> Anyway, detailed steps for enable debug messages are:
> 1) Connect usb cable
> 2) Execute `adb pull /system/b2g/defaults/pref/user.js .`
> 3) Edit pulled file user.js. Change value of "ril.debugging.enabled" to
> "true"
> 4) Execute `adb push user.js /system/b2g/defaults/pref`
> 5) reboot Unagi
> 6) Execute `adb logcat | tee filename.log` and send a SMS message that
> contains '€' in GUI. Press Ctrl-C.
> 7) Attach captured log.
> 
> Thanks.

Hi Vivamo, 
Attached you can find the log file you asked for. Hope it helps. In case it is not correct or whatever, please let me know so I will attach another one.
Thanks

Comment 47

[Beatriz Rodríguez [:brg]]

There is a new version of the table of characters to convert in: 
https://bug887745.bugzilla.mozilla.org/attachment.cgi?id=776220

Comment 48

[ekometri.com]

This is critical for product certification in latin countries, i'm getting error from this page http://www.ucakbileti.com.tr but there is no problem when i'm using chrome browser