Closed Bug 887745 Opened 6 years ago Closed 6 years ago

WebSMS: Update UTF-8 character table for SMS

Categories

(Core :: DOM: Device Interfaces, defect)

ARM
Gonk (Firefox OS)
defect
Not set

Tracking

()

RESOLVED FIXED
mozilla25
blocking-b2g leo+
Tracking Status
firefox23 --- wontfix
firefox24 --- wontfix
firefox25 --- fixed
b2g18 --- fixed
b2g18-v1.0.0 --- wontfix
b2g18-v1.0.1 --- wontfix
b2g-v1.1hd --- fixed

People

(Reporter: isabelrios, Assigned: vicamo)

References

Details

(Whiteboard: [u=commsapps-user c=messaging p=0][fixed-in-birch])

Attachments

(6 files, 6 obsolete files)

Unagi device 06/27 build:
Gecko-914243b
Gaia-477e572

STR
1.Create a new SMS
2.Type: Ã b
3.Send the message to a valid number

EXPECTED
The message is sent.

ACTUAL
The message is not sent. The red icon is shown with the message: 'The message could not be sent. Try again?'

More info:
Sending only Ã, works fine. Sending: Ã f also works. Sending: Ã e also fails.
This is something extrange that we need to investigate to find out the root cause.
Nominating to leo+
blocking-b2g: --- → leo?
OS: Windows 7 → Gonk (Firefox OS)
Hardware: x86_64 → ARM
I just tried "Ã b" with the latest b2g18 Gecko/Gaia codes (10 mins ago). It works for me.
SMS master 163b3c144a3d457a5c7d6def8dd15fc1a139b903
on Thu Jun 27 19:03:37 2013 -0700

works for me (I tried à b, à e, and à f)

Isabel: What commit are you testing on? Can you replicate on a recent commit as well?
Status: NEW → UNCONFIRMED
Ever confirmed: false
Flags: needinfo?(isabelrios)
Hi Gene, Eric,
I tried again with today's v1-train build 06/28:
Gecko-db60841
Gaia-508cfc8
Movistar SIM Card

The same problem is happening.
In fact, I have seen another problem. When sending: "Ã i", a message without text is received (receivers were ffos, android, iphone, symbian)
Flags: needinfo?(isabelrios)
This seems like a regression and is something we'd hold the release for so leo+.

David S. wondered if this could be related to the SIM card.
blocking-b2g: leo? → leo+
Keywords: regression
Isabel, does the issue happen with commercial RIL as well or are you already using commercial RIL?
Flags: needinfo?(isabelrios)
Sorry I did not add that info. This is happening with latest unagi v1-train build:
Gecko-38e6b25
Gaia-a890df3

With reference RIL but only with movistar commercial SIM Cards.
Do not why the behaviour is different depending on the SIM Card.
Attached two logs files with commercital sim card (failing) and with corporative sim card (sms sent)
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(isabelrios)
Attached file ref RIL Commercial sim (obsolete) —
Attached file ref RIL corporative sim (obsolete) —
Whiteboard: [u=commsapps-user c=messaging p=0]
Component: Gaia::SMS → DOM: Device Interfaces
Product: Boot2Gecko → Core
Attached patch Gecko test cases (obsolete) — Splinter Review
Test cases pass successfully with trunk Gecko.  Will build Unagi and test again with Movistar SIM.
Assignee: nobody → vyang
Obsolete log for corporative SIM because it doesn't have RIL debug message enabled.  SMS sent, strict7BitEncoding disabled.
Attachment #770763 - Attachment is obsolete: true
Attached file Mozilla RIL with MoviStar SIM.log (obsolete) —
Obsolete log for commercial SIM because it doesn't have RIL debug message enabled.  SMS not sent, strict7BitEncoding enabled, force connect to Hinet, modem returns generic failure.
Attachment #770761 - Attachment is obsolete: true
Can't really reproduce the problem actually.  Phone call works, +34-620-987-139, don't know whether this is a commercial or corporative SIM.  All outgoing SMS attempts failed.
Hi Vicamo, then I guess that is a corporative SIM because I can still see the problem with today's unagi ref. ril and v1-train build.
I will enable the RIL debug message and attach new logs.
Attached file Moz ril commercial sim
Logs trying to send  Ãb: failed,  Ãi: sms received as empty,  Ãe: failed
Trying to get newer logs as not sure you can see all the info you need. Will attach a new file in a while
Please let me know if you cannot see the info you would need.
Thanks
Re-do with the same text body and receiver phone number as attachment 772604 [details].  All four messages sent successfully this time!
Attachment #772587 - Attachment is obsolete: true
PDU sent in attachment 772604 [details]: (Isabel, MoviStar SIM, commercial, failed)
0,0,0,80,25,0,0,0,79,0,0,0,2,0,0,0,255,255,255,255,28,0,0,0,50,0,49,0,48,0,48,0,48,0,57,0,56,0,49,0,53,0,54,0,56,0,55,0,57,0,57,0,50,0,51,0,70,0,52,0,48,0,48,0,48,0,48,0,48,0,50,0,50,0,65,0,51,0,49,0,0,0,0,0

PDU sent in attachment 772626 [details]: (Vicamo, MoviStar SIM, corporative<?>, success)
0,0,0,80,25,0,0,0,105,0,0,0,2,0,0,0,255,255,255,255,28,0,0,0,48,0,49,0,48,0,48,0,48,0,57,0,56,0,49,0,53,0,54,0,56,0,55,0,57,0,57,0,50,0,51,0,70,0,52,0,48,0,48,0,48,0,48,0,48,0,50,0,50,0,65,0,51,0,49,0,0,0,0,0

The only two differences are request token id(79 vs. 105) and SRR(Status Report Request, 1 vs. 0).  Will force SRR locally and try again.
Confirmed.  If I turn on SRR, I have the same problem with Isabel.  So, the next question would be, is SRR necessary for MoviStar SIM?  From bug 887815 comment 12 we know you want SRR default off.

I'll also mark this as a Gaia Settings issue because the mechanism to disable SRR in Gecko is already there.
Component: DOM: Device Interfaces → Gaia::Settings
Flags: needinfo?(brg)
Product: Core → Boot2Gecko
(In reply to Isabel Rios [:isabel_rios] from comment #17)
> Please let me know if you cannot see the info you would need.
> Thanks

Thanks for your precious log dump. :)
Assignee: vyang → nobody
SRR should be disable by default in Telefonica builds.

However, we will make more testing in our side to ensure sms is working fine and compare the behaviour with other reference devices using the same kind of characters.
Besides, we will check how '*b' is working and check if changing "Ã" to 'A' can mitigate the issue.

Thanks for the investigation and feedback!
Flags: needinfo?(brg)
I must add, the char 'Ã' is actually not important here.  That character is actually transformed into '*' in this case, and the DCS(data coding scheme) is actually set to 0x00(default alphabet), so you should be able to reproduce the same failure with "*b", "*e", "*f", "*i".

Per offline discuss, we may consider transform 'Ã' to 'A' as well, but note this may not solve the root problem and an user may want to send something like "a*b=c" (but I'm not sure whether this leads to the same problem).
"*b" fails on as predicted.  However, "a*b=c", "*bc", "a*b" all succeed.
Flags: in-moztrap?
Isabel and me had been making more test being focus on Movistar commercial SIM cards, regardles of SRR value, the strings:
  'Ãb' and '*b' are never sent.
  'Ãi' is received empty.

In order to mitigate this scenario, we would like to change the codification of the character 'Ã' to 'A' instead of '*'.  Shall we open a follow up bug for this?

Movistar Corporate SIM cards use a different SMSC so it can be the reason of the different behaviours.

After Vicamo investigation, this issue looks like a network issue and we only suggest the change in the codification as a way to mitigate the impact and because IMHO it makes sense to modify 'Ã' to 'A'.
(In reply to Beatriz Rodríguez [:brg] from comment #25)
> After Vicamo investigation, this issue looks like a network issue and we
> only suggest the change in the codification as a way to mitigate the impact
> and because IMHO it makes sense to modify 'Ã' to 'A'.

Please just submit a new list of such transformation you need if that's the final solution you want.  And, we're not going to disable SRR upon Movistar commercial SIM cards inserted?
(In reply to Vicamo Yang [:vicamo][:vyang] from comment #26)
> (In reply to Beatriz Rodríguez [:brg] from comment #25)
> > After Vicamo investigation, this issue looks like a network issue and we
> > only suggest the change in the codification as a way to mitigate the impact
> > and because IMHO it makes sense to modify 'Ã' to 'A'.
> 
> Please just submit a new list of such transformation you need if that's the
> final solution you want. 
ok, we will do this tomorrow.

> And, we're not going to disable SRR upon Movistar
> commercial SIM cards inserted?
The build generated for Telefonica countries should have SRR value OFF. Is this part of OEM task?

I want to hightlight that during our testing with commercial SIM cards, the results were the same regardless of SRR value we put in settings( Delivery Notification). 

We tested with a commercial SIM card as we know that corporate SIM cards are using a different smsc.
I think that the SIM card that you are using in Taipe is a corporate one but in roaming which may cause a different behaviour.
Assignee: nobody → vyang
Blocks: b2g-sms
Component: Gaia::Settings → DOM: Device Interfaces
Product: Boot2Gecko → Core
Summary: [SMS] SMS containing some UTF-8 characters cannot be sent → WebSMS: SMS containing some UTF-8 characters cannot be sent
Flags: in-moztrap? → in-moztrap-
(In reply to Beatriz Rodríguez [:brg] from comment #27)
> (In reply to Vicamo Yang [:vicamo][:vyang] from comment #26)
> > (In reply to Beatriz Rodríguez [:brg] from comment #25)
> > > After Vicamo investigation, this issue looks like a network issue and we
> > > only suggest the change in the codification as a way to mitigate the impact
> > > and because IMHO it makes sense to modify 'Ã' to 'A'.
> > 
> > Please just submit a new list of such transformation you need if that's the
> > final solution you want. 
> ok, we will do this tomorrow.
As agreed, find attached the new list including characters conversion: 
'Ã' to 'A'
'ã' to 'a'
The only change in this list of charaters is the addition of two new lines:
'Ã' to 'A'
'ã' to 'a'
(In reply to Beatriz Rodríguez [:brg] from comment #29)
> Created attachment 776220 [details]
> new table of characters conversion
> 
> The only change in this list of charaters is the addition of two new lines:
> 'Ã' to 'A'
> 'ã' to 'a'

Are you sure that you don't need ẼẽĨĩÑñÕõP̃p̃ŨũṼṽỸỹ as well?  Do you agree that I should just mark similar bug as WONTFIX in the future?

See http://en.wikipedia.org/wiki/%C3%83
Flags: needinfo?(brg)
(In reply to Vicamo Yang [:vicamo][:vyang] from comment #30)
> Are you sure that you don't need ẼẽĨĩÑñÕõP̃p̃ŨũṼṽỸỹ as well?  Do you agree
> that I should just mark similar bug as WONTFIX in the future?
> 
> See http://en.wikipedia.org/wiki/%C3%83

Ññ --> those characters are already covered in the document and manteined as Ññ (are being used in Spanish quite often)
The rest of the caracters that you suggested cannot be typed by our current version of the keyboard(in Spanish, Portuguese and English). Thats why I did not include them in the list. 
Do you prefer to include them? Maybe for future releases in the keyboard those characters could be added and we can stay in the safe side.
Flags: needinfo?(brg) → needinfo?(vyang)
Please email me if this can't be resolved before the end of the week, so that we can discuss.
Keywords: regression
Summary: WebSMS: SMS containing some UTF-8 characters cannot be sent → WebSMS: Update UTF-8 character table for SMS
Attached patch patch (obsolete) — Splinter Review
Add ẼẽĨĩÕõŨũṼṽỸỹ.  P̃ and p̃ are excluded from the patch because `unicode encodes p with tilde with a combining diacritical mark (U+0303 COMBINING TILDE), rather than a precomposed character.`[1]  I think we need some further discussion for these special cases.

Drop previous Gecko test cases because they don't really mean anything.

[1]: http://en.wikipedia.org/wiki/P%CC%83
Attachment #772008 - Attachment is obsolete: true
Attachment #777069 - Flags: review?(allstars.chh)
Flags: needinfo?(vyang)
Attachment #777069 - Flags: review?(allstars.chh) → review?(gene.lian)
Attached patch patch : v2 (obsolete) — Splinter Review
Missed 'Ã' and 'ã'.  Also removes additional commas at the end.
Attachment #777069 - Attachment is obsolete: true
Attachment #777069 - Flags: review?(gene.lian)
Attachment #777567 - Flags: review?(gene.lian)
Comment on attachment 777567 [details] [diff] [review]
patch : v2

Review of attachment 777567 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/mobilemessage/tests/marionette/test_strict_7bit_encoding.js
@@ +37,5 @@
>    "\u00df": "\u00df", // "ß" => "ß", already in default alphabet
>    "\u00e0": "\u00e0", // "à" => "à", already in default alphabet
>    "\u00e1": "\u0061", // "á" => "a"
>    "\u00e2": "\u0061", // "â" => "a"
> +  "\u00e3": "\u0061", // "ã" => "ä"

s/ä/a

::: dom/system/gonk/ril_consts.js
@@ +2150,5 @@
>  //"\u00df": "\u00df", // "ß" => "ß", already in default alphabet
>  //"\u00e0": "\u00e0", // "à" => "à", already in default alphabet
>    "\u00e1": "\u0061", // "á" => "a"
>    "\u00e2": "\u0061", // "â" => "a"
> +  "\u00e3": "\u0061", // "ã" => "ä"

Ditto.
Attachment #777567 - Flags: review?(gene.lian) → review+
Attached patch patch : v3Splinter Review
Address last comment.
Attachment #777567 - Attachment is obsolete: true
Attachment #777685 - Flags: review+
https://hg.mozilla.org/projects/birch/rev/eee360814562
Whiteboard: [u=commsapps-user c=messaging p=0] → [u=commsapps-user c=messaging p=0][fixed-in-birch]
https://hg.mozilla.org/mozilla-central/rev/eee360814562
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla25
https://hg.mozilla.org/releases/mozilla-b2g18/rev/bc4f307764bd

test_strict_7bit_encoding.js was never landed on b2g18, so pushed without it here. If you feel it's important to have, I would suggest opening a new bug for getting it landed.
You need to log in before you can comment on or make changes to this bug.