Closed Bug 914454 Opened 7 years ago Closed 6 years ago

Ensure SMS messages in only uppercase Greek letters are sent with 7bit encoding on the carrier partner network

Categories

(Firefox OS Graveyard :: Gaia::SMS, defect, P1)

defect

Tracking

(blocking-b2g:leo+, b2g18 fixed, b2g-v1.1hd fixed, b2g-v1.2 fixed)

RESOLVED FIXED
blocking-b2g leo+
Tracking Status
b2g18 --- fixed
b2g-v1.1hd --- fixed
b2g-v1.2 --- fixed

People

(Reporter: sync-1, Assigned: rudyl)

References

Details

Attachments

(8 files, 1 obsolete file)

Mozilla build ID:20130902041201
 
 DEFECT DESCRIPTION:
  The device supports 70 Greek characters per SMS
 
  REPRODUCING PROCEDURES:
  The device supports 70 Greek characters per SMS--KO
  Customer Impact Statement: 
  Customer will be charged for two SMS in case he writes 160 Greek characters
 
  EXPECTED BEHAVIOUR:
  KO:The device must support 160 Greek characters per SMS.
     This is a minimum requirement for GR
 
  ASSOCIATE SPECIFICATION:
 
  TEST PLAN REFERENCE:
 
  TOOLS AND PLATFORMS USED:
 
  USER IMPACT:serious
 
  REPRODUCING RATE:
 
  For FT PR, Please list reference mobile's behavior:
blocking-b2g: --- → leo?
Clone from brother
Attached file PR521705.logcat
Summary: [Buri][T-mobile 51609][SMS]The device supports 70 Greek characters per SMS → [Buri][Shira][T-mobile 51609][SMS]The device supports 70 Greek characters per SMS
Taken from Wikipedia http://en.wikipedia.org/wiki/GSM_03.38#National_language_shift_tables):

There is still no defined national language shift table for French, Greek, Russian, Bulgarian, Arabic, Hebrew and most Central European languages that need a better coverage than the default 7-bit standard character set and its default 7-bit extension character set: if ever any character is composed that cannot be represented in those default GSM 7-bit sets, the message will be automatically reencoded using UCS-2, with the effect of dividing by more than two the maximum length in characters of messages that can be sent at the price of a single SMS (when a message is split in multiple parts, a few other octets are needed in the User Data Header to indicate the sequence number of each part).

Therefore I don't know how we're supposed to support 7-bit encoding for greek.

Or are we talking about capital greek letters that are contained in the default 7-bit character set ?
(In reply to Julien Wajsberg [:julienw] from comment #3)
> Therefore I don't know how we're supposed to support 7-bit encoding for
> greek.

We have only national languages listed on 23.038 and they're not enabled by default.

https://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/ril_consts.js#1537
https://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/ril_consts.js#1834
https://bugzilla.mozilla.org/show_bug.cgi?id=733331
Summary: [Buri][Shira][T-mobile 51609][SMS]The device supports 70 Greek characters per SMS → [Buri][Shira-51609][SMS]The device supports 70 Greek characters per SMS
perhaps we could have l10n team comment on Greek? Thanks
Flags: needinfo?(l10n)
I guess Julien said it all. Not knowing anything about SMS per se, but his comment makes sense to me.
Flags: needinfo?(l10n)
is anyone from the Greek community able to give us more clarify on this? Thanks
Flags: needinfo?(l10n)
I don't know what question you're asking. Anyhow, maybe Pierros or Fredy can comment on how many greek letters they're used to get into one SMS.
Flags: needinfo?(l10n)
Minus based on Julien's comment 3.
blocking-b2g: leo? → -
(In reply to Julien Wajsberg [:julienw] from comment #3)

At the same wikipedia article:
"This 7-bit encoding allows the transport of texts encoded in the Basic Latin subset of ASCII, as well as some characters of the ISO Latin 1 character set. It also allows the encoding of texts written in the Greek script, but only capitals; for such use in Greek, the Latin capital letters that look like the Greek letters are reused with the same code, so that the above character set is complete only for modern monotonic Greek restricted to capital letters. A complete support for the Greek alphabet (including small letters) requires a national version of the shifted 7-bit table (using the ESC code for each national character encoded in this shifted table), or an unspecified proprietary 8-bit encoding, or the use of the UCS2 encoding (see below)."

I have checked two mobile devices and on both of them when you choose to write an sms in Greek then they support only greek capital letters (I guess in order to get 160 characters).
Thanks a lot Fredy. Additional question: were your devices "normal" phones, feature phones, or smart phones (and in that case, Apple/Android/Windows ?).
(In reply to Julien Wajsberg [:julienw] from comment #11)
> Thanks a lot Fredy. Additional question: were your devices "normal" phones,
> feature phones, or smart phones (and in that case, Apple/Android/Windows ?).

oops I forgot to mention that... the devices were one generation before feature phones.

I've just tested also to send sms on android phones (one jellybean and one ics) and on both of them this bug exists, sms are limited to ~70 chars.

I'll try to check also on a feature phone and I'll comment here.
We just tested on iPhone and Android 4.3, with both small letters and with all-caps letters. In both system we're limited to 70 characters per SMS.

So we're the same.
However, maybe, if we can make the capital greek letters to _not_ use the unicode encoding (because it should be in the normal 7-bit encoding), then we could have a competitive advantage against the competition. 

That would probably mean to have a special keyboard that would output the latin capital letters instead of the "real" greek letters. I think we could use a third-party keyboard to do this.

But first I'd like to know how it currently works. Rudy, I found the "layout.js" file where we define the lower case greek letters, but I don't know where we "calculate" the capital letters. Could you tell me where it's done in the code ?
Flags: needinfo?(rlu)
Same bug for android:
https://code.google.com/p/android/issues/detail?id=5544

a custom greek keyboard on google play comes with a special mode for sms (check the screenshots)
https://play.google.com/store/apps/details?id=com.sql.greekkeyboard
(In reply to Julien Wajsberg [:julienw] from comment #14)
> However, maybe, if we can make the capital greek letters to _not_ use the
> unicode encoding (because it should be in the normal 7-bit encoding), then
> we could have a competitive advantage against the competition. 
> 
> That would probably mean to have a special keyboard that would output the
> latin capital letters instead of the "real" greek letters. I think we could
> use a third-party keyboard to do this.
> 
> But first I'd like to know how it currently works. Rudy, I found the
> "layout.js" file where we define the lower case greek letters, but I don't
> know where we "calculate" the capital letters. Could you tell me where it's
> done in the code ?

We would use toUpperCase() to get it from the lower case char unless it is specifically defined in layout.js,

toUpperCase(),
https://github.com/mozilla-b2g/gaia/blob/master/apps/keyboard/js/keyboard.js#L946

Upper case defined in layout.js,
https://github.com/mozilla-b2g/gaia/blob/master/apps/keyboard/js/layout.js#L867

--
I am not quite sure of the exact issue here, but I don't remember we did char conversion (for SMS char limit or other requirement) in keyboard app.
Flags: needinfo?(rlu)
Thanks Rudy !

Now I don't know if we should either:
1. modify the existing greek alphabet to use the capital latin letter instead of the capital greek letters where appropriate (eg: τ => latin T). I don't know how screen readers would react to this.
2. add a new layout that would be "capital greek letters for SMS"
3. do a 3rd party keyboard and put it on the marketplace

Advices ?
(In reply to Julien Wajsberg [:julienw] from comment #17)
> Thanks Rudy !
> 
> Now I don't know if we should either:
> 1. modify the existing greek alphabet to use the capital latin letter
> instead of the capital greek letters where appropriate (eg: τ => latin T). I
> don't know how screen readers would react to this.

This would be wrong, except screen readers, this may cause problems with spellchecking and sites that use greek characters, e.g. in passwords.

> 2. add a new layout that would be "capital greek letters for SMS"
> 3. do a 3rd party keyboard and put it on the marketplace
> 
> Advices ?

These are better solutions.

If we want to have a built-in gaia solution then we should follow the first one and I can help also by creating this layout.

If we want an approach like in android then we should follow the 2nd one and build a 3rd party keyboard. I can also help in this. ;)
Thinking more about this, I think the 3rd party keyboard is the best way: we (or partners) can decide to either install it built-in or leave it in the marketplace.

Let's do that !
Component: Gaia::SMS → Gaia::Keyboard
Summary: [Buri][Shira-51609][SMS]The device supports 70 Greek characters per SMS → Create a 3rd party keyboard for greek input for SMS
Whiteboard: [good-first-bug]
Attachment #801983 - Attachment mime type: application/octet-stream → application/x-rar-compressed
(In reply to sync-1 from comment #2)
> Created attachment 801983 [details]
> PR521705.logcat

Can't open this archive at all.
Flags: needinfo?(sync-1)
Attached file logcat_20130909_182444
Flags: needinfo?(sync-1)
(In reply to Vicamo Yang [:vicamo][:vyang] from comment #20)
> (In reply to sync-1 from comment #2)
> > Created attachment 801983 [details]
> > PR521705.logcat
> 
> Can't open this archive at all.

Dear  Vicamo Yang,

I have attached the log again, can it be opened now?
In the RIL layer, we have a table in charge of enabling Latin characters replacement with corresponding ones in GSM SMS 7-bit default alphabet [1]. It's possible to solve this issue within the Gecko.

[1] https://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/ril_consts.js#2136
Gene> But I think it's not enabled by default. Could we enable this for only greek, but not for other languages ? I don't think so... Or could it be controlled by a user-exposed setting ?

Could still be a good idea to add these characters in that table though.
I'd suggest adding an extra table to work for these special characters to do the conversion in any way, even if dom.sms.strict7BitEncoding is not enabled.

Needinfo to Vicamo to have more discussions. He is in charge of this encoding stuff. Vicamo, what do you think?
Flags: needinfo?(vyang)
Gene, would this be RIL-specific ? I mean, would this be only in MozRIL, or is it more generic and we would get this even in Commercial RILs ?
Hi all, it seems like Gaia flips 'ril.sms.strict7BitEncoding.enabled' pref by looking at MNC and MCC. We should just ensure we did flip this on when we detected the specific MNC/MCC in shared/resources/apn/operator_variant.(xml|js)

That said, we do need to verify that CommRIL do honor this setting like MozRIL do.

Joe, please loop partner to find out what's the MNC/MCC that we need to config.
Component: Gaia::Keyboard → Gaia
Flags: needinfo?(jcheng)
Summary: Create a 3rd party keyboard for greek input for SMS → Ensure SMS messages in only uppercase Greek letters are sent with 7bit encoding on the carrier partner network
(In reply to Julien Wajsberg [:julienw] from comment #19)
> Thinking more about this, I think the 3rd party keyboard is the best way: we
> (or partners) can decide to either install it built-in or leave it in the
> marketplace.
> 
> Let's do that !

You could actually tap the arrow key twice to get a uppercase-locked keyboard. :-/
Also, 3rd-party app development shouldn't be tracked on Bugzilla here...
Hi Michael,

Some questions for you. Could you (or your team members) please respond when you have a chance? Thanks!

1. Does the Commercial RIL do something like Moz RIL [1] to automatically convert some specific characters to 7-bit encoding before sending SMS?

2. If yes, does the Commecrial RIL also hook up to the preference 'dom.sms.strict7BitEncoding' to enable/disable the conversion?

[1] https://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/ril_consts.js#2136
Flags: needinfo?(mvines)
(In reply to Gene Lian [:gene] (please use needinfo?) from comment #25)
> I'd suggest adding an extra table to work for these special characters to do
> the conversion in any way, even if dom.sms.strict7BitEncoding is not enabled.
> 
> Needinfo to Vicamo to have more discussions. He is in charge of this
> encoding stuff. Vicamo, what do you think?

Were it possible, I would add Greek to Gecko supported national languages.  But the fact we have now, there is no such tables in 23.038, so we have neither the index value of that Greek table nor the characters, ordering of that table.  We at least need technical sheets to begin.  Besides, maybe somebody can do me a favor to figure it out that whether such need is based on legal rules or a common convention of the sms services deployed.
Flags: needinfo?(vyang)
(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TPE) from comment #28)


> You could actually tap the arrow key twice to get a uppercase-locked
> keyboard. :-/
> 

That's not the same.
Some uppercase greek letters _look like_ the latin ones, but are not :) And therefore they trigger the unicode encoding. The uppercase greek letters that _don't_ look like the latin ones are in the 7-bit encoding though, so currently, if your SMS message has only these letters, you can write 160-characters SMS.

Changing the table to add those capital letters would make it work with an uppercase-locked keyboard, but I don't know how users will discover this.

(In reply to Vicamo Yang [:vicamo][:vyang] from comment #30)
> Besides, maybe
> somebody can do me a favor to figure it out that whether such need is based
> on legal rules or a common convention of the sms services deployed.

see comment 10, comment 12 and comment 15.

No legal rules afaik, and the conversion is done on feature phones but not on most current smartphones, except the user uses a third-party keyboard.
(In reply to Julien Wajsberg [:julienw] from comment #31)
> No legal rules afaik, and the conversion is done on feature phones but not
> on most current smartphones, except the user uses a third-party keyboard.

Then I would probably mark this bug as INVALID or WONTFIX if somehow it's moved to B2G::RIL.
So Vicamo you confirm that you think it should be done with an 3rd party app then, right ?
(In reply to Julien Wajsberg [:julienw] from comment #33)
> So Vicamo you confirm that you think it should be done with an 3rd party app
> then, right ?

No.  I would like to have all related tech documents to evaluate the scope of this request.  I still don't know how many characters are include in Greek alphabets and what are these characters exactly?  Without this information, I don't know whether a 3rd party app or the 'dom.sms.strict7BitEncoding' feature may really help the encoding process.  You know for some people on the other side of Earth, all Latin characters are Greek, and I'm just one of them.  Please feed me with some technical documents first.  I'm adopted to hex/binary codes most.  01101010111011 ...

Even it's proven that we need some Gecko help here, unless our PM confirms that this feature is a must, I don't really want to build anything not based on open standard to Gecko.  Gecko is open source, everyone is free to clone and modify it to whatever fits him best.  Not every feature should go into Gecko.  Even it makes its way into Gecko, such features will be disabled by default.

(In reply to xiupinglong from comment #22)
> I have attached the log again, can it be opened now?

Yes, it's working, thank you.  But that's captured with a commercial ril and that's beyond our reach.  Could you provide some documents about the Greek national language?  I'm expecting something like single/locking shift tables defined in 3GPP TS 23.038.  Thank you again.
s/adopted/adapted/
Let me try to explain again what I understood and somewhat verified.

* Greek does not have a standard shift table
* you can see the greek alphabet at [1]
* however, as you see in [2], there are indeed "real" capital greek characters in the 7bit SMS encoding table: 0x10, 0x12 to 0x1A
* also, you can see in [1]  that the other capital greek characters look like latin capital letters  (eg: A, B, E, etc).
* However, the capital greek letters that look like latin capital letters have their unicode characters (eg U+0391 for the greek capital alpha "A"). This is what we get when we call toUpperCase on the lower case letter.

Therefore, the 7-bit SMS encoding table contains all real greek capital letters that don't look like latin capital letters, and of course the latin capital letters. As far as I understand, feature phones that permits 160-characters SMS for greek swap greek capital letters that look like latin capital letters with the latin capital letters.

However, this is not really greek anymore, so we can't generally do that in the greek uppercase letters keyboard. I also think we can't do this automatically in the 2013 unicode world, and that's why I advocated for another keyboard layout, either built-in or 3rd-party.

I personally verified that a SMS message containing only 'Θ' (Capital Theta) characters could hold 160 characters.

Is the issue clearer for you Vicamo ?

[1] http://en.wikipedia.org/wiki/Greek_alphabet
[2] http://en.wikipedia.org/wiki/GSM_03.38#GSM_7_bit_default_alphabet_and_extension_table_of_3GPP_TS_23.038_.2F_GSM_03.38
(In reply to Gene Lian [:gene] (national holidays until 9/23) from comment #29)
> ... does the Commecrial RIL also hook up to the preference
> 'dom.sms.strict7BitEncoding' to enable/disable the conversion?

Yes, we honour this preference.
Flags: needinfo?(mvines)
(In reply to Julien Wajsberg [:julienw] from comment #36)
> Therefore, the 7-bit SMS encoding table contains all real greek capital
> letters that don't look like latin capital letters, and of course the latin
> capital letters. As far as I understand, feature phones that permits
> 160-characters SMS for greek swap greek capital letters that look like latin
> capital letters with the latin capital letters.
> 
> However, this is not really greek anymore, so we can't generally do that in
> the greek uppercase letters keyboard. I also think we can't do this
> automatically in the 2013 unicode world, and that's why I advocated for
> another keyboard layout, either built-in or 3rd-party.

if I read this correctly, it seems like if someone can provide a requirement (mapping of the encoding table) of a new layout with capital latin letters that look like captital greek letters, then this new simulated capital Greek keyboard should work for the Greek market to send 160 characters of text messages.

Thanks
Flags: needinfo?(jcheng)
Joe, this is it, the only question is: how should we do it. The mapping itself is quite easy to do, I could do a patch once we decide how to do this.
Hi all, we discussed the issue with the DT local team in Greece. Result is that the usual workaround for this problem is that devices use the standard 7-bit SMS alphabet and users make due with the letters available in this alphabet. This leads to the device only supporting upper case greek letters, but this is the preferred solution over using UCS-2, which does support both upper and lower case letters, but allows only 80 characters per SMS.

Keyboard needs to be adapted to only allow entry of characters from the default alphabet. Device should notify the user that a switch to UCS-2 is required if other characters are entered.
As another data point, a Galaxy S II with the default Samsung keyboard and an iPhone (both on the latest versions) are displaying the keyboard in the SMS app with caps lock enabled, and seem to be converting the messages that contain greek characters to the 7-bit alphabet that was mentioned above (stripping accents, etc.).
FxOS Deutsche Telekom> I think you want to do this only in the SMS app, right ?

I'm afraid we can't show different keyboards layout depending on the app, at least right now. Moreover, as explained in comment 17 and comment 18, this would make problems with screen readers and autocorrection.

Would it be ok to let users trigger the caps lock themselves, but then do a conversion inside the SMS subsystem, as proposed by Gene in comment 23 and comment 25? (provided that Vicamo accepts this of course)
(see previous comment)
Flags: needinfo?(steffen.kruessel)
(In reply to Julien Wajsberg [:julienw] from comment #36)
> Let me try to explain again what I understood and somewhat verified.
> 
> * Greek does not have a standard shift table
> * you can see the greek alphabet at [1]
> * however, as you see in [2], there are indeed "real" capital greek
> characters in the 7bit SMS encoding table: 0x10, 0x12 to 0x1A
> * also, you can see in [1]  that the other capital greek characters look
> like latin capital letters  (eg: A, B, E, etc).
> * However, the capital greek letters that look like latin capital letters
> have their unicode characters (eg U+0391 for the greek capital alpha "A").
> This is what we get when we call toUpperCase on the lower case letter.
> 
> Therefore, the 7-bit SMS encoding table contains all real greek capital
> letters that don't look like latin capital letters, and of course the latin
> capital letters. As far as I understand, feature phones that permits
> 160-characters SMS for greek swap greek capital letters that look like latin
> capital letters with the latin capital letters.
> 
> However, this is not really greek anymore, so we can't generally do that in
> the greek uppercase letters keyboard. I also think we can't do this
> automatically in the 2013 unicode world, and that's why I advocated for
> another keyboard layout, either built-in or 3rd-party.
> 
> I personally verified that a SMS message containing only 'Θ' (Capital Theta)
> characters could hold 160 characters.
> 
> Is the issue clearer for you Vicamo ?
> 
> [1] http://en.wikipedia.org/wiki/Greek_alphabet
> [2]
> http://en.wikipedia.org/wiki/GSM_03.
> 38#GSM_7_bit_default_alphabet_and_extension_table_of_3GPP_TS_23.038_.
> 2F_GSM_03.38

Julien, thanks for the sum-up! Now I recognized this is some how a bigger issue than it look like!

IMHO since Vicamo don't think it make sense to put the "hack" in RIL, the only place left to "translate" the string written in capital Greek letters to the SMS Greek script is the Message App. Does that sounds like valid solution (i.e. preprocess the sms msg string before sending it to SMS API)?
Component: Gaia → Gaia::SMS
Hey Tim,

I think Vicamo has not clearly said he doesn't want it in the RIL ;)

I really think it should not be in the Messages app, this sounds horrible. If this doesn't go in the RIL (which Vicamo has not rejected yet), I'd really prefer a special keyboard for this use case, as I proposed before already. It should be very easy to add a new layout in layout.js.
(In reply to Michael Vines [:m1] [:evilmachines] from comment #37)
> (In reply to Gene Lian [:gene] (national holidays until 9/23) from comment
> #29)
> > ... does the Commecrial RIL also hook up to the preference
> > 'dom.sms.strict7BitEncoding' to enable/disable the conversion?
> 
> Yes, we honour this preference.

So, we can add more mappings for those special Greek characters both in Moz RIL and Commercial RIL to force them to be 7-bit encoded, which is a solution from the point of view of Gecko. It's also fine to solve it in Gaia. I don't have strong opinions.
(In reply to Julien Wajsberg [:julienw] from comment #45)
> Hey Tim,
> 
> I think Vicamo has not clearly said he doesn't want it in the RIL ;)
> 
> I really think it should not be in the Messages app, this sounds horrible.
> If this doesn't go in the RIL (which Vicamo has not rejected yet), I'd
> really prefer a special keyboard for this use case, as I proposed before
> already. It should be very easy to add a new layout in layout.js.

I object adding such layout the the built-in keyboard app because it's confusing to user -- you will easily ended up having user switching between the "Greek" and "SMS Greek" with the global button. Unless there is a precedent on other OSes I don't think such UX make any remote sense in terms of UX.
I perfectly agree with comment 46, let's move towards this then.

However, we'll still need a way to force caps lock in the SMS app in that situation.

I'll produce a mapping table later today.
Flags: needinfo?(felash)
To solve this issue, here is an alternative solution,

 1. Gaia Keyboard app
    a. Need to show a caps-locked Greek Layout for entering SMS
    b. For now, keyboard app itself does not know which app triggers it, so we might need to introduce a new inputmode for this (not sure if this is another abuse of inputmode) 
      <textarea x-inputmode="sms">
    c. Do the encoding conversion as commented as Comment 36 and comment 46.

For 1.c, I was wondering if this conversion could be done purely in Gaia. Take "Σ" as an example, it the keyboard app output "Σ", U+03A3, would Gecko be able to convert this to 7-bit encoding, as specified here?
http://en.wikipedia.org/wiki/GSM_03.38#GSM_7_bit_default_alphabet_and_extension_table_of_3GPP_TS_23.038_.2F_GSM_03.38
Rudy> gecko already does this for "Σ" actually.
The "Σ" (U+03A3) is the same one as the one in the 7-bit encoding, and if you have a SMS full with this letter, it will be a 160-character SMS.

What it doesn't right now, is eg translate "Χ" (U+03A7) (capital "χ") to "X" (U+0058), and same for the other latin-look-alike green capital letters.
Flags: needinfo?(felash)
Flags: needinfo?(felash)
Flags: needinfo?(felash)
I attached the greek mapping with unicode code points, hopefully this will make things clearer for everyone.
I understood this is a blocker for 1.1 -> leo?
blocking-b2g: - → leo?
(In reply to Julien Wajsberg [:julienw] from comment #48)
> I perfectly agree with comment 46, let's move towards this then.
> 
> However, we'll still need a way to force caps lock in the SMS app in that
> situation.
> 
> I'll produce a mapping table later today.

The mapping is the way forward as discussed. I guess this makes the needinfo request obsolete.

(In reply to Julien Wajsberg [:julienw] from comment #54)
> I understood this is a blocker for 1.1 -> leo?

yes
Flags: needinfo?(steffen.kruessel)
Given that it blocks IOT, blocking.
blocking-b2g: leo? → leo+
Please note that you need to map accented lowercase and uppercase characters as well to their non-accented uppercase counterparts. Which reminds me of the work I had done in bug 307039 that you may find of some use.
Panos> I strongly think we should not do this.

I think we should only map capital greek letters to their latin look-alike counterparts. We should _not_ map small letters or accented letters, except in "7-bit strict mode".
Whiteboard: [good-first-bug]
Julien

Please clarify what the next steps will be.
Flags: needinfo?(felash)
Next step is we need to discuss with Rudy to find what we can do for 1.1 and what we can do for subsequent releases, if different.
Flags: needinfo?(felash)
(In reply to Julien Wajsberg [:julienw] from comment #58)
> Panos> I strongly think we should not do this.
> 
> I think we should only map capital greek letters to their latin look-alike
> counterparts. We should _not_ map small letters or accented letters, except
> in "7-bit strict mode".

But that's precisely what I'm saying, when in 7-bit mode convert everything to this minimal subset. That's what iOS and Android do. I'm not sure I understand how the user will trigger this mode, but I'm sure you guys will come up with something nice :-) The Samsung keyboard in my Android device doesn't seem to allow me to send non-7bit messages, but IIRC iOS used to do that somehow.
If I get it correctly, here we don't want to force automatically 7-bit mode for greek.

But maybe we should ?
I think so, yes. Just tested again and both iOS and Android (Samsung keyboard) use 7-bit mode by default and neither provides an option to just send the text as is. If we decide to provide the alternative to not use this conversion, I'm sure I would prefer having to go through settings to figure out how to send the text the way I want to, instead of finding myself being charged more than I expected.
Panos> we do have a counter on the right of the composer that shows you how many SMS a message will need. Don't you think this is enough for a user to discover this ?

Some french characters trigger also the 70-characters mode, and this is like this for a long time. (now we generally have unlimited SMS but it was not the case several months ago).
I think seeing the amount of characters remaining for the current message is a reasonable way to indicate whether this special mode is in effect or not. If we decide to provide a way to toggle this mode, I assume that it will be through the messages app preferences though, right?
I'd say currently it would be in Settings > Cellular & Data > Message settings. We don't have preferences in the Messages app currently.
Minus - based on comment 24, unsupported feature.
blocking-b2g: leo+ → -
this is blocking release. eng has came up with a solution. leo+
blocking-b2g: - → leo?
back to leo+
blocking-b2g: leo? → leo+
As mentioned in comment 49, we are going to resolve this issue by adding a keyboard layout, i.e. "Greek SMS" to bulit-in Gaia keyboard.
  1. Add x-inputmode="sms" to the contenteditable of SMS, so that keyboard app would know it is "Greek" + SMS app case.
   
  2. For this keyboard layout, 
     a. All uppercase, and it is caps-locked.
     b. No need to show accented letters.
     c. Output the characters which could be converted to GSM 7-bit table.
        (Thanks to Julien for the details)
Assignee: nobody → rlu
Status: NEW → ASSIGNED
Rudy: this inputmode should trigger word suggestions for other languages, but not for greek (otherwise suggestions will use "real" letters instead of the "7-bit" letters). Or if it's enabled for greek, it will need to convert :/ Not sure what's the best solution is here.

Question to DT: should the user be able to trigger the normal greek keyboard too ?
That should be the case, informing the user about the consequences (i.e. only 80 characters) on change.(In reply to Julien Wajsberg [:julienw] from comment #71)
> Question to DT: should the user be able to trigger the normal greek keyboard
> too ?

That should be the case, informing the user about the consequences (i.e. only 80 characters) on change.
I think we can't inform him when he'll switch keyboard. However as you probably know we have a counter that is displayed as soon as we have more than one SMS Segment, so I think this is enough. What do you think ?
Yes, this should work.
(In reply to Julien Wajsberg [:julienw] from comment #71)
> Rudy: this inputmode should trigger word suggestions for other languages,
> but not for greek (otherwise suggestions will use "real" letters instead of
> the "7-bit" letters). Or if it's enabled for greek, it will need to convert
> :/ Not sure what's the best solution is here.

Yes, thanks for the reminding.
I think we cannot just disable the suggestion here and should convert the suggestion result to GSM 7-bit as well.

> 
> Question to DT: should the user be able to trigger the normal greek keyboard
> too ?

Is this a must have requirement or just nice-to-have?
I think this would definitely make the solution even more complicated.
The main problem would be that how would we want to make the keyboard switch between "Greek SMS" and "Normal Greek".

  1. If we use "Global" button to switch to "Normal Greek".
     This will only work on v1.1 but not on v1.2, because with 3rd-party IME framework into v1.2, the layout switching function will be controlled by the system instead of the keyboard app itself.
     So this will introduce a UX inconsistency issue between v1.1 and v1.2 if we go this way.
  
  2. We can use "CapsLock" button to switch to "Normal Greek".
     This might be able to solve the UX inconsistency issue, mentioned in #1, but not sure this is a good idea and we might need UX input here to have a detailed design around this.
     When referencing an Android implementation (HTC), we find the Capslock key is grayed out.

--
+cc Carrie, our UX on keyboard to ask for her opinion.
Flags: needinfo?(cawang)
(In reply to Rudy Lu [:rudyl] from comment #70)
> As mentioned in comment 49, we are going to resolve this issue by adding a
> keyboard layout, i.e. "Greek SMS" to bulit-in Gaia keyboard.
>   1. Add x-inputmode="sms" to the contenteditable of SMS, so that keyboard
> app would know it is "Greek" + SMS app case.
>    
>   2. For this keyboard layout, 
>      a. All uppercase, and it is caps-locked.
>      b. No need to show accented letters.
>      c. Output the characters which could be converted to GSM 7-bit table.
>         (Thanks to Julien for the details)


I'd like to suggest not using "sms", but instead "messages". The Messages app is both sms and mms :)
(In reply to Rick Waldron from comment #76)

> 
> I'd like to suggest not using "sms", but instead "messages". The Messages
> app is both sms and mms :)

Except in that case we want to do this only for SMS and not for MMS ;-)
(In reply to FxOS Deutsche Telekom from comment #72)
> That should be the case, informing the user about the consequences (i.e.
> only 80 characters) on change.(In reply to Julien Wajsberg [:julienw] from
> comment #71)
> > Question to DT: should the user be able to trigger the normal greek keyboard
> > too ?
> 
> That should be the case, informing the user about the consequences (i.e.
> only 80 characters) on change.

i suggest that we stick to all CAP GREEK keyboard only in v1.1
we may consider further changes in later versions
I agree with Joe in Comment 78 above. Let's avoid additional (secondary) requirements that would delay the availability of this patch. It is a launch blocking requirement for Greek subscribers to send SMS of 160 characters, without being charged for 2 messages.

Any further refinement of the requirement that would, for example, allow switching to a normal Greek keyboard in Messaging app, and warn them about potential additional charges, should be avoided.
Comment on attachment 811371 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12518

Hi David,

Please help review this change on adding a new layout for Greek SMS.
The solution is based on Comment 70.

--
Please note that I have not handled the "grey out" visual for the capslock key because I am waiting for the visual design.

Thanks.
Attachment #811371 - Flags: review?(dflanagan)
(In reply to Rudy Lu [:rudyl] from comment #81)
> Comment on attachment 811371 [details]
> Pointer to Github pull request:
> https://github.com/mozilla-b2g/gaia/pull/12518
> 
> Hi David,
> 
> Please help review this change on adding a new layout for Greek SMS.
> The solution is based on Comment 70.
> 
> --
> Please note that I have not handled the "grey out" visual for the capslock
> key because I am waiting for the visual design.
> 
> Thanks.

Dear Rudy,

But user can not input normal GREEK any more with this patch.
Hi,

We ask customer, they does not accept such a modification. It must have a change for user to select between normal GREEK and 160 charactors GREEK. Thanks!
Comment on attachment 811371 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12518

Rudy,

Thanks for a quick patch for a really tricky issue.  I'm giving r- here because I think your changes to needsCandidatePanel() will cause the candidate panel to disappear whenever the user switches to alternate layouts for any keyboard. Having the keyboard change sizes whenever the user wants to enter a number seems unacceptable.

(Note that I have not actually tested this yet, so forgive me if I'm mistaken about it.)

In any case, I think comments 81 and 82 constitute an r- from our partners.

What if we use a layout group so that greek users always have both the el and el-sms layouts available. We wouldn't use x-inputmode="sms" and users would always have to manually switch layouts when texting.

If "el" and "el-sms" are completely different layouts (like sr-cyrl and sr-latn) then we can even define different autocorrect dictionaries for them, and could actually autocorrect this weird sms version of greek.

It does mean, however, that users would have to know to switch layouts if they want to send longer text messages.
Attachment #811371 - Flags: review?(dflanagan) → review-
(In reply to xiupinglong from comment #83)
> Hi,
> 
> We ask customer, they does not accept such a modification. It must have a
> change for user to select between normal GREEK and 160 charactors GREEK.
> Thanks!

Hi Steffen, do you mind confirming this?
Please also see comment 78/79. At this moment, we should finalize v1.1 instead of adding more features / scope. Thanks
Flags: needinfo?(steffen.kruessel)
(In reply to David Flanagan [:djf] from comment #84)

> 
> It does mean, however, that users would have to know to switch layouts if
> they want to send longer text messages.

Actually the partners don't want that the user needs to know to switch. But maybe we need some compromise here ?
Hi all,

From UX perspective, providing two Greek keyboards is quite redundant and confusing.    
We'd suggest provide a caps-locked keyboard with grayed-out caps key in Messages APP. In addition, we want to replace the up arrow icon (on caps key) with the word "SMS" to inform users that this is exclusive for Messages. In fact, this issue has already existed since feature phones came out. 
Hence, using uppercase letters when texting is quite common for Greek users (I heard it from Greek friends, but would like to have more people to confirm it).
Flags: needinfo?(cawang)
I can confirm that greek SMS text is commonly uppercase in both featurephones and smartphones. Moreover, I have never seen a separate keyboard layout as a user-facing option. In the devices I have tested the user cannot send a message in lower case, even though typing the text may look like he could.
(In reply to cawang from comment #87)
> Hi all,
> 
> From UX perspective, providing two Greek keyboards is quite redundant and
> confusing.    
> We'd suggest provide a caps-locked keyboard with grayed-out caps key in
> Messages APP. In addition, we want to replace the up arrow icon (on caps
> key) with the word "SMS" to inform users that this is exclusive for
> Messages. In fact, this issue has already existed since feature phones came
> out. 
> Hence, using uppercase letters when texting is quite common for Greek users
> (I heard it from Greek friends, but would like to have more people to
> confirm it).

Hi all, I've checked with our local team. They are fine with the above approach. Hope that helps. --Steffen
Flags: needinfo?(steffen.kruessel)
My suggestion in comment 84 is easy to implement: not much more difficult than adding support for a new language like we just did for Italian. And because it does not have to modify the keyboard logic, it has very low risk of causing regressions. For the user it means that they would have to learn to switch keyboards to send 160-character text messages. But it would also mean that they could send lower-case texts.  Perhaps this can be seen as a feature and not a bug. Just because other smart phones copy the broken features of older dumb phones does not mean that we have to follow suit.

The proposal discussed in comment 87 and 89 is a new feature, not just a bug fix, and carries significant risk of regression. The first attempt to implement something like that has already been rejected because it caused a regression in all of the other keyboards.

Obviously we all want to make something work for Greek users in 1.1.  But given that we're already past the feature complete deadline for version 1.2, it is way, way, too late to ask us to implement anything beyond the bare minimum functionality.

I suggest two independent greek layouts. Or that we modify the SMS app to convert greek characters as they are typed on the normal greek keyboard.
Julian: if we are not going to give users a choice, is there any reason that we can't resolve this bug in the SMS app just by converting characters to uppercase and/or latin as they are typed on the standard greek keyboard?
Flags: needinfo?(felash)
Comment on attachment 811371 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12518

Hi David,

Thanks for the first review and sorry to make you think that was risky and caused some regression.

[Rationale behind the proposed solution]
I guess why we take this approach is based on 2 facts,
  1. We cannot resolve this issue in Message app because "All uppercase keyboard" is a requirement according to our partner.
  2. The user should see "Greek SMS" layout right away when he/she starts to compose SMS message, with no need to switch layout or something.
  3. I understand adding a new layout, "Greek SMS", as you proposed would make the patch straight-forward, but this would not fulfill #2.

--
[Patch  updated]
I have updated the pull request to add the 2nd commit. (will squash if it gets r+'ed)

In this patch, I just add a flag "forceDisableSuggestion" to control the candidatePanel not to appear in this specific case, which I think make it look more band-aided, but I hope this could do the trick and minimize the risk.

Please help review it again if this makes sense to you.
Attachment #811371 - Flags: review- → review?(dflanagan)
Comment on attachment 811371 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12518

Hi Julien,

We might need your help to check the changes on Message app.

Since this solution is targeted at SMS only, is it OK we add this special inputmode "-moz-sms-7bit" to <div id="messages-input">, which seems to be used for SMS and MMS?

Or we need to add the input mode only when it is SMS.
(Can we do that?)
Attachment #811371 - Flags: review?(felash)
Comment on attachment 811371 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12518

r+, but see my suggestions on github. It would be good also if you could actually disable the "sms" key so it doesn't highlight when you touch it.
Attachment #811371 - Flags: review?(dflanagan) → review+
(In reply to David Flanagan [:djf] from comment #91)
> Julian: if we are not going to give users a choice, is there any reason that
> we can't resolve this bug in the SMS app just by converting characters to
> uppercase and/or latin as they are typed on the standard greek keyboard?

One of the original idea was to do just this in the Gecko API.
If it's done in the Messages app it should be done only when sending the message, but it doesn't feel right to send something different than what is displayed.

Converting on the fly would mess up with the suggestions.

(In reply to Rudy Lu [:rudyl] from comment #93)
> Comment on attachment 811371 [details]
> Pointer to Github pull request:
> https://github.com/mozilla-b2g/gaia/pull/12518
> 
> Hi Julien,
> 
> We might need your help to check the changes on Message app.
> 
> Since this solution is targeted at SMS only, is it OK we add this special
> inputmode "-moz-sms-7bit" to <div id="messages-input">, which seems to be
> used for SMS and MMS?

I don't exactly know how the input mode works: can it be on any parent element ?

> 
> Or we need to add the input mode only when it is SMS.
> (Can we do that?)

I would do this somewhere in https://github.com/mozilla-b2g/gaia/blob/master/apps/sms/js/compose.js#L523

I can work on a patch (based on yours) tomorrow if you want. (my tomorrow is now the Toronto time, so it will be later than usual).
Flags: needinfo?(felash)
(In reply to Rudy Lu [:rudyl] from comment #93)
> Comment on attachment 811371 [details]
> Pointer to Github pull request:
> https://github.com/mozilla-b2g/gaia/pull/12518
> 
> Hi Julien,
> 
> We might need your help to check the changes on Message app.
> 
> Since this solution is targeted at SMS only, is it OK we add this special
> inputmode "-moz-sms-7bit" to <div id="messages-input">, which seems to be
> used for SMS and MMS?
> 
> Or we need to add the input mode only when it is SMS.
> (Can we do that?)

i feel that we should stick to applying this ALL CAP greek keyboard when typing a message content in message app. Distinguishing between SMS/MMS is adding more scope than originally anticipated and let's try to make things simple for v1.1 and we may discuss further for the future keyboards
Joe, I really think this is wrong to do this, especially that it's not a lot more complicated. Since the start this is the scope I had in mind...
The attached is for you guys to see how it looks like for now.
Now we would have a grayed out Caps-locked key for Greek SMS as suggested by Joe.

I've not landed the code because I might need Carrie to confirm this is good to go.

--
Julien,

Are you saying that you think we should separate SMS case from MMS case?
If that is the case, I am going to check in the keyboard related changes, so that you can follow up to address that (adding the x-inputmode for SMS only) in messages app.
Thanks.
Rudy: yep I think we should do that.

I'll produce a patch today and I think I can even make it reviewed by Rick before the end of the day ;)
Attached patch sms patch v1 (obsolete) — Splinter Review
Borja and Steve are in PTO so there is nobody before the US timezone. Therefore I'm requesting review from Rick.

Requesting feedback from rudy, to test both patches together.

Set x-inputmode=sms for SMS only
---
 apps/sms/index.html                |    3 ++-
 apps/sms/js/compose.js             |   16 +++++++++++++---
 apps/sms/test/unit/compose_test.js |   19 +++++++++++++++++++
 3 files changed, 34 insertions(+), 4 deletions(-)

Pull request is at https://github.com/mozilla-b2g/gaia/pull/12602
Attachment #812954 - Flags: review?(waldron.rick)
Attachment #812954 - Flags: feedback?(rlu)
Comment on attachment 812954 [details] [diff] [review]
sms patch v1

Hi Julien,

Basically, I think this is OK, except that we have to change the value for x-inputmode from "sms" to "-moz-sms-7bit" per my patch.
I did not try your patch on device because my patch is based on v1-train and seems yours is on master.

Besides, I have a question that, when we switched to MMS mode by attaching a picture, we can delete the picture by pressing [Delete] key on keyboard.
Will this make it back to "SMS" mode?

If not, then we are fine, since in this case, keyboard app would not get an update of the inputmode change.
If yes, it would be changed to "SMS", then since keyboard app would not know, it would still show the "normal Greek" layout.

--
Thanks for your quick response on this issue.
Attachment #812954 - Flags: feedback?(rlu) → feedback+
(In reply to Rudy Lu [:rudyl] from comment #101)
> Comment on attachment 812954 [details] [diff] [review]
> sms patch v1

> 
> Besides, I have a question that, when we switched to MMS mode by attaching a
> picture, we can delete the picture by pressing [Delete] key on keyboard.
> Will this make it back to "SMS" mode?

Yep this moves to SMS mode.

> 
> If not, then we are fine, since in this case, keyboard app would not get an
> update of the inputmode change.
> If yes, it would be changed to "SMS", then since keyboard app would not
> know, it would still show the "normal Greek" layout.

Ok, I get it, you only get the change when we focus/blur the input, right ?

Do you think this issue is serious ? (I mean: the user already added an attachment, then removed it, so this is a somewhat corner case). Do you have an idea to overcome it ? Maybe, in keyboard app or in the keyboard module in gecko, you shuld listen to the attribute value change mutation events for the element it is displayed for ?
The keyboard part patch has been merged to v1-train,
https://github.com/mozilla-b2g/gaia/commit/735d9f7871d4352989c54ce7143c8a0163f3418f

Please refer to the attachment for the final design.
(In reply to Julien Wajsberg [:julienw] (PTO today) from comment #102)
> (In reply to Rudy Lu [:rudyl] from comment #101)
> > Comment on attachment 812954 [details] [diff] [review]
> > sms patch v1
> 
> > 
> > Besides, I have a question that, when we switched to MMS mode by attaching a
> > picture, we can delete the picture by pressing [Delete] key on keyboard.
> > Will this make it back to "SMS" mode?
> 
> Yep this moves to SMS mode.
> 
> > 
> > If not, then we are fine, since in this case, keyboard app would not get an
> > update of the inputmode change.
> > If yes, it would be changed to "SMS", then since keyboard app would not
> > know, it would still show the "normal Greek" layout.
> 
> Ok, I get it, you only get the change when we focus/blur the input, right ?

Correct.

> 
> Do you think this issue is serious ? (I mean: the user already added an
> attachment, then removed it, so this is a somewhat corner case). Do you have
> an idea to overcome it ? Maybe, in keyboard app or in the keyboard module in

IMHO, I don't think this is a serious issue, and it seems we don't have an easy solution for this.

> gecko, you shuld listen to the attribute value change mutation events for
> the element it is displayed for ?

Yes, I think you're right, but seems it is too late to make any changes to Gecko part at this point and this still requires some Gaia changes as well.

So, I guess we have to live with that for v1-train.
Comment on attachment 812954 [details] [diff] [review]
sms patch v1

Review of attachment 812954 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with the changes that rudyl noted and the nits I marked
Attachment #812954 - Flags: review?(waldron.rick) → review+
Attached patch sms patch v2Splinter Review
final patch with follow-ups, pushed to PR https://github.com/mozilla-b2g/gaia/pull/12602

waiting for travis before merging to master.

v1-train patch ready too.
Attachment #812954 - Attachment is obsolete: true
Attachment #813513 - Flags: review+
More importantly: I checked on the device and it works perfectly for me :)

Rudy, I would have used -moz-sms instead of -moz-sms-7bit, because using -moz-sms-7bit is misleading, as we don't for 7bit for most layouts.

Do you want that I open another bug for doing a patch for master and 1.2 for this issue (with mutation events ;) ) ?
master: 4e14ffad41827a6aa10f3d62d3a8831ea626c738
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
sms patch v1-train: e5adcf3c23b6709a51087539bc4cdb67683ca4a0
Attachment #811371 - Flags: review?(felash)
v1.1.0hd: e5adcf3c23b6709a51087539bc4cdb67683ca4a0
v1.1.0hd: 735d9f7871d4352989c54ce7143c8a0163f3418f
v1.1.0hd: 0f71331e00c49fb2b754c843bfa3f739da82d834
(In reply to Julien Wajsberg [:julienw] (in MozSummit until next monday) from comment #107)
> More importantly: I checked on the device and it works perfectly for me :)
> 
> Rudy, I would have used -moz-sms instead of -moz-sms-7bit, because using
> -moz-sms-7bit is misleading, as we don't for 7bit for most layouts.

Yes, I think you're right.
Sorry that did catch the subtleties here.

> 
> Do you want that I open another bug for doing a patch for master and 1.2 for
> this issue (with mutation events ;) ) ?

Sounds good to me to open another one since this bug has been closed.
Thanks for your help.
Uplifted 4e14ffad41827a6aa10f3d62d3a8831ea626c738 to:
v1.2: c23385ec2719f7bc0652718764a6a544f63ed865
It seems like autocorrection is partially disabled in my 1.1.1.0 build (20131009041203), in that suggestions are not displayed, but what I type is changed after I press space. Is that intentional? Can't we either have autocorrection fully enabled or fully disabled?
I think it's supposed to be fully disabled.
Flags: needinfo?(rlu)
Bug 927321 has addressed the issue Comment 114 and 115 mentioned.
Thanks for bringing this up.
Flags: needinfo?(rlu)
See Also: → 925003
You need to log in before you can comment on or make changes to this bug.