953002 - [B2G] [Contacts] Contact imported from SIM will not show fully if the last name contains special characters

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::Contacts, defect)

Description

[Chris Kreinbring [:CKreinbring]]

Attachment: Contacts with special characters - full name

Description:
If the user imports a contact from the SIM with special characters in the last name, the contact's last name will become truncated.

Repro Steps:
1) Update Buri to Build ID: 20131223004001
2) Have contacts saved to the SIM with special characters.
3) Launch the Contacts app.
4) Tap the gear icon, then Import Contacts, then SIM.
5) Wait for the contacts to be imported from the SIM to the device.
6) Return to the contact list.
7) Find the contact with special characters and observe its appearance.

Actual:
The last name is truncated if it has special characters.

Expected:
The entire name is shown regardless of the presence of special characters.

Environmental Variables
Device: Buri 1.3 mozilla RIL
Build ID: 20131223004001
Gecko: http://hg.mozilla.org/releases/mozilla-aurora/rev/af28fe58e263
Gaia: 01e9da49be2cc4bc134eeefc434740d572ec2246
Platform Version: 28.0a2

Notes:
Also repros on Buri 1.3 mozilla RIL
Repro frequency: 100%
Link to failed test case: https://moztrap.mozilla.org/manage/cases/?filter-id=5609
See attached screenshots

Comment 1

[Chris Kreinbring [:CKreinbring]]

Attachment: Contacts with special characters - Truncated after import from SIM

Comment 2

[Jason Smith [:jsmith]]

Does this reproduce on 1.2?

Comment 3

[Chris Kreinbring [:CKreinbring]]

It does.  I put 1.3 instead of 1.2 by accident in the notes.

Comment 4

[Jason Smith [:jsmith]]

Does this reproduce on 1.1?

Comment 5

[Montoya Clemmons (MClemmons)]

In reply to Jason Smith [:jsmith] from Comment 4

Does this reproduce on 1.1?

Unable to reproduce this issue on Buri 1.1 with the below Environmental Variables. Entire name displayed on device regardless of the presence of special characters. 

Since Buri 1.1 does not have ability to export contacts, contact with special characters created on a personal Blackberry device and exported to SIM. SIM was removed from Blackberry device and inserted in Buri 1.1 device. Imported contact with special characters on Buri 1.1 device. Did not observe truncation behavior on 1.1 Buri device. 


Environmental Variables:
Device: buri 1.1 MOZ
BuildID: 20140102041202
Gaia: 6ff3a607f873320d00cb036fa76117f6fadd010f
Gecko: bdac595a4e46
Version: 18.0
RIL Version: 01.01.00.019.281

Comment 6

[Jason Smith [:jsmith]]

This sounds like a unicode parsing regression - we're losing unicode characters on import. We're no longer tracking 1.2 bugs anymore, but this would have blocked 1.2 originally, given that many target launch locales we work with deal with a lot of different unicode characters in names. As such, this needs to block 1.3.

Comment 7

[Montoya Clemmons (MClemmons)]

The first 1.2 build that reproduces the actual result (broken) following the steps to reproduce in Comment 0 is August 1 2013. Below are the Environmental Variables. 

Environmental Variables
Device: Buri 1.2 MOZ
BuildID: 20130801030224
Gaia: c1620a242c937c7fcb171b88c85ef63561165081
Gecko: 05d3797276d3
Version: 25.0a1
RIL Version: 01.01.00.019.281

I was unable to locate a 1.2 build that shows the expected result (last working) following the steps to reproduce in Comment 0. From the very first 1.2 build (June 21) through July 31 2013 - user is unable to import contacts from SIM. The message displayed on device 'Could not import from SIM' after attempting Step 4 from the Steps to Reproduce in Comment 0 - Tap the gear icon, then Import Contacts, then SIM.

Below are the Environmental Variables from the 1.2 July 31 build
Device: buri 1.2 MOZ
BuildID: 20130731030205
Gaia: 9bfceaa90e8b92a379432b67121afa3cd3f14c90
Gecko: c2b375f3a909
Version: 25.0a1
RIL Version: 01.01.00.019.281

Comment 8

[Montoya Clemmons (MClemmons)]

Attachment: Screenshot ContactsErrorMsg

Comment 9

[Joe Cheng [:jcheng] (please needinfo)]

triage: 1.3+ regression

Comment 10

[Francisco Jordano [:arcturus] [:francisco]]

Will need to take a look to know if that problem is happening in the gaia side or the API one.

Comment 11

[Francisco Jordano [:arcturus] [:francisco]]

Hi after doing some tests, I realised that we are already receiving truncated information from the backend when we perform the |icc.readContacts| Dom request.

I also assume this will be happening in Settings with the FDN contacts.

For an entry on the sim card with:
name: π|√$© ™•••
value: 666666666

We get the folling answer from the backend:
[{"id":"89340711070024031851","name":[" | $      "],"tel":[{"value":"666666666"}]}]

So seems the RIL is not returning us the correct values.

Pinging Yoshi for more info.

Comment 12

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Is there RIL log for this?
I need to check the encoding scheme stored on the SIM, which I didn't find it in this bug.

Comment 13

[Jason Smith [:jsmith]]

Adding qawanted to get a RIL debug log with this bug.

Comment 14

[Sarah Parsons]

Attachment: 953002_RIL_Log

See RIL Log attached.

Comment 15

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

(In reply to Sarah Parsons from comment #14)
> Created attachment 8359346 [details]
> 953002_RIL_Log
> 
> See RIL Log attached.

Hi Sarah
Is the contact in Comment 0 and Comment 1 in this log ?
From the log line 6615 to line 6867, I didn't see any similar alphabet related to contact mentioned in Comment 0 and Comment 1.

Also I don't know how to type 'π', '√', '™', '•••' characters on the phone from Comment 11.

I've also tried French alphabets : http://en.wikipedia.org/wiki/French_alphabet
Polish alphabets: http://en.wikipedia.org/wiki/Polish_alphabet

but they're working well.

Can you also post the *original* characters for the contact here ?
For example, in Comment 0 and Comment 1, what characters are missing/wrong ?

Comment 16

[Francisco Jordano [:arcturus] [:francisco]]

Hi folks,

to try this, I just got an android phone and export to the sim a contact which those characters (my android keyboard allows me to write them), then import from in FirefoxOS.

Hope that can help.

Thanks!

Comment 17

[Chris Kreinbring [:CKreinbring]]

One contact is Ḿõţľĕŷ Ṣḉṝḛẅḗḋ, the other is Mötlé¥ $crếẉēď.

Comment 18

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

I didn't find Ḿõţľĕŷ nor Mötlé¥ in the log attached in Comment 14,
Can  [e-mail]  confirm this ?

Also My Android phone doesn't have European language.....
Can Francisco help to capture the RIL log for me ?
Or can someone help to explain how do I key-in ḉ on the phone (Android, B2G) ?
Which input-method should I choose ?

From Wiki it doesn't tell much information.
http://en.wiktionary.org/wiki/Ḉ

On B2G the Polski keyboard has C with Cedilla, but without the accent.

Comment 19

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

I didn't find Ḿõţľĕŷ nor Mötlé¥ in the log attached in Comment 14,
Can  [e-mail]  confirm this ?

Comment 20

[isabelrios]

Hi, we are trying to reproduce this issue but we are facing a limitation when creating contacts with all that characters. With an android device we can only create a contact with 8 strange characters (name and last name) in total. And exporting that contact to SIM and then importing it to FXOS device, we cannot see that problem. The name and last name are shown fine.

Reading the str seems that the contacts were in the SIM (created previously) and then imported into the FXOS device but ni [e-mail]?  to confirm that, because looking at the screenshots attached looks like they were created manually in the device.

Comment 21

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Just write an automation test to verify this, RIL can decode ḉ successfully, its unicode is 0x1e09.

However I still cannot find out which input method could type 'ḉ', it seems from French, but with the accent.

I still need a correct RIL log to check this.

Comment 22

[isabelrios]

Hi Yoshi,

I am able to reproduce this issue following these steps:
1.	Manually create this contact on a FXOS device: Name: Mîō$ Last Name: Žâçłēú¥
2.	Then export it to SIM Card
3.	On contacts app , delete that contact
4.	And then, go to gear icon to import from SIM and import the contacts available in the SIM card
5.	Tap on Done and go back to contacts list
6.	The contact is now shown as:  Name: Mîō$ Last Name:Ž

Please have in mind that the steps are a bit different than the ones in comment 0. Attached you could find the RIL logs I got while reproducing the bug the way I am able to reproduce it. Hope it helps, and please let us know in case the bug is different to file a new one. Thanks

Comment 23

[isabelrios]

Attachment: RIL logs ContactStrangeChars.txt

Comment 24

[Sarah Parsons]

Removing qawanted per comment 22 and comment 23

Comment 25

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Thanks Isabel,
your log is very useful. :)

After checking your log, I found actually the characters are truncated, not "didn't show up'.

From the 3GG spec, the maximum size of the name filed for a SIM Record is 14 bytes.

1 byte for the encoding byte (0x80), indicating this is UCS2 

2 bytes for the first character 'M'
, same for the following characters 'îō$ Ž',

6 UCS2 characters (6*2) + 1 encoding chars(1*1) = 13 bytes used.

writing one more UCS2 character will cause it overrun (15 bytes),
so you can see the following character are actually not written into SIM.

If you'd like to test the tail characters can be shown or not,
you could remove the first 'Mîō$ Ž' characters.

Comment 26

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Noemi, what do you think about this bug?
Do you still think it's a Gecko bug ?

Comment 27

[Noemí Freire (:noemi)]

Hi,

Per Yoshi comment 25, due to the max length for 'Name' (FirtName, LastName) field in the SIM Card is set to 14bytes, an existing local contact with FirstName: Mîō$ and LastName: Žâçłēú¥ will be always truncated when exported to SIM Card to 6 UCS2 characters at most, it means the information stored in the SIM card will be FirstName: Mîō$ LastName:Ž so when importing that contact from SIM Card that will be the info available for the contact imported. It seems this issue should be marked as RESOLVED WONTFIX, it is a SIM card limitation. Anyway it would be fine to understand the steps followed in comment 5 since it seems that it was properly working when exporting the contact to SIM card using a BB device and importing it using a Buri one, ni to [e-mail] in order to clarify that scenario, maybe some SIM cards have a bigger max length value set for 'Name' field?. Thanks!

Comment 28

[Montoya Clemmons (MClemmons)]

In response to Comment 27  Noemí Freire (:noemi) 2014-01-22 08:09:01 PST 

"Anyway it would be fine to understand the steps followed in comment 5 since it seems that it was properly working when exporting the contact to SIM card using a BB device and importing it using a Buri one, ni to [e-mail] in order to clarify that scenario, maybe some SIM cards have a bigger max length value set for 'Name' field"

Below were the steps followed in Comment 5 

1. Removed SIM card from Buri 1.1 device and placed it within user personal Blackberry device
2. User created contact with special characters on Blackberry device
3. User exported contact to SIM card that was removed from Buri 1.1 device and inserted within Blackberry device
4. Removed SIM card from Blackberry device and inserted within Buri 1.1 device
5. Within Contacts App on Buri 1.1 device, imported contact from SIM card 
6. Contact created on Blackberry device with special characters did not display with truncation behavior described in Comment 0

Comment 29

[Jason Smith [:jsmith]]

What happens if you use a 1.3 Buri device in the STR followed in comment 28?

Comment 30

[Montoya Clemmons (MClemmons)]

In response to Comment 29  Jason Smith [:jsmith] 2014-01-22 10:45:47 PST 

What happens if you use a 1.3 Buri device in the STR followed in comment 28?

Following the exact steps from comment 28 on the latest 1.3 Buri does not reproduce the issue in Comment 0.

However, modifying the STR in comment 28 slightly does reproduce the issue in Comment 0. Please note, it was recently discovered that the fields inputted on the Blackberry didn't correspond field for field when observed on the Buri. Adjustment below had to be made on the Buri to get reproducible behavior from Comment 0.

Below are the modified STRs and Environmental Variables:

1. User with a personal Blackberry device create a Contact with special characters included
2. On Blackberry device, exported contact to SIM card that was removed from a Buri 1.1 device and inserted within Blackberry device
3. Removed SIM Card from Blackberry device and inserted in Buri 1.1 device
4. Imported contact from SIM card within Buri 1.1 device
5. Contacts appear in the first name field only; therefore, open contact and insert a last name (ex: NoNo)
6. Export Contact to SIM card 
7. Delete contact from within Contacts app that was exported to SIM card
8. Import contact from SIM card back to Buri device

Actual: Contact last name field truncated (ex: NoNo last name appears as N)
Expected: Contact last name appears in full (ex: NoNo last name appears as NoNo)

Environmental Variables
Device: buri 1.3 MOZ
BuildID: 20140122004001
Gaia: 744fb691c2b2a25a07c5d19fabf5748ae9aba4d9
Gecko: 71a8786c3815
Version: 28.0a2
Base Image: v1.2-device.cfg

Other information: Attached file of the truncation (bottom name that has a last name beginning with N). It should read NoNo

Comment 31

[Montoya Clemmons (MClemmons)]

Attachment: TruncationBottomNameOnly

Comment 32

[Jason Smith [:jsmith]]

If we're technically following the spec here, then we're doing the right behavior in terms of what gets cut off at the RIL layer. However, the UX is a problem here - the user should be aware when the information from the contact stored on the phone differs from what's being exported. I'm going to open a separate bug for this.

Comment 33

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

This is not regression, remove 'regression' keyword.