878820 - Contacts imported from VCF files disappear

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Contacts, defect)

Description

[Sheeri Cabral [:sheeri]]

Peak phone, on 5/31 nightly build.

When I enter a contact, they disappear within a few hours. I have tried this importing contacts from a VCF file on the SD card, as well as when entering phone numbers manually. There is no error or crash from the contacts program, just when you go to contacts, it's empty. (If you type in the phone number, it exists under the name "UTF-8" though. Very odd.)

Comment 1

[Francisco Jordano [:arcturus] [:francisco]]

Hi,

could you specify if your build is 1.0.1 or 1.1

As well, when you comment that this happens after some few hours, could you try to import your contacts, and actively kill the contacts application and try to launch it again?

Thanks.

Comment 2

[Sheeri Cabral [:sheeri]]

If I import, the contacts are there. I go to home screen, open contacts, the contacts are there still. I go to the home screen, kill the app (holding down the home button and closing contacts), indeed, the contacts disappear.

Software says Boot2Gecko2.0.0.0-prerelease
OS version says 2.0.0.0-prerelease
Platform version says 24.0a1
Build Identifier says 20130531022309
Update Channel is nightly
Git Commit info is 2013-05-30 20:33:07, md5sum starts with 126ca68 and ends with 52dc6bf

Not sure if that helps at all?

Comment 3

[Francisco Jordano [:arcturus] [:francisco]]

Thanks Sheeri, you are running a build against master.

Will try to reproduce with master.

Thanks!
F.

Comment 4

[Sheeri Cabral [:sheeri]]

Is there any update on this? It's been 10 days and I stopped using the phone and went back to using my regular phone, because if it can't save my phone contacts, it's not really useful as a phone.

Comment 5

[Francisco Jordano [:arcturus] [:francisco]]

@reuben, are you aware of any problem with the db migration for contacts?

@Sheeri, still happening with latest builds provided by geeksphone here:

http://forum.geeksphone.com/index.php?topic=5275.0

Thanks!

Comment 6

[Reuben Morais [:reuben]]

(In reply to Francisco Jordano [:arcturus] from comment #5)
> @reuben, are you aware of any problem with the db migration for contacts?

No, but it does look like a backend bug ("If you type in the phone number, it exists under the name "UTF-8" though"). I wonder if the GP builds do something funny to the Contacts code. I can't really test this because I don't have a Peak…

Comment 7

[Gregor Wagner [:gwagner]]

It doesn't seem to reproduce locally. Can we get some help from QA here? I would like to see the logcat and the actual database file.

Comment 8

[Reuben Morais [:reuben]]

Sheeri, can you run |adb logcat|, then try saving a contact, killing the app, and opening it again, and attach the logcat output in the bug?

Also, can you grab the Contacts DB file from your phone and send it to my email? You can get it by doing:

adb pull /data/local/indexedDB/chrome/idb/3406066227csotncta.sqlite .

Comment 9

[Sheeri Cabral [:sheeri]]

Attachment: adb logcat

Hrm, I just flashed the nightly build from last night and the Contacts app was able to retain the contact information. I tried with one contact, adding manually, and then restarting the phone.

Then I tried and added the contacts from the SD card (where I had saved them earlier) and it loaded the 564 contacts I have. I was able to context switch and come back with the contacts still there. Upon restart, my contacts are empty and if I dial a phone number in my contacts, the dialer still shows "UTF-8".

I had logcat running during some of this, after I did all this and went back to my session with logcat, it had completed, so I'm not sure exactly when it completed.

Comment 10

[Gregor Wagner [:gwagner]]

Adding Sergi. This might be a vcard parsing error.

Comment 11

[Sheeri Cabral [:sheeri]]

To be clear, when it imports the contacts, it gets them right. So the initial import worked. But close and re-open the contacts program, and it's gone :(

Comment 12

[Reuben Morais [:reuben]]

Sheeri, how did you create the VCF file that you imported? Can you email it to me? It seems something is going wrong during the import code, so you see the contacts in the app, but they don't get properly saved in the DB, so if you reboot or kill the app they're not there anymore.

Comment 13

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Just FYI, Gregor and I looked through the parsed data and found the following obvious problems:

- telephone number type == "UTF-8", not "Work"...
- email type == "UTF-8", not "Home"
- genderIdentity/sex == the string "undefined"
- search search for name is completely wrong, one letter only, e.g ["h", "o", "o", "v", "e", "r"] instead of ["h", "ho", "hoo", "hoov", "hoove", "hoover"]
- updated/published may be a string instead of Date object?
- name field has array of objects with 'meta'/'value' properties
- givenName/familyName are not arrays

Comment 14

[Gregor Wagner [:gwagner]]

Sheeri, are you in any irc channels? It would make this async conversation easier :)

Comment 15

[Sheeri Cabral [:sheeri]]

Yes, I'm in #db. (I had Mon and Wed off for PTO, which is why this conversation is dragging on, too).

I have e-mailed Reuben the VCF file I imported (15.5 Mb), which worked on initial import - all the names showed up, with the proper phone numbers. It's only after I closed the Contacts app and reopened it that the names all changed to UTF-8.

Comment 16

[Ben Turner (not reading bugmail, use the needinfo flag!)]

(In reply to Sheeri Cabral [:sheeri] from comment #15)

Thanks!

Additionally, from Sheeri: "The contacts db is an iPhone export using a program called MC Backups"

Comment 17

[Reuben Morais [:reuben]]

There are a couple of problems going on here. First, the importer is not creating the mozContact objects correctly, and second, we're not sanitizing the incorrect data.

For example:
{"properties":{"name":[{"meta":{"charset":"UTF-8"},"value":["Red Acted"]}],"givenName":"Red","familyName":"Acted","adr":[],"tel":[{"value":"1 [phone]","type":"UTF-8"}],"org":["RedAct"],"sex":"undefined","genderIdentity":"undefined"},"search":{"givenName":["r","e","d"],"familyName":["a","c","t","e","d"],"email":[],"category":[],"tel":[*snip* (this one is right)],"exactTel":["12345678901"],"parsedTel":["12345678901","null","","45678901"]}}

0) What are we doing wrong with the object in |name|?
1) Why are givenName and familyName not arrays? (Also breaks search.{givenName,familyName})
2) "undefined" should be filtered out.
3) What's up with parsedTel?

Comment 18

[Reuben Morais [:reuben]]

The VCF importer is creating its own distorted version of a mozContact object, then passing that directly to ContactManager.save. We don't do any extra validation in save because we expect it to be a mozContact object, not a random JS object.

https://mxr.mozilla.org/gaia/source/apps/communications/contacts/js/utilities/vcard_parser.js?mark=31,59,60#14

Comment 19

[Sergi Mansilla [:sergi] (Telenor) [PTO]]

I'll take it. Can I have the VCF file you were testing with?

Comment 20

[Reuben Morais [:reuben]]

Attachment: Patch

Comment 21

[Sheeri Cabral [:sheeri]]

Sergi, I sent it to Reuben the other day (and he might have already fixed it?) Just clearing the needinfo....

Comment 22

[Sergi Mansilla [:sergi] (Telenor) [PTO]]

Hi Sheeri and Reuben,

I'm finalizing a patch for this that has more fixes and refactoring and speeds up and clarifies the code. It also contains extra unit tests.

Also, Reuben's patch breaks the tests and doesn't work.

Since I have taken the issue, please allow me to submit the patch I am working on, I believe it makes the VCF import much more solid. Hopefully I will have it ready today.

Comment 23

[Augustin Trancart [:autra]]

Just for information, I experience the same bug on Keon.

Comment 24

[Reuben Morais [:reuben]]

Yep, this is a VCF importer bug, and I've reproduced on the Keon while testing that patch. (It works in that it fixes the main bug, but we have other problems). Changing the bug title to match reality.

Comment 25

[Sergi Mansilla [:sergi] (Telenor) [PTO]]

Attachment: Github PR

Added reference to Github PR fixing the bug and some general issues, added more tests.

Comment 26

[Sergi Mansilla [:sergi] (Telenor) [PTO]]

Comment on attachment 766642 [details] [diff] [review]
Github PR

>https://github.com/mozilla-b2g/gaia/pull/10573

Comment 27

[Sergi Mansilla [:sergi] (Telenor) [PTO]]

Attachment: New Github PR

In the previous review of this issue, there was an error with photo fields. It turns out that all the API docs and tests in Gaia (!!) are doing the wrong thing when assigning values to the photo property of a mozContact object. The value of photo HAS to be a Blob, and not a String. Upon pointing this out, some API docs have been corrected, but many tests in Gaia are not (yet).

Since there is another issue regarding Vcard photo import (https://bugzilla.mozilla.org/show_bug.cgi?id=885545), I will tackle that one and add photo importing there. For now, photo importing for the contact is disabled since it was wrong anyway.

Comment 28

[Francisco Jordano [:arcturus] [:francisco]]

Comment on attachment 767124 [details] [diff] [review]
New Github PR

\o/

Thanks Sergi!

Comment 29

[Sergi Mansilla [:sergi] (Telenor) [PTO]]

Merged at cc91e29e74038d575583ee359d3642533403a11d

Comment 30

[Julien Wajsberg [:julienw]]

please fix the tests (bug 886992) or I'll back out this change as this makes travis red, and travis red makes me sad, and sad julien is angry ;)

Comment 31

[Francisco Jordano [:arcturus] [:francisco]]

+1 to become leo+ since this functionality will be half broken without this.

Comment 32

[Sergi Mansilla [:sergi] (Telenor) [PTO]]

Promoting to leo? because without this fix imported contacts via Vcard will disappear and that's terrible for the user.

Comment 33

[Tony Chung [:tchung]]

Hi Sheeri, it's not clear if this bug is happening on 1.1 or master only.  Can you try reproducing this on 1.1?  if so, we will revisit this patch for leo+ triage.  Thanks.

Comment 34

[Sheeri Cabral [:sheeri]]

Hi Tony,

I have a geeksphone so I've been using whatever they provide at http://forum.geeksphone.com/index.php?topic=5275.0

Can you link me to a place where there's some documentation on how to get the 1.1 branch, especially if the installation instructions to flash a ROM are different? thanx!

Comment 35

[Julien Wajsberg [:julienw]]

This is a master branch.

Unfortunately their "stable" branch comes from 1.0.1, geeksphone provides no 1.1 build. Maybe it could be easier to just ask them ? I'll ask on the mailing list.

Comment 36

[Reuben Morais [:reuben]]

(In reply to Tony Chung [:tchung] from comment #33)
> Hi Sheeri, it's not clear if this bug is happening on 1.1 or master only. 
> Can you try reproducing this on 1.1?  if so, we will revisit this patch for
> leo+ triage.  Thanks.

Tony, this bug happens on 1.1. Do you need a VCF file to reproduce and test locally?

Comment 37

[Sheeri Cabral [:sheeri]]

Checking my assumptions - I'm assuming Reuben's comment 36 means I don't have to try to figure out where to flash 1.1 from. Please let me know if there is something you're waiting on me for, but for right now I will assume there is not.

Comment 38

[Sergi Mansilla [:sergi] (Telenor) [PTO]]

Merged in v1-train: 368489e25f9406d6339e4414d6ce52b0caac0e5d

Comment 39

[John Ford [:jhford] CET/CEST Berlin Time]

v1.1.0hd: 368489e25f9406d6339e4414d6ce52b0caac0e5d
v1.1.0hd: 16003fe6a7b88bb51a2e13943b8fc8fc06e12172

Comment 40

[sarsenyev]

Verified fixed, the issue is no longer reproduces, imported contacts from SIM card are not disappearing, after killing the app and relaunching 

Environmental  Variables:
Build ID: 20130730070228
Gecko: http://hg.mozilla.org/releases/mozilla-b2g18/rev/1fe3339e3d96
Gaia: 6221737cb50d6e8435ac5d3fe3b7e2788bd8a37c
Platform Version: 18.1