1030637 - Improve the performance of importing contacts from SD card

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::Contacts, defect, P2)

Description

[Na Hao]

Steps to reproduce:
-------------------------------------------------------
1. Launch Contact
2. Tap on the setting icon on the top right corner
3. Select 'Import Contacts'
4. Select the Memory Card


Actual result:
-------------------------------------------------------
Import contacts from SD card which total space is 16G and 7G left,It takes over 1min to find the available vcf files which contain 488 contacts before import them.It's longer when the left space is smaller.

root cause:
-------------------------------------------------------
The retrieveFiles method of SdCard in communications/contacts/js/utilities/sdcard.js takes the most of time to find the possible vcard type files,than the filearray is delivered to getTextFromFiles method to import the contacts.

Comment 1

[[:fabrice] Fabrice Desré]

Do you mind attaching the test file you use for imports?

Comment 2

[Na Hao]

(In reply to Fabrice Desré [:fabrice] from comment #1)
> Do you mind attaching the test file you use for imports?

sorry,I'm a novice programmer,would you mind helping me to write the test file? I don't how to start it?

Comment 3

[[:fabrice] Fabrice Desré]

(In reply to na.hao from comment #2)
> (In reply to Fabrice Desré [:fabrice] from comment #1)
> > Do you mind attaching the test file you use for imports?
> 
> sorry,I'm a novice programmer,would you mind helping me to write the test
> file? I don't how to start it?

I just need the file with the 488 contacts that you talk about in comment 0 to reproduce the issue on my side. Feel free to email it to me if you don't want it to be public. thanks!

Comment 4

[Na Hao]

Attachment: 4vcard.zip

the 488 contacts

Comment 5

[Na Hao]

(In reply to Fabrice Desré [:fabrice] from comment #3)
> (In reply to na.hao from comment #2)
> > (In reply to Fabrice Desré [:fabrice] from comment #1)
> > > Do you mind attaching the test file you use for imports?
> > 
> > sorry,I'm a novice programmer,would you mind helping me to write the test
> > file? I don't how to start it?
> 
> I just need the file with the 488 contacts that you talk about in comment 0
> to reproduce the issue on my side. Feel free to email it to me if you don't
> want it to be public. thanks!

I have post the vcf files,and also send an email to you,thank u.

Comment 6

[Na Hao]

In contrast，with the same SD card, the tcl one touch takes 20s and flame_v1.4 takes over 2min to find the vcf files.I found that most of the time is spent on retrieveFiles method.

Taking an example of one test， the total time spent on retrieveFiles method is 68.647s and 66.775s of them is used for searching and judging if it is the vcf file.so when there are many small files,its search time is longer than the big ones even though they take the same space in SD card.So the perfomance of importing sort of depends on the SD status.

Comment 7

[Joe Cheng [:jcheng] (please needinfo)]

not blocking as function works and its not frequently used.
add to backlog

Comment 8

[Peipei Cheng (needinfo if you need my action)]

It seems Buri is faster on retrieving vcf files. Following is my test results with the same SD card(4G, 2.6G free). 
Both tests are based on the same steps: Insert SD card -> restart device)

Buri v1.4: 
-> It only took less than 10s to find the vcf file.
-> Total time is 1 min 20s

Dolphin:  
-> It took about 20s to find the vcf file
-> Total time is 1 min 40s

Comment 9

[Na Hao]

(In reply to pcheng from comment #8)
> It seems Buri is faster on retrieving vcf files. Following is my test
> results with the same SD card(4G, 2.6G free). 
> Both tests are based on the same steps: Insert SD card -> restart device)
> 
> Buri v1.4: 
> -> It only took less than 10s to find the vcf file.
> -> Total time is 1 min 20s
> 
> Dolphin:  
> -> It took about 20s to find the vcf file
> -> Total time is 1 min 40s

hi pcheng,
thank u for your test.Could you please try an 8G or 16G SD card （more usually be used）with the same percent of left space as mine ? Because when there is lots of small files,it really takes a long time to find vcf file,the test condition is important.

Comment 10

[Peipei Cheng (needinfo if you need my action)]

I agree that with more small files, the performance is worse. Here is my test results.

8G SD card (7G free). 
---------------------------------------------------------------------------------
Buri v1.4: 
-> It only took about 6s to find the vcf file.
-> Total time is 1 min 25s
Dolphin:  
-> It took about 12s to find the vcf file
-> Total time is 1 min 20s. 

8G SD card (5.9G free): I copied some filed to the same sd card.
--------------------------------------------------------------------------------------
Buri v1.4: 
-> It only took about 20s to find the vcf file.
-> Total time is 2 min
Dolphin:  
-> It took about 40s to find the vcf file
-> Total time is 2 min 10s.

Comment 11

[renfeng.mei]

i have tried this case. The slow API is mozContact.save, it takes 150ms average per contact on 6821.

@Fabrice, do you think it is possible to improve it?

Comment 12

[[:fabrice] Fabrice Desré]

(In reply to renfeng.mei from comment #11)
> i have tried this case. The slow API is mozContact.save, it takes 150ms
> average per contact on 6821.
> 
> @Fabrice, do you think it is possible to improve it?

I don't think there is any easy gain there. ni? on people that worked on that to confirm.

Comment 13

[Gregor Wagner [:gwagner]]

It's a trade-off between lookup and save performance. We want fast lookup and create many indexes we can query for. We can optimize the save time by removing indexes but with the drawback that people can't search for middle-names, nick names, case-insensitive or local phone numbers for example.

Comment 14

[Reuben Morais [:reuben]]

Has anyone profiled where we spend those 150ms? If there's significant time spent in IPC, we could optimize some of it by figuring out a way to batch save operations.

Comment 15

[Gregor Wagner [:gwagner]]

(In reply to Reuben Morais [:reuben] from comment #14)
> Has anyone profiled where we spend those 150ms? If there's significant time
> spent in IPC, we could optimize some of it by figuring out a way to batch
> save operations.

We looked at it in other bugs and basically all of it is spent in sqlite committing memory.

Comment 16

[renfeng.mei]

(In reply to Reuben Morais [:reuben] from comment #14)
> Has anyone profiled where we spend those 150ms? If there's significant time
> spent in IPC, we could optimize some of it by figuring out a way to batch
> save operations.

I quite agree with you. Is it possible to make another thread to build indexes? to save performance, do not build indexes so immediately. but for query, we can build indexes somewhere such as when idle, or before query, etc

Comment 17

[Wayne Chang [:wchang]]

Note here - comment 0 claims the time seeking the file is long rather than the actual importing.

Given the size/content of the sd card in the description, is this really a problem?

Comment 18

[Na Hao]

(In reply to Wayne Chang [:wchang] from comment #17)
> Note here - comment 0 claims the time seeking the file is long rather than
> the actual importing.
> 
> Given the size/content of the sd card in the description, is this really a
> problem?

It's already given 16G and 7G free.

Comment 19

[BMO Automation]

Firefox OS is not being worked on