1115329 - [Contacts]Device fails to import contacts and go back to Home automatically, when you import a large number of contacts from memory card.

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::Contacts, defect, P2)

Description

[SandKing]

Attachment: Bug video: 1432.MP4

[1.Description]:
[Flame][v2.1&2.2][Woodduck][v2.0][Contacts]When you import a large number of contacts from memory card (Internal Storage or SD card storage), the page will auto back to Home.
See log: logcat_1432.txt
See video: 1432.mp4
Found Time:14:32

[2.Testing Steps]: 
Prism: There is a "vcf" file with 2500 contacts in Memory card(Seen attachment)
1. Launch "Contacts".
2. Tap the "gear" icon.
3. Choose "Import Contacts" ->"Memory card". 

[3.Expected Result]: 
3. All of the contacts should be imported.

[4.Actual Result]: 
3. Device fails to import contacts, and the page will back to Home automatically.

[5.Reproduction build]: 

Flame 2.1:
Gaia-Rev        17c7ad2e4919a994f0844239b483116090412dee
Gecko-Rev       https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/39dfb662c82a
Build-ID        20141223001203
Version         34.0
Device-Name     flame
FW-Release      4.4.2
FW-Incremental  eng.cltbld.20141223.035107
FW-Date         Tue Dec 23 03:51:18 EST 2014
Bootloader      L1TC00011880

Flame 2.2:
Gaia-Rev        c2da2bafd4e809317e2ca70c9bf5c11136a32818
Gecko-Rev       https://hg.mozilla.org/mozilla-central/rev/0532f2509f3f
Build-ID        20141223010202
Version         37.0a1
Device-Name     flame
FW-Release      4.4.2
FW-Incremental  eng.cltbld.20141223.043429
FW-Date         Tue Dec 23 04:34:39 EST 2014
Bootloader      L1TC00011880

woodduck 2.0 build:
Gaia-Rev        afa87cffbd3cd9e2070b26d45dd556a9324bd4d5
Gecko-Rev       911e6cd6aecf8d37d42c203e162847b78a68a8d8
Build-ID        20141224050313
Version         32.0
Device-Name     jrdhz72_w_ff
FW-Release      4.4.2
FW-Incremental  1419368730
FW-Date         Wed Dec 24 05:05:52 CST 2014

[6.Reproduction Frequency]: 
Always Recurrence,5/5
TCID: Free Test

Comment 1

[SandKing]

Attachment: Bug log: logcat_1432.txt

Comment 2

[SandKing]

http://pan.baidu.com/s/1sjFfQB7
Pass word: rgfq
If you can't reproduce this bug, please download the file(2500 contacts file) in this link to test

Comment 3

[Eric Chang [:ericcc] [:echang]]

Hi SandKing, I cannot reproduce this on my flame with V2.2, could you help to reproduce again and get adb logcat again, thanks.

Comment 4

[SandKing]

Attachment: Bug video：0521.mp4

Hi Eric,
I've reproduced this issue, but it seems this poroblem dose not always happens on Flame2.2
Occur Rate:4/5.
See log:"logcat_0521.txt"
See video:"0521.MP4"

Flame 2.2 build:
Gaia-Rev        cb1dad4881533bff9f06d47e34983c7b10c04a8c
Gecko-Rev       https://hg.mozilla.org/mozilla-central/rev/b17e7747d3fb
Build-ID        20141224010204
Version         37.0a1
Device-Name     flame
FW-Release      4.4.2
FW-Incremental  eng.cltbld.20141224.042955
FW-Date         Wed Dec 24 04:30:04 EST 2014
Bootloader      L1TC00011880

Comment 5

[SandKing]

Attachment: Bug log on Flame 2.2 :logcat_0521.txt

Comment 6

[Eric Chang [:ericcc] [:echang]]

Hi Francisco, Could you help to check this bug and the log, thanks..

Comment 7

[Mike Lien[:mlien]]

I suspect this is due to memory pressure with 2500 contacts.

Hi SandKing, do you know what is the contacts app's memory usage while this issue happen?

Comment 8

[Mike Lien[:mlien]]

(In reply to Mike Lien[:mlien] from comment #7)
> I suspect this is due to memory pressure with 2500 contacts.
> 
> Hi SandKing, do you know what is the contacts app's memory usage while this
> issue happen?

Sorry, please ignore the memory usage request.
For memory pressure suspicion, we need to see what's going on with in the dmesg log : adb shell dmesg

Could you attach the dmesg log while this problem happen?

Comment 9

[SandKing]

Attachment: Dmesg log: dmesg.txt

Hi Mike,
Please  find Dmesg log from attachment. You can refer to it.
Thanks!

Comment 10

[SandKing]

Flame 2.1 version:
Gaia-Rev        73be51f998031f06db0cd660c0e388fa621c9f4c
Gecko-Rev       https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/ea426e47bfc4
Build-ID        20141229001204
Version         34.0
Device-Name     flame
FW-Release      4.4.2
FW-Incremental  eng.cltbld.20141229.035147
FW-Date         Mon Dec 29 03:51:58 EST 2014
Bootloader      L1TC000118D0

Comment 11

[Mike Lien[:mlien]]

From dmesg, this issue is due to oom, wait for Francisco's reply

Comment 12

[Francisco Jordano [:arcturus] [:francisco]]

Will need to check with Sergi, since he created the streaming parser for vcard.

Comment 13

[Francisco Jordano [:arcturus] [:francisco]]

Seeing comments on bug 905167, perhaps we are leaking some memory in the process.

Comment 14

[SandKing]

Attachment: Bug log: logcat_0802.txt

Hi William,
This bug still exists on latest Flame 2.1&2.2&3.0
See log:"logcat_0802.txt"

Flame 2.1:(affected)
Build ID               20150327161204
Gaia Revision          6f39e4e876152de1dcdcc0e7656197f22f105e4b
Gaia Date              2015-03-25 11:16:16
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/f4d7c0e7292e
Gecko Version          34.0
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150327.193800
Firmware Date          Fri Mar 27 19:38:11 EDT 2015
Bootloader             L1TC000118D0

Flame 2.2:(affected)
Build ID               20150327162502
Gaia Revision          473cd63f53c855299b719285d9b95e3f2910782f
Gaia Date              2015-03-27 20:14:43
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/b358619def45
Gecko Version          37.0
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150327.194944
Firmware Date          Fri Mar 27 19:49:53 EDT 2015
Bootloader             L1TC000118D0

Flame 3.0:(affected)
Build ID               20150327160203
Gaia Revision          9cc496cecc37d7a29f9279827cdf6e4891211f67
Gaia Date              2015-03-27 13:55:18
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/44e454b5e93b
Gecko Version          39.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150327.192632
Firmware Date          Fri Mar 27 19:26:42 EDT 2015
Bootloader             L1TC000118D0

Comment 15

[William Hsu [:whsu]]

Thanks for your confirmation, SandKing!
Let's wait for Sergi's solution. (Bug 905167)

Comment 16

[Josh Cheng [:josh]]

[Blocking Requested - why for this release]:
Function broken and bad user experience.

Comment 17

[Francisco Jordano [:arcturus] [:francisco]]

Triage team:

Blocking on this, since the import it's killing the app and shouldn't be the case.

Comment 18

[Ethan Tseng [:ethan]]

Hi Francisco,
Should this bug block 2.5?
And is there anyone in your team could help on this bug?

Comment 19

[Francisco Jordano [:arcturus] [:francisco]]

Hi Ethan,

it's already blocking 2.5, we will try to get a look, but doesn't seem to be soon.

Comment 20

[Fernando Jiménez Moreno [:ferjm]]

(In reply to SandKing from comment #2)
> http://pan.baidu.com/s/1sjFfQB7
> Pass word: rgfq
> If you can't reproduce this bug, please download the file(2500 contacts
> file) in this link to test

I can't download anything from here. Could you uploaded somewhere else, please? (better if it's a site translated to English :)). Thanks

Comment 21

[Fernando Jiménez Moreno [:ferjm]]

s/"uploaded"/"upload it"

Comment 22

[William Hsu [:whsu]]

@ Lancy. As comment 20 mentioned, Please do something to help this problem.
Thanks!

Comment 23

[Lancy(Leave from Mozilla)]

Hi Fernando,
The test file has been uploaded to Google cloud drive, you can get the file from https://goo.gl/07D70Q 
Please contact me if you have any question. :)

Comment 24

[Fernando Jiménez Moreno [:ferjm]]

Got it. Thank you!

Comment 25

[Fernando Jiménez Moreno [:ferjm]]

I was able to reproduce this issue with a 319Mb RAM Flame. It's hard to reproduce it with a 512Mb RAM one.

I found that we are running OOM when reading the content of the vcard file via FileReader [1]. We are being killed before receiving the onload event [2]. We get several onprogress events though.

I honestly have no idea about how to solve this issue.

If I understand correctly, we are trying to load in memory the whole vcard file before processing it. In this case, that's ~22.5Mb but it could be 100Mb... I am afraid that this doesn't scale. Even with the NGA refactor that will make the Contacts app slimmer, we may found ourselves trying to load 100Mb vcard files. We need a way to read and process the file in chunks or set a limit of allowed size for these kind of files.

Jonas, I believe you designed the FileReader API. I am really not familiar with it, so this might be a bad usage of the API that I am not able to identify. Could you give us any pointer here, please? Thanks!

[1] https://mxr.mozilla.org/gaia/source/shared/js/contacts/import/utilities/sdcard.js?force=1#161
[2] https://mxr.mozilla.org/gaia/source/shared/js/contacts/import/utilities/sdcard.js?force=1#155

Comment 26

[Mahendra Potharaju [:mahe]]

Marking as P2.

Comment 27

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

There's a couple of ways that you can incrementally read a Blob.

One solution is to use Blob.slice() to create a blob that represents the first X kb of data and then read that blob using a FileReader. I.e. something like:

function readBlobIncrementally(blob, callback) {
  var startPos = 0;
  var fr = new FileReader;
  function readSection() {
    if (startPos >= blob.size) {
      callback(null);
      return;
    }
    var part = blob.slice(startPos, startPos + 4096); // This doesn't actually copy any data
    startPos += 4096;
    fr.readAsArrayBuffer(part);
    fr.onloadend = function() {
      callback(fr.error || fr.result).then(readSection);
    }
  }
}

(the callback is expected to return a promise indicating when it's prepared to receive another chunk of data).

Alternatively you can create a blob:-url and use .responseType = "moz-chunked-arraybuffer"

I.e. something like:

var xhr = new XMLHttpRequest;
var blobURL = URL.createObjectURL(blob);
xhr.open("GET", blobURL);
xhr.onprogress = function() {
  processData(xhr.result);
}
xhr.onloadend = function() {
  URL.revokeObjectURL(blobURL);
}

The downside with this approach is that you don't have control over how much data is read into memory at once. But I think that we'll generally do pretty reasonably small chunks. You also can't control how quickly data comes in, which may or may not be a problem.

Comment 28

[GitHub Autolander Bot]

Attachment: [gaia] ferjm:bug1115329.contactssdcard > mozilla-b2g:master

Comment 29

[Francisco Jordano [:arcturus] [:francisco]]

Comment on attachment 8666833 [details] [review]
[gaia] ferjm:bug1115329.contactssdcard > mozilla-b2g:master

Tested on the phone and looking good.

Just left some nits on gh.

Please address them before merging. Also could you cherry-pick the commit once landed and merge it to nga branch?

Comment 30

[Fernando Jiménez Moreno [:ferjm]]

Master: https://github.com/mozilla-b2g/gaia/commit/4bf2db336772423ea65bab48c83672de20db6de2

nga: https://github.com/mozilla-b2g/gaia/commit/4bf2db336772423ea65bab48c83672de20db6de2

Comment 31

[SandKing]

Attachment: Verify log: logcat_1954.txt

The bug still exists on the latest flame2.5 (319M).
Same STR with comment 0
Actual result:
After imported about 1000 contacts device will back to Home automatically.
See log:"logcat_1954.txt"
See video:"Flame_v2.5.3GP"
Found Time:19:54

From the beginning of the behaviour, we can find that when user imports contacts from SD card, only a few seconds later, Contacts app will exit automatically.

But now it will exit after you've imported about 1000 contacts. 

So I think this bug gets benefit from the fix. But because device's memory is too small for importing too many contacts. Thereby Contacts can not run smoothly and will exit automatically.

Flame 2.5:
Build ID               20151008150210
Gaia Revision          e698df503ff700eb5782e3d50c6eb753567d3451
Gaia Date              2015-10-08 17:26:52
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/e5f1bc63ad52d0eb86f7fb838226ca6036774660
Gecko Version          44.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20151008.182756
Firmware Date          Thu Oct  8 18:28:08 EDT 2015
Firmware Version       v18D v4
Bootloader             L1TC000118D0

Comment 32

[SandKing]

Attachment: Verify video: Flame_v2.5.3GP

Comment 33

[Max Murrell [MMurrell]]

I experienced some blocking issues/bugs while importing and exporting mass contacts to an exact number of 2,500. I was able to reproduce a similar result with a larger number on Flame 2.5. I was able to repro the same similar occurrence with 2,558 contacts on Flame 2.2. On both Flame builds the device stalled on the importing screen then returned to the previous screen (where the option to import/export is selectable) instead of returning to the home screen.  


Environmental Variables: (affected with 3,200 contacts) Stalls for close to a half hour then returns to the previous screen.
Device: Flame 2.5
BuildID: 20151006030203
Gaia: 60cdaa3d3424db3432dc903e7f9c6c8fa099c06d
Gecko: 3edc8d4a1e198314f5d7ebd2967b85842beef602
Gonk: c4779d6da0f85894b1f78f0351b43f2949e8decd
Version: 44.0a1 (2.5) 
Firmware Version: v18D


Environmental Variables: (affected with 2,500 contacts) Stalls for close to a half hour then returns to the previous screen.
Device: Flame 2.2
BuildID: 20151006032504
Gaia: 5dd95cfb9f1d6501ce0e34414596ef3dd9c2f583
Gecko: fc588eb28eab
Gonk: bd9cb3af2a0354577a6903917bc826489050b40d
Version: 37.0 (2.2) 
Firmware Version: v18D

Environmental Variables: (Not affected with 1,800 contacts)
Device: Aries 2.5
BuildID: 20151008110640
Gaia: 4973f57cd8f9a62a95f783a24eac32da2bde99fc
Gecko: a955ea9382afedd66ea0da21fcd2932465168d84
Gonk: 2916e2368074b5383c80bf5a0fba3fc83ba310bd
Version: 44.0a1 (2.5) 
Firmware Version: D5803_23.1.A.1.28_NCB.ftf

Comment 34

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

Maria, this issue still seems to be occurring on low memory Flame and showing other issues on Flame as well.  See comments 31 and 33.

Comment 35

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

This issue has caused bug 1214280 which is a smoketest blocker.

Comment 36

[Maria Angeles Oteo (:oteo)]

Setting ni to Isabel to check if it continues happening and if it's necessary to reopen it

Comment 37

[Isabel Rios[:isabel_rios]]

Hi,
Tried to reproduce this issue but seems to be working fine.
I tried 3 times with each device importing from memory card a .vcf file with 3498 contacts and the import finishes successfully.

Flame 2.5 
BuildID:20151125230113
Gaia:93b8bf2d
Platform: 44.0a2

Flame 2.6
BuildID: 20151126030209
Gaia: 86959c40
Platform: 45.0a1

Comment 38

[SandKing]

Hi Isabel,
May I know the memory of your phone which you are using to repro? I'm using Flame with 319M , and bug can be repro on the latest v2.5 and master.

Flame v2.5(319M):
Build ID               20151126113148
Gaia Revision          34ccc2c8f17b87a1fab95a4186b0019ec78c7f75
Gaia Date              2015-11-26 09:44:10
Gecko Revision         http://hg.mozilla.org/releases/mozilla-b2g44_v2_5/rev/fbaba398bd98fd1837ef2fd7c13ed8ee69640cfb
Gecko Version          44.0a2
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.worker.20151126.104320
Firmware Date          Thu Nov 26 10:43:29 UTC 2015
Firmware Version       v18D v4
Bootloader             L1TC000118D0

Flame mater(319M):
Build ID               20151126172432
Gaia Revision          86959c405348d27ba5686956ae3a8ffc274d3db8
Gaia Date              2015-11-26 06:53:43
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/74c7941a9e22d50057800771ebae07f69deecc9f
Gecko Version          45.0a1
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.worker.20151126.164303
Firmware Date          Thu Nov 26 16:43:13 UTC 2015
Firmware Version       v18D v4
Bootloader             L1TC000118D0

Comment 39

[Isabel Rios[:isabel_rios]]

Hi SandKing,

I am using Flame 512MB. As we moved to this thought it was the one to check.

Comment 40

[SandKing]

Attachment: Bug log: logcat_0725.txt

See log:"logcat_0725.txt"
Found Time: 07:25

Comment 41

[SandKing]

Accordding comment 37 and comment 39, this bug has been verified as pass.So I will change the Status to "verified".