796473 - [Gallery App] Performance crawls and scans endlessly with large dataset

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Gaia::Gallery, defect)

Description

[GH to BZ]

[GitHub issue by tonychung on 2012-08-11T01:20:57Z, https://github.com/mozilla-b2g/gaia/issues/3361]
I'm trying this on a Mac b2g desktop build, and have yet to try Otoro.   but with a database of 23Gb of images, sorted into top and subdirectories, my build is stuck scanning, taking nearly 30+ minutes

It's been about 30 minutes and counting.  (23Gb of files)

Screenshot: http://i.imgur.com/1Lf2k.png

no logs were found to help debug.

## Environment :
* B2G Desktop build for Mac: 8-10-2012
* did a git pull to update @davidflanagan's changes on MediaDB.js and gallery.js

## Repro :
1.launch B2G desktop build
2. have a large set of images in your mac folder ~/Pictures  (23 GB worth)
3. verify importing took a really long time.  my mac beachballed, the scanning overlay was stuck there, and i finally got back 30+ minutes later to see it working

## Expected :
* We need some way to not parse a large set of images at once.   It's not unusual for users to have 16Gb of images, and we'll need to parse those in the gallery much faster.   

## Actual :
* Performance of images loading in gallery crawls.  and this was on a mac.  i havent trie dthe otoro phone yet.

Comment 1

[GH to BZ]

[GitHub comment by jammink on 2012-08-24T15:14:08Z]
Let's talk to user research group - what is the average (and what is a reasonable amount of music, video etc) for this use-case?  How much should we support?

Comment 2

[GH to BZ]

[GitHub comment by autonome on 2012-08-24T15:15:41Z]
We should figure out the ceiling of what we can support and determine if that matches the use-case in Brazil. Blocking.

Comment 3

[GH to BZ]

[GitHub comment by davidflanagan on 2012-08-24T15:30:30Z]
@tonychung: keep that directory of photos around.  That will be a great test for improvements in scanning and metadata parsing.  I'm working on both of those things now.

I'm not too worried about this on the phone, unless Brazilians often use their phones to store wallet photos that they load on from external sources.  If someone is just taking one picture at a time with the camera, we're never going to get in the situation of having gigabytes of photos to scan (unless they take photos and never ever look at them.)  If someone loads a bunch of photos on by USB mass storage, and then tries to look, we run into a long scanning situation.  But that's probably not a typical use case, and I don't think this has to block.

Comment 4

[GH to BZ]

[GitHub comment by davidflanagan on 2012-08-24T21:37:32Z]
@tonychung I've landed #3807 and am curious to know whether it improves the scanning performance on your giant set of images. This fix should prevent it from beachballing, at least.  When you have a chance, would you try again?

I'm also unclear from your description whether the scan ever finished, or if you just gave up and quit after 30 minutes.

Comment 5

[GH to BZ]

[GitHub comment by tonychung on 2012-08-24T22:07:43Z]
Sure, I'll try this on a newer daily build that picks up #3807.   Fwiw, it does finish the scan, it just took about 40mins.  I almost gave up though, but it did finish it.

Comment 6

[GH to BZ]

[GitHub comment by autonome on 2012-09-14T22:53:05Z]
I'm going to rescind blocking+ here until we get a better test scenario. We should test on Otoro, and pick a reasonable ceiling to support - a couple of gb maybe?

Please re-nom after you've tested a more reasonable number of photos, and on Otoro specifically.

Comment 7

[Jon Hylands [:jhylands]]

David,
The performance team is triaging old bugs, and this one came up - I know you've done work on this area - is this bug a duplicate of something that was already closed, or is it still valid?

Comment 8

[David Flanagan [:djf]]

Jon,

I'm guessing that this bug is long since resolved. But I don't know for sure.  Note that this bug was reported by someone running on a Mac with 16gb of photos in the ~/Pictures directory.  I suspect it is much better now, but have not tested.

Comment 9

[Dave Huseby [:huseby]]

We think this might be fixed in master and would like some QA to confirm this bug.

Comment 10

[Mike Lee [:mlee]]

Still slow as of 2013.08.12 comment 7 on bug 898067.

Comment 11

[gbennett]

Environmental Variables:
Device: Buri 1.3 master/mc mozRIL
BuildID: 20131018040217
Gaia: 18c2ab4b7b2d4cacee90fb85e97354142b2fad4a
Gecko: 855da6d8a327
Version: 27.0a1
Firmware Version: 20131015

This issue no longer repros. GH to BZ's expected "## Expected :
* We need some way to not parse a large set of images at once.   It's not unusual for users to have 16Gb of images, and we'll need to parse those in the gallery much faster." now occurs as the images load one by one, newest to oldest, and no "loading" UI overlaps the photos. While this specific issue no longer occurs I have 2g of images on my SD card and after 1.3 hours they still all haven't loaded yet. Let me know if this is still a concern as I'll write a bug against it.

Comment 12

[Ben Kelly [:bkelly, not reviewing]]

WFM per QA feedback in comment 11.  Thanks!