[Gallery App] Performance crawls and scans endlessly with large dataset

RESOLVED WORKSFORME

Status

RESOLVED WORKSFORME
6 years ago
5 years ago

People

(Reporter: ghtobz, Unassigned)

Tracking

({perf})

unspecified

Firefox Tracking Flags

(blocking-basecamp:-)

Details

(Whiteboard: [c=progress p= s=2013.10.25 u=] [label:gallery])

(Reporter)

Description

6 years ago
[GitHub issue by tonychung on 2012-08-11T01:20:57Z, https://github.com/mozilla-b2g/gaia/issues/3361]
I'm trying this on a Mac b2g desktop build, and have yet to try Otoro.   but with a database of 23Gb of images, sorted into top and subdirectories, my build is stuck scanning, taking nearly 30+ minutes

It's been about 30 minutes and counting.  (23Gb of files)

Screenshot: http://i.imgur.com/1Lf2k.png

no logs were found to help debug.

## Environment :
* B2G Desktop build for Mac: 8-10-2012
* did a git pull to update @davidflanagan's changes on MediaDB.js and gallery.js

## Repro :
1.launch B2G desktop build
2. have a large set of images in your mac folder ~/Pictures  (23 GB worth)
3. verify importing took a really long time.  my mac beachballed, the scanning overlay was stuck there, and i finally got back 30+ minutes later to see it working

## Expected :
* We need some way to not parse a large set of images at once.   It's not unusual for users to have 16Gb of images, and we'll need to parse those in the gallery much faster.   

## Actual :
* Performance of images loading in gallery crawls.  and this was on a mac.  i havent trie dthe otoro phone yet.
(Reporter)

Comment 1

6 years ago
[GitHub comment by jammink on 2012-08-24T15:14:08Z]
Let's talk to user research group - what is the average (and what is a reasonable amount of music, video etc) for this use-case?  How much should we support?
(Reporter)

Comment 2

6 years ago
[GitHub comment by autonome on 2012-08-24T15:15:41Z]
We should figure out the ceiling of what we can support and determine if that matches the use-case in Brazil. Blocking.
(Reporter)

Comment 3

6 years ago
[GitHub comment by davidflanagan on 2012-08-24T15:30:30Z]
@tonychung: keep that directory of photos around.  That will be a great test for improvements in scanning and metadata parsing.  I'm working on both of those things now.

I'm not too worried about this on the phone, unless Brazilians often use their phones to store wallet photos that they load on from external sources.  If someone is just taking one picture at a time with the camera, we're never going to get in the situation of having gigabytes of photos to scan (unless they take photos and never ever look at them.)  If someone loads a bunch of photos on by USB mass storage, and then tries to look, we run into a long scanning situation.  But that's probably not a typical use case, and I don't think this has to block.
(Reporter)

Comment 4

6 years ago
[GitHub comment by davidflanagan on 2012-08-24T21:37:32Z]
@tonychung I've landed #3807 and am curious to know whether it improves the scanning performance on your giant set of images. This fix should prevent it from beachballing, at least.  When you have a chance, would you try again?

I'm also unclear from your description whether the scan ever finished, or if you just gave up and quit after 30 minutes.
(Reporter)

Comment 5

6 years ago
[GitHub comment by tonychung on 2012-08-24T22:07:43Z]
Sure, I'll try this on a newer daily build that picks up #3807.   Fwiw, it does finish the scan, it just took about 40mins.  I almost gave up though, but it did finish it.
(Reporter)

Comment 6

6 years ago
[GitHub comment by autonome on 2012-09-14T22:53:05Z]
I'm going to rescind blocking+ here until we get a better test scenario. We should test on Otoro, and pick a reasonable ceiling to support - a couple of gb maybe?

Please re-nom after you've tested a more reasonable number of photos, and on Otoro specifically.

Updated

6 years ago
Component: Gaia → Gaia::Gallery

Updated

5 years ago
Keywords: perf

Updated

5 years ago
Whiteboard: [label:gallery][label:perf] → [c= p= s= u=] [label:gallery][label:perf]
David,
The performance team is triaging old bugs, and this one came up - I know you've done work on this area - is this bug a duplicate of something that was already closed, or is it still valid?
Flags: needinfo?(dflanagan)
Jon,

I'm guessing that this bug is long since resolved. But I don't know for sure.  Note that this bug was reported by someone running on a Mac with 16gb of photos in the ~/Pictures directory.  I suspect it is much better now, but have not tested.
Flags: needinfo?(dflanagan)
We think this might be fixed in master and would like some QA to confirm this bug.
Keywords: qawanted

Updated

5 years ago
See Also: → bug 898067

Comment 10

5 years ago
Still slow as of 2013.08.12 comment 7 on bug 898067.
Whiteboard: [c= p= s= u=] [label:gallery][label:perf] → [c=progress p= s= u=] [label:gallery]

Updated

5 years ago
QA Contact: gbennett

Comment 11

5 years ago
Environmental Variables:
Device: Buri 1.3 master/mc mozRIL
BuildID: 20131018040217
Gaia: 18c2ab4b7b2d4cacee90fb85e97354142b2fad4a
Gecko: 855da6d8a327
Version: 27.0a1
Firmware Version: 20131015

This issue no longer repros. GH to BZ's expected "## Expected :
* We need some way to not parse a large set of images at once.   It's not unusual for users to have 16Gb of images, and we'll need to parse those in the gallery much faster." now occurs as the images load one by one, newest to oldest, and no "loading" UI overlaps the photos. While this specific issue no longer occurs I have 2g of images on my SD card and after 1.3 hours they still all haven't loaded yet. Let me know if this is still a concern as I'll write a bug against it.
Keywords: qawanted
WFM per QA feedback in comment 11.  Thanks!
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → WORKSFORME

Updated

5 years ago
Whiteboard: [c=progress p= s= u=] [label:gallery] → [c=progress p= s=2013.10.25 u=] [label:gallery]
You need to log in before you can comment on or make changes to this bug.