760653 - Investigate ways to reduce the apk size and size on disk

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Hubert Figuiere [:hub]]

Firefox beta in Android is using too much space on the internal storage. It grew from 25 to 45MB. This is a Nexus One and it has less than 200MB of storage.

Firefox should instead use the SD card.

Comment 1

[Kevin Brosnan [Ex-Mozilla]]

Non-profile data can be moved to the sdcard by using the Android Application settings this will save around 17mb of storage. 

SDcards tend to be slower than internal memory especially on class 2 cards and devices with bad filesystems such as the Samsung Galaxy S.

That being said we are looking into using the desktop smart cache sizing code. On phones without much storage it should result in a 10mb cap on the cache size. Current cap is 25mb.

Comment 2

[Hubert Figuiere [:hub]]

I already did that. Application is 1.52MB, Data 42.64MB.

Comment 3

[Michelle Luna]

Hi,

I have a report of this on SUMO over the weekend, Nexus One, using 193Mb memory.
https://support.mozilla.org/en-US/questions/928490

-Michelle

Comment 4

[Chris Emerson]

I also have the same problem.

I've:
* Moved FF beta to the SD card
* set the disk cache size to 10MB

but the application manager still shows 1.53MB Application and 26.11MB Data.

In more detail, browsing with the shell and du, in /data/data/org.mozilla.firefox_beta/files/mozilla/blah.default, the big users are Cache (17MB - larger than the 10MB about:cache claims!), and some db files:

# du |grep -v '/.*/' | sort -n | tail
2       ./mozilla-media-cache
2       ./sessionstoreCache
2       ./webapps
8       ./extensions
142     ./minidumps
274     ./OfflineCache
496     ./startupCache
17239   ./Cache
28136   .

but a lot of space is taken in some of the .db or .sqlite files, especially browser.db:

# ls -lrS |tail -10
-rw-------    1 app_85   app_85      229944 Jun 12 18:34 addons.sqlite-journal
-rw-------    1 app_85   app_85      294912 May 19 08:52 key4.db
-rw-------    1 app_85   app_85      294912 Jun 19 15:05 cert9.db
-rw-------    1 app_85   app_85      327680 Jun 17 06:12 signons.sqlite
-rw-------    1 app_85   app_85      425984 May 21 17:00 extensions.sqlite
-rw-------    1 app_85   app_85      425984 Jun 12 18:34 addons.sqlite
-rw-------    1 app_85   app_85      524288 Jun 20 07:12 webappsstore.sqlite
-rw-------    1 app_85   app_85      557056 Jun 20 09:58 cookies.sqlite
-rw-------    1 app_85   app_85      590288 Jun 20 09:58 cookies.sqlite-wal
-rw-rw----    1 app_85   app_85     5404672 Jun 20 07:06 browser.db

And annoyingly, there isn't an option to clear the disk cache in the UI!

Comment 5

[Chris Emerson]

I've had a quick look inside the 5.4MB browser.db, and found that most of the size (4MB) comes from thumbnail BLOBs in the "images" table:

$ sqlite3 browser.db
[...]
sqlite> select sum(length(thumbnail)) from images;
3932909
sqlite> select max(length(thumbnail)) from images;
43639
sqlite> select count(*) from images where thumbnail is not null;
388

I guess the thumbnails are used for the four "top sites" presented on the home page?

Chris

Comment 6

[Mark Finkle (:mfinkle) (use needinfo?)]

Sounds like a job for bug 744961

Comment 7

[Chris Emerson]

The expiry would help, yes.  (But do those thumbnails actually need storing?)

Also, I've got 7MB in the Cache directory which about:cache doesn't account for.  There are ~1600 directories which accounts for some but not all of it.

Comment 8

[Geoff Brown [:gbrown]]

(In reply to Chris Emerson from comment #7)
> Also, I've got 7MB in the Cache directory which about:cache doesn't account
> for.  There are ~1600 directories which accounts for some but not all of it.

On my device, I also see a (smaller) difference between about:cache and du. I believe there are some problems with under-reporting cache storage use in about:cache -- see bug 224441 (yes, it's that old!)

Keep in mind that about:cache reports KiB, or units of 1024 bytes. I am not sure what Android du reports, but after some quick experiments on my device, I am now quite suspicious of du accuracy:

# ls -a -l
-rw------- app_83   app_83      24622 2012-06-20 09:37 3577Fd01
# du .
32	.
# 

!!!

Comment 9

[Chris Emerson]

Is there any reason the cache directory couldn't go on the SD card?  That would solve that one nicely for me (and I could make it much larger).

Comment 10

[Kartikaya Gupta (email:[e-mail])]

I think it's considered a security risk. See https://bugzilla.mozilla.org/show_bug.cgi?id=721084#c0

Comment 11

[Hubert Figuiere [:hub]]

Maybe that's were you start splitting the storage off.

Comment 12

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Chris Emerson from comment #9)
> Is there any reason the cache directory couldn't go on the SD card?  That
> would solve that one nicely for me (and I could make it much larger).

In addition to security, SD cards are notoriously slow. Not something we want for the cache.

Comment 13

[Chris Emerson]

The security issue is a good point.

I'm not convinced about the speed.  This is on a phone - if it's not cached it's a potentially unreliable and expensive network to fetch it again, so from that point of view I think I'd prefer a large SD card cache to a necessarily small internal one (I regularly have the low storage warning, and Firefox is 4 times the size of anything else even when moved to the SD card).

How hard is it to force the profile and/or cache to be elsewhere?  It doesn't help the average user, but ISTR it's possible to reformat the SD card with a non-FAT filesystem which supports permissions, so (I guess) the security issue goes away.

Comment 14

[u397327]

Unfortunately I can't use FireFox now. I have Sony xperia Mini PRO with 300 MB on internal storage. I moved Firefox to SDcard. Old Firefox was slow too much for using, but have own data on SD card too. New Firefox is quick but forcedly store data on internal memory. Unfortunately some apps can't be moved to SD and now I have 40 MB free in internal. But when I set Sync, and visit one page, my internal storage is 0 free. And I can't use my phone, Firefox.... 

It's pity, because I bought smart phone for using Firefox. Can You please make ability to move data on SD card, to make Firefox usable or I have to uninstall it?

Comment 15

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Please investigate the issue, especially because you want to target ARMv6 devices which likely also have lower internal storage than ARMv7 ones.

On my Nexus S with 1 GB internal storage,  Nightly uses 102 MB of internal storage (I already moved it to the SD card some months ago). There are ~100 apps installed, removing other browsers would reduce internal storage consumption by ~160 MB and move it down from 820 MB (sometimes it's higher) to 660 MB.

Clearing the cache reduces memory used by something like 25 MB, but this will be filled again soon.

Comment 16

[Mark Finkle (:mfinkle) (use needinfo?)]

Bug 784086 will also help reduce the size of browser.db

Comment 17

[Kartikaya Gupta (email:[e-mail])]

(In reply to Chris Emerson from comment #7)
> The expiry would help, yes.  (But do those thumbnails actually need storing?)
> 

Bloating the browser.db with unneeded thumbnails is something that should be fixable fairly easily, I think. I've split off bug 789923 for that.

Comment 18

[Kartikaya Gupta (email:[e-mail])]

(In reply to Chris Emerson from comment #7)
> Also, I've got 7MB in the Cache directory which about:cache doesn't account
> for.  There are ~1600 directories which accounts for some but not all of it.

I don't know if you still care about this, but if you do, can you check to see if the browser.cache.disk.smart_size.enabled pref is set to true? From some playing around with this and looking at the code, it looks like smart-sizing is allowed to clobber the manually-set pref in some cases. Try setting that pref to false, and then re-setting the browser.cache.disk.capacity pref to your desired value, and see if that fixes the problem.

Comment 19

[Kartikaya Gupta (email:[e-mail])]

Also, try going to about:cache to see what the maximum storage size for the disk cache device is. This is the value that the disk cache is *actually* capped at. (Note: you may need to reload about:cache once for data to show up).

Comment 20

[Kartikaya Gupta (email:[e-mail])]

I've filed bug 792242 and bug 792248 for these issues.

Comment 21

[Aaron Train [:aaronmt]]

* random thought, would removing migration code make any difference here? (Also, would we want to remove XUL migration code in ARMv7 too)?

Comment 22

[Richard Newman [:rnewman]]

Migration class files are about 50KB pre-dex. Every little helps, but it wouldn't be a big win.

Comment 23

[Richard Newman [:rnewman]]

Duped to fatfennec.