854863 - 20MB regression in application 'Data' size after first startup

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: General, defect)

Description

[Peter Retzer (:pretzer)]

I noticed a regression between FF20 (Beta) and FF21 (Aurora) regarding the size of the 'Data' Firefox generates after first startup, which is visible via the 'Application manager' in the Android settings.

STR:
1) Go to Android settings --> Application manager
2) Select Firefox Beta
3) Select 'Clear data'
4) Start Firefox Beta
5) Go back to the Application manager and check the value under 'Data' (Megabyte)
6) Do the steps above for Aurora as well and compare the values

These are the values I get on my device (Galaxy S2, Android 4.1.2):
Firefox 20 (Beta) --> 1.86 MB
Firefox 21 (Aurora) --> 21.66 MB
Firefox 22 (Nightly) --> 21.64 MB

This suggests an approximate regression of 20MB between FF20 and FF21. I'm not sure if this data is intended/needed or if it is just a bug, but considering that Firefox is increasingly extending its focus to lower end phones this seems like a pretty big deal.

Comment 1

[Aaron Train [:aaronmt]]

On my LGE Nexus 4 (Android 4.2.2)

* Firefox 20 (Beta) → 1.78 MB
* Firefox 21 (Aurora) → 9.66MB
* Firefox 22 (Nighlty) → 9.54MB

Comment 2

[Aaron Train [:aaronmt]]

Pulling the profiles of Nightly (http://cl.ly/image/1J2R2L0G0s2x) against Beta (http://cl.ly/image/2l0I0C0W0p1a), I only see a difference in determined cache size:

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Looks like maybe it's safebrowsing related? CC'ing gcp

Comment 4

[Geoff Brown [:gbrown]]

Keep in mind that the Cache size depends on the amount of space available on the device ("smart sizing"). But that behavior should be the same, for a given device, since 14 (see bug 745340).

Comment 5

[Geoff Brown [:gbrown]]

Actually, cache smart sizing determines the maximum size of the disk cache -- but these measurements are taken before any significant page loads, so I would expect the cache to be mostly empty, and its size dominated by the pre-allocated block files, _CACHE_001_, etc. Maybe the size of the block files changed between 20 and 21?

Comment 6

[Gian-Carlo Pascutto [:gcp]]

>Looks like maybe it's safebrowsing related?

The screenshots AaronMT posted clearly show that that is not the case...

Comment 7

[Geoff Brown [:gbrown]]

There is a change in the disk cache block files found in <profile>/Cache.

On Firefox 20, initial block files look like this:

-rw------- app_139  app_139     16538 2013-03-26 09:30 _CACHE_001_
-rw------- app_139  app_139      4096 2013-03-26 09:30 _CACHE_002_
-rw------- app_139  app_139      5978 2013-03-26 09:30 _CACHE_003_
-rw------- app_139  app_139       276 2013-03-26 09:30 _CACHE_MAP_

On Firefox 22, initial block files look like this:

-rw------- app_123  app_123   4194304 2013-03-26 09:30 _CACHE_001_
-rw------- app_123  app_123   4194304 2013-03-26 09:30 _CACHE_002_
-rw------- app_123  app_123   4194304 2013-03-26 09:30 _CACHE_003_
-rw------- app_123  app_123      8468 2013-03-26 09:30 _CACHE_MAP_

I'm pretty sure this is intentional, but I haven't found a bug or changeset.

Comment 8

[Aaron Train [:aaronmt]]

Re comment #7, that matches what I'm seeing in my pulled profiles too

Comment 9

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #7)
> I'm pretty sure this is intentional, but I haven't found a bug or changeset.

See https://bugzilla.mozilla.org/show_bug.cgi?id=816642#c5

Comment 10

[Peter Retzer (:pretzer)]

So is there anything we can do about this or do we have to live with it? Did anyone contact the Snappy/Performance team or the assignee of bug 816642 about this?

Comment 11

[Kartikaya Gupta (email:[e-mail])]

CC'ing :michal and :taras. Michal, I'd like to understand better how bug 816642 affects Fennec. It looks like based on bug 816642 comment 0 slow disk performance on Android was one of the causes the patch was landed in the first place, but is there some heuristic that limits how much of the available disk space it uses up? Eating up 20 MB on lower end phones seems significant.

As an aside, it looks like bug 816642 never actually got an r+ on the bug but landed with r=jduell. Please make sure all reviews are recorded on the bug, even if they happen elsewhere like on IRC.

Comment 12

[Brad Lassey [:blassey] (use needinfo?)]

Doug, assigning to you to get an assignee

Comment 13

[Michal Novotny [:michal]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #11)
> CC'ing :michal and :taras. Michal, I'd like to understand better how bug
> 816642 affects Fennec. It looks like based on bug 816642 comment 0 slow disk
> performance on Android was one of the causes the patch was landed in the
> first place, but is there some heuristic that limits how much of the
> available disk space it uses up? Eating up 20 MB on lower end phones seems
> significant.

Preallocation of the cache blockfiles eats only 12MB. Given that minimum cache size on Android is 10MB regardless the available space, I think this is OK.


> As an aside, it looks like bug 816642 never actually got an r+ on the bug
> but landed with r=jduell. Please make sure all reviews are recorded on the
> bug, even if they happen elsewhere like on IRC.

That's not true. Patch v1 (#687065) has r+ from Jason. The final patch differs only in that fallocate() is disabled on MacOSX due to Tp5 regression. Such change doesn't need a new review.

Comment 14

[Kartikaya Gupta (email:[e-mail])]

(In reply to Michal Novotny (:michal) from comment #13)
> Preallocation of the cache blockfiles eats only 12MB. Given that minimum
> cache size on Android is 10MB regardless the available space, I think this
> is OK.
> 

If I'm understanding correctly, you're saying that the cache blockfiles should take up 12MB, and the cache will live inside this space. So the startup disk-usage regression should be 12MB, and as the cache fills up, the "regression" should shrink because the used blockfile space will be similar to the space used on disk without the blockfiles. Am I understanding that correctly? If so, why is there a 20MB regression noted in comment #0? I would expect that to be only 12MB.

> That's not true. Patch v1 (#687065) has r+ from Jason. The final patch
> differs only in that fallocate() is disabled on MacOSX due to Tp5
> regression. Such change doesn't need a new review.

Ok, but please still mark the new attachment as having an r+ carried over from the previous patch for clarity.

Comment 15

[Brad Lassey [:blassey] (use needinfo?)]

Doug, still looking for an assignee

Comment 16

[(dormant account)]

Summary: the test is flawed.

preallocation: 
In prior releases we sparsely allocated _CACHE_* block files which used less space for a particular run with a new profile. However they would eventually fill up on real clients anyway. The difference is that now the files are packed together, so you don't get really shitty IO patterns on read/write(eg less slow).

I expect that this will be fixed in the new cache design once we get rid of block files.

Comment 17

[Geoff Brown [:gbrown]]

I don't disagree, but urge us to keep in mind that a certain class of user will still be alarmed/disappointed, taking the perspective "I just installed this browser and it's using X MB of data even before I have done anything!"

Comment 18

[Doug Turner (:dougt)]

Geoff, your team needs to make this trade off.  Please let us know what you want to do.

Comment 19

[Kartikaya Gupta (email:[e-mail])]

(In reply to Taras Glek (:taras) from comment #16)
> Summary: the test is flawed.
> 

Taras clarified on IRC that the "test" he is referring to is "starting the browser for the first time and checking the disk usage". I agree that this space would fill up anyway after some usage, so I don't have a problem with it getting pre-allocated to improve performance. However my question in comment 14 (why 20 MB instead of 12 MB?) is still unanswered. I'm ok with leaving the behaviour as-is as long as we can account for all of the space being pre-allocated. (gbrown also pointed out that in comment #1 Aaron saw a ~8MB regression rather than the ~20MB regression that Peter reported, so maybe this depends on device?)

Comment 20

[Geoff Brown [:gbrown]]

(In reply to Kartikaya Gupta (email:[e-mail]
> However my question in
> comment 14 (why 20 MB instead of 12 MB?) is still unanswered. I'm ok with
> leaving the behaviour as-is as long as we can account for all of the space
> being pre-allocated. (gbrown also pointed out that in comment #1 Aaron saw a
> ~8MB regression rather than the ~20MB regression that Peter reported, so
> maybe this depends on device?)

There are other differences in data files and file sizes between 20 and 21. The most notable is res/fonts, empty in 20 but populated in 21:

shell@android:/data/data/org.mozilla.firefox_beta # ls -l res/fonts
-rw-rw-r-- app_139  app_139   1672660 2013-04-15 16:11 CharisSILCompact-B.ttf
-rw-rw-r-- app_139  app_139   1664448 2013-04-15 16:11 CharisSILCompact-BI.ttf
-rw-rw-r-- app_139  app_139   1690628 2013-04-15 16:11 CharisSILCompact-I.ttf
-rw-rw-r-- app_139  app_139   1724344 2013-04-15 16:11 CharisSILCompact-R.ttf
-rw-rw-r-- app_139  app_139    224592 2013-04-15 16:11 OpenSans-Bold.ttf
-rw-rw-r-- app_139  app_139    213292 2013-04-15 16:11 OpenSans-BoldItalic.ttf
-rw-rw-r-- app_139  app_139    212896 2013-04-15 16:11 OpenSans-Italic.ttf
-rw-rw-r-- app_139  app_139    222412 2013-04-15 16:11 OpenSans-Light.ttf
-rw-rw-r-- app_139  app_139    213128 2013-04-15 16:11 OpenSans-LightItalic.ttf
-rw-rw-r-- app_139  app_139    217360 2013-04-15 16:11 OpenSans-Regular.ttf

The total size of those files is 8055760 bytes. See bug 831354.

There are additionally about 2 or 3 MB of other files in a typical Firefox 21 profile.

I think it makes sense for Android to report 21 MB to 23 MB of data usage for Firefox 21+: 12 MB cache, 8 MB fonts, 2 or 3 MB other.

If I delete the cache files, Android reports 9 to 10 MB of data. This likely accounts for Comment 1: If :aaronmt's cache was not allocated yet, he would have seen numbers like that. It seems like the cache is sometimes allocated immediately on first run, and other times, it doesn't happen until you visit a page.

I am satisfied that the increase in data usage is by design and bounded. I don't have any outstanding concerns.


Closing as WontFix to emphasize that there was a 20 MB regression here but we are willing to live with that given the additional functionality and performance.