Firefox writes 3 times as much bytes on disk compared to Chrome. Bad for SSD

NEW
Unassigned

Status

()

Firefox
General
P3
normal
2 months ago
a month ago

People

(Reporter: Massimo, Unassigned, NeedInfo)

Tracking

(Depends on: 1 bug, {perf})

57 Branch
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [qf])

Attachments

(2 attachments)

(Reporter)

Description

2 months ago
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36

Steps to reproduce:

installed Firefox, checked and compared to Chrome written bytes by opening the same amount of tabs and same websites.


Actual results:

Firefox written bytes is 2-3 times higher than Chrome. It's bad for SSD


Expected results:

less I/O by Firefox

Updated

2 months ago
Component: Untriaged → General
Product: Firefox → Core
(Reporter)

Comment 1

2 months ago
Created attachment 8930374 [details]
Screenshot_4.jpg

Comment 2

2 months ago
Was chrome a fresh profile as well?

On first install firefox won't have any http cache, site storage, etc written.  If you have been using this chrome profile for a while that would have been written on your disk long ago.  We either want to compare both fresh installs/profiles or both "warm" installs/profiles.
Flags: needinfo?(theflyingcelt)
(Reporter)

Comment 3

2 months ago
No, Chrome was installed already, and so was Firefox really. I downloaded the new Firefox 57 from the website and installed it/updated it(?). I'm not sure if that can be considered an update or a fresh installation. Sorry.
This morning I launched at the same time Chrome and Firefox and opened the same websites on both, about 6 tabs each. At the moment, after about 10 hours running, Chrome has been writing 800MB, and Firefox 2.25GB, therefore about 3 times more.
Flags: needinfo?(theflyingcelt)

Comment 4

2 months ago
Ok, thanks for clarifying.

Disk bytes written isn't currently a performance metric we track.  Maybe we should.  Marking [qf] for that.

This is probably session store which has been a problem in the past.  We improved things a lot by at least compressing session store, but we need to be smarted about how often we write the data out.  See bug 1304389.
Depends on: 1304389
Whiteboard: [qf]
(Reporter)

Comment 5

2 months ago
Sorry for confusing you.
Well yes, that's a lot of data written which is quite bad for SSDs.
Great browser apart from that.
I hope this will be fixed.
Cheers

Comment 6

2 months ago
Session store performance is being actively worked on, so I expect this will get looked at.  See bug 1330635 for the top level meta session store performance work.

Comment 7

2 months ago
Mike, do you have any more thoughts here?
Flags: needinfo?(mdeboer)

Comment 8

2 months ago
Is there actual data that the majority of Massimo's disk IO is in fact sessionstore?  (I don't see that)

Easy  to determine how much Firefox sessionstore IO is being done - look at about:sessionstore. 
  Disk writes to sessionstore.js over an 8 hour day: 8,768,228,640 bytes (9 GB)
And you can change the session save interval via browser.sessionstore.interval - I have mine set to 1 minute = 60000

Comment 9

2 months ago
(In reply to Wayne Mery (:wsmwk) from comment #8)
> Is there actual data that the majority of Massimo's disk IO is in fact
> sessionstore?  (I don't see that)

No, but its highly likely given our session store implementation rewrites the entire store on every change (or still on a timer as you suggest???).

And I don't see about:sessionstore in either FF57 or nightly 59.
(Reporter)

Comment 10

2 months ago
Sorry guys but I don't know much about how this works here. I reported the bug but I wonder what I'm supposed to expect now. Are you developers? Will I get a solution here or a fix in next release? How does it work exactly?
Thanks
(In reply to Wayne Mery (:wsmwk) from comment #8)
> Easy  to determine how much Firefox sessionstore IO is being done - look at
> about:sessionstore. 
>   Disk writes to sessionstore.js over an 8 hour day: 8,768,228,640 bytes (9
> GB)

This is a good way to know the traffic, and I believe this can only be provided by the addon[1], which don't support fx 57 yet.
I'm also looking forward the addon update :)


[1] https://addons.mozilla.org/en-US/firefox/addon/about-sessionstore/
Besides,
Without the about-sessionstore addon, we can still observe the traffic by seeing the byte written on the file 'recovery.jsonlz4'.

Comment 13

2 months ago
You can see that, and I can see that on my systemt. That was why I was asking the user to confirm the specific source of his file IO. If it is the same then this is merely a duplicate of bug 1304389
(Reporter)

Comment 14

2 months ago
I'm not an expert. Bytes written by Firefox that I can see are only from task manager in Windows. 
I'm considering giving up Firefox for the time being as it's really eating up my SSD :(

Comment 15

2 months ago
Simple steps:
1. type in about:config and ignore the warning
2. in search field paste browser.sessionstore.interval 
3. right+click that item, pick modify
4. change it to 60000 and hit OK
(Reporter)

Comment 16

2 months ago
Done. No difference.

Comment 17

2 months ago
(In reply to Massimo from comment #16)
> Done. No difference.

If I remember correctly the change takes effect immediately, and changing to 60000 would cut the IO by 75% if your issue is sessionstore IO.  But if there was not difference, then your  problem might not be sessionstore.
Hacky but effective: Find your profile folder, sort by date modified, and see what's always floating to the top.

On Mac and Linux there are loads of tools to watch exactly what an app is doing with the filesystem, but I don't know on Windows what the best tool is for that :(
(Reporter)

Comment 19

2 months ago
(In reply to Wayne Mery (:wsmwk) from comment #17)
> (In reply to Massimo from comment #16)
> > Done. No difference.
> 
> If I remember correctly the change takes effect immediately, and changing to
> 60000 would cut the IO by 75% if your issue is sessionstore IO.  But if
> there was not difference, then your  problem might not be sessionstore.

This morning I turned my computer on and opened both Firefox and Chrome, same tabs (about 10) each, same websites and let them be. After only 2 hours these are the written bytes by both browsers despite having set the sessionstore value to 60000 as suggested (see screenshot on next post if I can manage to upload it here).
(Reporter)

Comment 20

2 months ago
Sorry I don't know how to upload a screenshot with the post. Anyway, 200MB written by Chrome, and 1,4GB written by Firefox. 7 times more. Same tabs, same websites, launched at the same time 2 hours ago. :(
(In reply to Massimo from comment #14)
> I'm not an expert. Bytes written by Firefox that I can see are only from
> task manager in Windows. 

Hi Massimo,
You can use the built-in tool of windows: Resource Monitor, to see the disk activity.


[1] https://en.wikipedia.org/wiki/Resource_Monitor


====

Unset the bug dependency since it's not about the session restore.
No longer depends on: 1304389
Clear the ni to reduce the noise for Mike since it's not about the session restore.

====


(In reply to Massimo from comment #20)
> Sorry I don't know how to upload a screenshot with the post.

Massimo, you can use the 'Attach File' link on this bug page to upload :)
Flags: needinfo?(mdeboer)
(Reporter)

Comment 23

2 months ago
Created attachment 8931635 [details]
Screenshot_5.jpg

Hi Will and thanks. I opened some tabs on Firefox and took a screenshot in resources monitor once listed by written bytes (while loading pages).
Please see attached "screenshot_5.jpg" file.
Thanks
(Reporter)

Updated

2 months ago
Attachment #8930374 - Attachment description: Screenshot_5.jpg → Screenshot_4.jpg
(In reply to Massimo from comment #23)
> Hi Will and thanks. I opened some tabs on Firefox and took a screenshot in
> resources monitor once listed by written bytes (while loading pages).
> Please see attached "screenshot_5.jpg" file.

Looks like related to the places.sqlite, which is mainly for saving the bookmark and browsing history.
Using a fresh Firefox profile and see the disk activity again can clarify this.

Besides,
AFAIK browser extension could also cause issues[1], but I am not sure whether the web extension API nowadays have such ability or not?
Restarting the firefox with add-ons disabled can clarify this. (You can find this action in the menu button [2])


[1]
Bug 489173 - Latest Google Toolbar Appears to Make places.sqlite Massive
https://bugzilla.mozilla.org/show_bug.cgi?id=489173

[2] https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode
(Reporter)

Comment 25

2 months ago
(In reply to Will Wang [:WillWang] from comment #24)
> (In reply to Massimo from comment #23)
> > Hi Will and thanks. I opened some tabs on Firefox and took a screenshot in
> > resources monitor once listed by written bytes (while loading pages).
> > Please see attached "screenshot_5.jpg" file.
> 
> Looks like related to the places.sqlite, which is mainly for saving the
> bookmark and browsing history.
> Using a fresh Firefox profile and see the disk activity again can clarify
> this.
> 
> Besides,
> AFAIK browser extension could also cause issues[1], but I am not sure
> whether the web extension API nowadays have such ability or not?
> Restarting the firefox with add-ons disabled can clarify this. (You can find
> this action in the menu button [2])
> 
> 
> [1]
> Bug 489173 - Latest Google Toolbar Appears to Make places.sqlite Massive
> https://bugzilla.mozilla.org/show_bug.cgi?id=489173
> 
> [2]
> https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-
> mode

The other day I tried to disable all add-ons and restart but disk usage was quite the same. Perhaps not 5 times Chrome but x4 yeah. I don't have google toolbar installed.
I'll probably give it a try with a fresh installation of Firefox then, although I have a feeling it won't solve the issue.
Anyone else having the same problem or is it just me?

Comment 26

2 months ago
Marco, is there an easy way to see why we're doing so much writing to places.sqlite?
Flags: needinfo?(mak77)
It's unclear to me if the Chrome writes count took into account all of its processes or just one.
The numbers reported in comment 23, while not extremely nice, don't look critical by themselves, it's less than 300KB/s for the larger db we have. While there is likely a relation with the 4 times more writes, that relation is not well defined. We should run the measurement for an extended timeframe and split usage by percentages to prioritize work.

It's surely worth investigating and I agree with Ben, we should have a Talos test for the amount of I/O (but I actually think we have something already with xperf?)

(In reply to :Gijs from comment #26)
> Marco, is there an easy way to see why we're doing so much writing to
> places.sqlite?

We can check the queries traffic through MOZ_LOG=mozStorage:5 and MOZ_LOG_FILE set to a local file path. That will store any queries executed by Firefox (may grow quite a bit).
There are surely known deficiencies in the schema and the way we store things, that we are constantly working on. Also WAL gives us a little bit more speed, but may write a bit more data.

If there is no specific user interaction causing large I/O (moving bookmarks, restoring...), the things that could be running are:
1. history additions
2. additional data stored for Activity Stream (that other browsers may not store at all)
3. if the db reached its size limit (about 70MB) expiration may be running
4. If Sync is enabled, it can modify both history and bookmarks
5. Add-ons can clearly modify history and bookmarks
6. frecency and input history decay every day, on idle-daily IIRC (one-shot)
7. bookmarks backups happen almost every day on idle (one-shot)
As a personal suggestion, I'd tell to not worry too much about SSD tearing in itself, the current units can easily last far more time than your computer average life (10 years and more) even at large write ratios.
Regardless of that, we'll work on improving I/O, because many users are still on mechanical disks that are MUCH slower.
Joel, how is our test situation regarding I/O measurement?

David, I don't know if you're the best person to ask to, I'm looking for someone who could organize a small investigation team to evaluate I/O and identify some bottlenecks to be filed in the right components
Status: UNCONFIRMED → NEW
Component: General → General
Ever confirmed: true
Flags: needinfo?(mak77)
Flags: needinfo?(jmaher)
Flags: needinfo?(ddurst)
Keywords: perf
Product: Core → Firefox

Comment 30

2 months ago
Is context graph or activity stream hitting places harder than we used to?

Comment 31

2 months ago
I would also note that this problem would be much worse on mobile.  Those devices have less performant flash and some manufacturers don't use flash friendly file systems.

Updated

2 months ago
Depends on: 1420571
(In reply to Ben Kelly [:bkelly] from comment #30)
> Is context graph or activity stream hitting places harder than we used to?

For writes I just filed one bug related to the new AS code. Surely storing the page description for every page, rather than just bookmarks, increases our writes. We could evaluate reducing the current description limit from 1024 chars to 512? Mardak?
Most of the new traffic should be reads btw.
Flags: needinfo?(edilee)

Updated

2 months ago
Depends on: 1420597

Comment 33

2 months ago
At least currently, activity stream is able to show up to 3 lines of description text. Looks like 256 characters "should be enough for everybody". ? nanj
Flags: needinfo?(edilee) → needinfo?(najiang)
for measuring IO we have the xperf job which runs on windows 7 only- this has been running for many years and measures main thread, non main thread, startup, and everything else.  While this isn't linux or android, it should give us a good idea of how much IO we are doing- There is minimal information in the logs, but if there are specific questions, I can usually hack something up and get a try push to get more specifics without too much trouble.
Flags: needinfo?(jmaher)
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #35)
> for measuring IO we have the xperf job which runs on windows 7 only- this
> has been running for many years and measures main thread, non main thread,
> startup, and everything else.

From the description here https://wiki.mozilla.org/Buildbot/Talos/Tests#xperf it seems to only measure files used on startup, not during a common browsing session. It would be extremely valuable to measure a whole session that maybe also includes some specific copies of top N websites. Is that right?
Flags: needinfo?(jmaher)
Isn't the resource-usage.json which can be found as artifact on any test job a valuable resource to identify disk I/O usages? Or what kind of usage do we record there?

Here an example:
https://public-artifacts.taskcluster.net/ScW9FpExRWKVF-AjcghdEw/0/public/test_info//resource-usage.json
we actually measure both startup and normal operations.  Take a recent job for example:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=cad9c9573579698c223b4b6cb53ca723cd930ad2&filter-searchStr=xperf&selectedJob=147795786

we run the tp5 pageset and collect the IO.  you can see:
main_startup_fileio
nonmain_startup_fileio
main_normal_fileio
nonmain_normal_fileio

we also connect netio as well :)
Flags: needinfo?(jmaher)
(In reply to Ben Kelly [:bkelly] from comment #31)
> I would also note that this problem would be much worse on mobile.  Those
> devices have less performant flash and some manufacturers don't use flash
> friendly file systems.

Fortunately we don't use any of the same code on iOS, or any desktop code for bookmarks or history on Android, and neither platform does significant work when not foregrounded.

Updated

2 months ago
Depends on: 1421562
No longer depends on: 1420571, 1420597

Comment 40

2 months ago
https://bugzilla.mozilla.org/show_bug.cgi?id=814942#c8

 User image Nicholas Nethercote [:njn] 2012-11-30 04:17:05 IST

"I/O Read Bytes" means "The number of read input/output operations generated by the process, including file, network, and device I/Os", according to http://windows.microsoft.com/en-US/windows-vista/What-do-the-Task-Manager-memory-columns-mean.


So this may not be actual disk reads.
Priority: -- → P3
You need to log in before you can comment on or make changes to this bug.