User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36 Steps to reproduce: installed Firefox, checked and compared to Chrome written bytes by opening the same amount of tabs and same websites. Actual results: Firefox written bytes is 2-3 times higher than Chrome. It's bad for SSD Expected results: less I/O by Firefox
Was chrome a fresh profile as well? On first install firefox won't have any http cache, site storage, etc written. If you have been using this chrome profile for a while that would have been written on your disk long ago. We either want to compare both fresh installs/profiles or both "warm" installs/profiles.
No, Chrome was installed already, and so was Firefox really. I downloaded the new Firefox 57 from the website and installed it/updated it(?). I'm not sure if that can be considered an update or a fresh installation. Sorry. This morning I launched at the same time Chrome and Firefox and opened the same websites on both, about 6 tabs each. At the moment, after about 10 hours running, Chrome has been writing 800MB, and Firefox 2.25GB, therefore about 3 times more.
Ok, thanks for clarifying. Disk bytes written isn't currently a performance metric we track. Maybe we should. Marking [qf] for that. This is probably session store which has been a problem in the past. We improved things a lot by at least compressing session store, but we need to be smarted about how often we write the data out. See bug 1304389.
Depends on: 1304389
Sorry for confusing you. Well yes, that's a lot of data written which is quite bad for SSDs. Great browser apart from that. I hope this will be fixed. Cheers
Session store performance is being actively worked on, so I expect this will get looked at. See bug 1330635 for the top level meta session store performance work.
Mike, do you have any more thoughts here?
Is there actual data that the majority of Massimo's disk IO is in fact sessionstore? (I don't see that) Easy to determine how much Firefox sessionstore IO is being done - look at about:sessionstore. Disk writes to sessionstore.js over an 8 hour day: 8,768,228,640 bytes (9 GB) And you can change the session save interval via browser.sessionstore.interval - I have mine set to 1 minute = 60000
(In reply to Wayne Mery (:wsmwk) from comment #8) > Is there actual data that the majority of Massimo's disk IO is in fact > sessionstore? (I don't see that) No, but its highly likely given our session store implementation rewrites the entire store on every change (or still on a timer as you suggest???). And I don't see about:sessionstore in either FF57 or nightly 59.
Sorry guys but I don't know much about how this works here. I reported the bug but I wonder what I'm supposed to expect now. Are you developers? Will I get a solution here or a fix in next release? How does it work exactly? Thanks
(In reply to Wayne Mery (:wsmwk) from comment #8) > Easy to determine how much Firefox sessionstore IO is being done - look at > about:sessionstore. > Disk writes to sessionstore.js over an 8 hour day: 8,768,228,640 bytes (9 > GB) This is a good way to know the traffic, and I believe this can only be provided by the addon, which don't support fx 57 yet. I'm also looking forward the addon update :)  https://addons.mozilla.org/en-US/firefox/addon/about-sessionstore/
Besides, Without the about-sessionstore addon, we can still observe the traffic by seeing the byte written on the file 'recovery.jsonlz4'.
You can see that, and I can see that on my systemt. That was why I was asking the user to confirm the specific source of his file IO. If it is the same then this is merely a duplicate of bug 1304389
I'm not an expert. Bytes written by Firefox that I can see are only from task manager in Windows. I'm considering giving up Firefox for the time being as it's really eating up my SSD :(
Simple steps: 1. type in about:config and ignore the warning 2. in search field paste browser.sessionstore.interval 3. right+click that item, pick modify 4. change it to 60000 and hit OK
Done. No difference.
(In reply to Massimo from comment #16) > Done. No difference. If I remember correctly the change takes effect immediately, and changing to 60000 would cut the IO by 75% if your issue is sessionstore IO. But if there was not difference, then your problem might not be sessionstore.
Hacky but effective: Find your profile folder, sort by date modified, and see what's always floating to the top. On Mac and Linux there are loads of tools to watch exactly what an app is doing with the filesystem, but I don't know on Windows what the best tool is for that :(
(In reply to Wayne Mery (:wsmwk) from comment #17) > (In reply to Massimo from comment #16) > > Done. No difference. > > If I remember correctly the change takes effect immediately, and changing to > 60000 would cut the IO by 75% if your issue is sessionstore IO. But if > there was not difference, then your problem might not be sessionstore. This morning I turned my computer on and opened both Firefox and Chrome, same tabs (about 10) each, same websites and let them be. After only 2 hours these are the written bytes by both browsers despite having set the sessionstore value to 60000 as suggested (see screenshot on next post if I can manage to upload it here).
Sorry I don't know how to upload a screenshot with the post. Anyway, 200MB written by Chrome, and 1,4GB written by Firefox. 7 times more. Same tabs, same websites, launched at the same time 2 hours ago. :(
(In reply to Massimo from comment #14) > I'm not an expert. Bytes written by Firefox that I can see are only from > task manager in Windows. Hi Massimo, You can use the built-in tool of windows: Resource Monitor, to see the disk activity.  https://en.wikipedia.org/wiki/Resource_Monitor ==== Unset the bug dependency since it's not about the session restore.
No longer depends on: 1304389
Clear the ni to reduce the noise for Mike since it's not about the session restore. ==== (In reply to Massimo from comment #20) > Sorry I don't know how to upload a screenshot with the post. Massimo, you can use the 'Attach File' link on this bug page to upload :)
Created attachment 8931635 [details] Screenshot_5.jpg Hi Will and thanks. I opened some tabs on Firefox and took a screenshot in resources monitor once listed by written bytes (while loading pages). Please see attached "screenshot_5.jpg" file. Thanks
Attachment #8930374 - Attachment description: Screenshot_5.jpg → Screenshot_4.jpg
(In reply to Massimo from comment #23) > Hi Will and thanks. I opened some tabs on Firefox and took a screenshot in > resources monitor once listed by written bytes (while loading pages). > Please see attached "screenshot_5.jpg" file. Looks like related to the places.sqlite, which is mainly for saving the bookmark and browsing history. Using a fresh Firefox profile and see the disk activity again can clarify this. Besides, AFAIK browser extension could also cause issues, but I am not sure whether the web extension API nowadays have such ability or not? Restarting the firefox with add-ons disabled can clarify this. (You can find this action in the menu button )  Bug 489173 - Latest Google Toolbar Appears to Make places.sqlite Massive https://bugzilla.mozilla.org/show_bug.cgi?id=489173  https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode
(In reply to Will Wang [:WillWang] from comment #24) > (In reply to Massimo from comment #23) > > Hi Will and thanks. I opened some tabs on Firefox and took a screenshot in > > resources monitor once listed by written bytes (while loading pages). > > Please see attached "screenshot_5.jpg" file. > > Looks like related to the places.sqlite, which is mainly for saving the > bookmark and browsing history. > Using a fresh Firefox profile and see the disk activity again can clarify > this. > > Besides, > AFAIK browser extension could also cause issues, but I am not sure > whether the web extension API nowadays have such ability or not? > Restarting the firefox with add-ons disabled can clarify this. (You can find > this action in the menu button ) > > >  > Bug 489173 - Latest Google Toolbar Appears to Make places.sqlite Massive > https://bugzilla.mozilla.org/show_bug.cgi?id=489173 > >  > https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe- > mode The other day I tried to disable all add-ons and restart but disk usage was quite the same. Perhaps not 5 times Chrome but x4 yeah. I don't have google toolbar installed. I'll probably give it a try with a fresh installation of Firefox then, although I have a feeling it won't solve the issue. Anyone else having the same problem or is it just me?
Marco, is there an easy way to see why we're doing so much writing to places.sqlite?
It's unclear to me if the Chrome writes count took into account all of its processes or just one. The numbers reported in comment 23, while not extremely nice, don't look critical by themselves, it's less than 300KB/s for the larger db we have. While there is likely a relation with the 4 times more writes, that relation is not well defined. We should run the measurement for an extended timeframe and split usage by percentages to prioritize work. It's surely worth investigating and I agree with Ben, we should have a Talos test for the amount of I/O (but I actually think we have something already with xperf?) (In reply to :Gijs from comment #26) > Marco, is there an easy way to see why we're doing so much writing to > places.sqlite? We can check the queries traffic through MOZ_LOG=mozStorage:5 and MOZ_LOG_FILE set to a local file path. That will store any queries executed by Firefox (may grow quite a bit). There are surely known deficiencies in the schema and the way we store things, that we are constantly working on. Also WAL gives us a little bit more speed, but may write a bit more data. If there is no specific user interaction causing large I/O (moving bookmarks, restoring...), the things that could be running are: 1. history additions 2. additional data stored for Activity Stream (that other browsers may not store at all) 3. if the db reached its size limit (about 70MB) expiration may be running 4. If Sync is enabled, it can modify both history and bookmarks 5. Add-ons can clearly modify history and bookmarks 6. frecency and input history decay every day, on idle-daily IIRC (one-shot) 7. bookmarks backups happen almost every day on idle (one-shot)
As a personal suggestion, I'd tell to not worry too much about SSD tearing in itself, the current units can easily last far more time than your computer average life (10 years and more) even at large write ratios. Regardless of that, we'll work on improving I/O, because many users are still on mechanical disks that are MUCH slower.
Joel, how is our test situation regarding I/O measurement? David, I don't know if you're the best person to ask to, I'm looking for someone who could organize a small investigation team to evaluate I/O and identify some bottlenecks to be filed in the right components
Status: UNCONFIRMED → NEW
Component: General → General
Ever confirmed: true
Product: Core → Firefox
Is context graph or activity stream hitting places harder than we used to?
I would also note that this problem would be much worse on mobile. Those devices have less performant flash and some manufacturers don't use flash friendly file systems.
(In reply to Ben Kelly [:bkelly] from comment #30) > Is context graph or activity stream hitting places harder than we used to? For writes I just filed one bug related to the new AS code. Surely storing the page description for every page, rather than just bookmarks, increases our writes. We could evaluate reducing the current description limit from 1024 chars to 512? Mardak? Most of the new traffic should be reads btw.
At least currently, activity stream is able to show up to 3 lines of description text. Looks like 256 characters "should be enough for everybody". ? nanj
Flags: needinfo?(edilee) → needinfo?(najiang)
Yes, 256 should be good enough. Also, we do have some telemetry on the page metadata (preview image link + description) here. https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2017-11-24&keys=__none__!__none__!__none__&max_channel_version=nightly%252F59&measure=PAGE_METADATA_SIZE&min_channel_version=null&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2017-11-13&table=0&trim=1&use_submission_date=0
for measuring IO we have the xperf job which runs on windows 7 only- this has been running for many years and measures main thread, non main thread, startup, and everything else. While this isn't linux or android, it should give us a good idea of how much IO we are doing- There is minimal information in the logs, but if there are specific questions, I can usually hack something up and get a try push to get more specifics without too much trouble.
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #35) > for measuring IO we have the xperf job which runs on windows 7 only- this > has been running for many years and measures main thread, non main thread, > startup, and everything else. From the description here https://wiki.mozilla.org/Buildbot/Talos/Tests#xperf it seems to only measure files used on startup, not during a common browsing session. It would be extremely valuable to measure a whole session that maybe also includes some specific copies of top N websites. Is that right?
Isn't the resource-usage.json which can be found as artifact on any test job a valuable resource to identify disk I/O usages? Or what kind of usage do we record there? Here an example: https://public-artifacts.taskcluster.net/ScW9FpExRWKVF-AjcghdEw/0/public/test_info//resource-usage.json
we actually measure both startup and normal operations. Take a recent job for example: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=cad9c9573579698c223b4b6cb53ca723cd930ad2&filter-searchStr=xperf&selectedJob=147795786 we run the tp5 pageset and collect the IO. you can see: main_startup_fileio nonmain_startup_fileio main_normal_fileio nonmain_normal_fileio we also connect netio as well :)
(In reply to Ben Kelly [:bkelly] from comment #31) > I would also note that this problem would be much worse on mobile. Those > devices have less performant flash and some manufacturers don't use flash > friendly file systems. Fortunately we don't use any of the same code on iOS, or any desktop code for bookmarks or history on Android, and neither platform does significant work when not foregrounded.
https://bugzilla.mozilla.org/show_bug.cgi?id=814942#c8 User image Nicholas Nethercote [:njn] 2012-11-30 04:17:05 IST "I/O Read Bytes" means "The number of read input/output operations generated by the process, including file, network, and device I/Os", according to http://windows.microsoft.com/en-US/windows-vista/What-do-the-Task-Manager-memory-columns-mean. So this may not be actual disk reads.
You need to log in before you can comment on or make changes to this bug.