Reduce fsyncs and writes in Places

RESOLVED FIXED

Status

()

Firefox
Bookmarks & History
P1
critical
RESOLVED FIXED
9 years ago
7 years ago

People

(Reporter: dietrich, Unassigned)

Tracking

({meta, perf})

Trunk
meta, perf
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [tsnap], URL)

Attachments

(1 attachment)

(Reporter)

Description

9 years ago
See bug 421482 for background. For discussion of the various approaches to resolving the issue see http://wiki.mozilla.org/Places:FsyncApproach.
(Reporter)

Updated

9 years ago
Flags: blocking-firefox3.1?

Updated

9 years ago
Keywords: meta, perf
Summary: Reduce fsyncs in Places → Reduce fsyncs and writes in Places
(Reporter)

Comment 1

9 years ago
Created attachment 327647 [details]
testing duration

Updated

9 years ago
Attachment #327647 - Attachment mime type: text/x-c → text/plain
(Reporter)

Updated

9 years ago
Priority: -- → P1
(Reporter)

Updated

9 years ago
Depends on: 449086

Updated

9 years ago
No longer depends on: 449086

Updated

9 years ago
Depends on: 450290
(Reporter)

Updated

9 years ago
Blocks: 437244
You can find some experimental builds here:
https://build.mozilla.org/tryserver-builds/2008-08-27_15:17-sdwilsh@shawnwilsher.com-try-a8c1213b615/

BIG-LETTERED-WARNING:
Do not use your normal profile - create a copy of it.  This modifies the places database schema, and downgrading will causes some strange behavior.
Newer builds with perf optimizations.  Same warning as last time:
https://build.mozilla.org/tryserver-builds/2008-08-28_11:12-sdwilsh@shawnwilsher.com-try-df9b4f955b9/
(Reporter)

Updated

9 years ago
Blocks: 443328
I've packaged it up with our standard Firefox Portable launcher and installer for testing purposes and posted it to the PortableApps.com Forums here:
http://portableapps.com/node/15474

This build is based on the 2008-08-28 build by Shawn Wilsher above.
My own experience and those of the other PortableApps.com users is that this definitely seems to improve the speed of browsing between pages when used on an  average speed flash drive.  While slower than 2.0.0.16 (as expected), it is much more usable than the current 3.0.1 release.

Updated

9 years ago
Duplicate of this bug: 415315
(Reporter)

Updated

9 years ago
Depends on: 337512
New builds here:
https://build.mozilla.org/tryserver-builds/2008-09-11_18:15-sdwilsh@shawnwilsher.com-try-15e9a9e910b/

Last build had a nasty bug that totally resulted in database corruption.  Same warning applies to this build though.  More testing on it would be greatly appreciated.

Updated

9 years ago
Duplicate of this bug: 455253

Comment 9

9 years ago
This bug could be related to Bug 453704 - It would make sense, given LTSP is a multi-user environment, all utilizing a single computer, with a single ext3 filesystem, which would definitely be a bottleneck when it comes to excessive fsync'ing. I would imagine that fsync would be even harder on the server while CREATING the firefox profile/places.sqlite, which is precisely where most of the issues are cropping up (new users launching Firefox taking 5-10min to launch, entire LTSP network becomes unresponsive, little/no error messages, "Firefox is already running" possibly after they try to launch again)....This is a particular mess for LTSP users. :(

Comment 10

9 years ago
Please see Bug 453704 for an update on my troubleshooting (Comment #8). Much potentially useful information.

Comment 11

9 years ago
Maybe this is a silly question, but can an fsync on/off system be adopted?

For example, during startup, if anything is dead might as well start over.  So, in the beginning of the entire thing:

moz_places_disable_fsync();

And then, at the end:

moz_places_enable_fsync();

This would be your traditional increment/decrement system, and the enable would run an immediate fsync if it hit 0, e.g just like reference counting, save/restore with svgs, etc.

Then, when deleting a batch of history items (e.g. 1000), which I'm assuming currently causes many fsyncs, do the same thing.  No need to fsync each time, we haven't done that for years as programmers.

That said, I don't mean to patronize.  I realize this is a very old technique, and given that it's not being used, there may be a good reason I don't know of.

Thanks,
-[Unknown]
(In reply to comment #10)
> Please see Bug 453704 for an update on my troubleshooting (Comment #8). Much
> potentially useful information.

Did you tried/Can you try with the build in comment 7? even if i doubt it will make better timings on startup since there we are mostly creating bookmarks

a network system like that is most likely to suffer the fsync problem

Comment 13

9 years ago
(In reply to comment #12)
> Did you tried/Can you try with the build in comment 7? even if i doubt it will
> make better timings on startup since there we are mostly creating bookmarks

I can try, but if you think it's not going to help, even just with the profile creation, it's going to be pretty useless - this is a BIG ISSUE with LTSP networks (very common in schools, computer labs, Internet cafes, etc) and really should be rectified apart from simply trying to 'trim down' fsyncs - as in my situation, it really won't matter since 35 users are launching Firefox (some for the first time, creating their profiles) simultaneously when a class comes in to do classwork.
 
> a network system like that is most likely to suffer the fsync problem

I couldn't agree more, just based on my experience with my 7 school labs. They are completely unusable, since Firefox/web browsing is really the primary application students use to do research, homework, and even their typing tutor program (TypingMaster) is done through a web browser.

Is there anything we can do to rectify this situation, apart from testing the build in #7? I'm definitely not the only one running LTSP setups - they are worldwide, and I'm sure most others are experiencing this problem as well - just aren't speaking up.
(In reply to comment #13)
> I can try, but if you think it's not going to help, even just with the profile
> creation, it's going to be pretty useless - this is a BIG ISSUE with LTSP
> networks (very common in schools, computer labs, Internet cafes, etc) and
> really should be rectified apart from simply trying to 'trim down' fsyncs - as
> in my situation, it really won't matter since 35 users are launching Firefox
> (some for the first time, creating their profiles) simultaneously when a class
> comes in to do classwork.
If it is profiles being migrated from firefox 2, then it should help.  If it's new profiles, then your assessment is correct.

Comment 15

9 years ago
(In reply to comment #14)
> If it is profiles being migrated from firefox 2, then it should help.  If it's
> new profiles, then your assessment is correct.

It's a mix - some are new school setups for this year, some are migrated/upgraded from Ubuntu 7.10 -> 8.04 LTS. There is always a mix of upgrades/new installs for everyone, I'm sure.
New build on the try server should be a bit faster (might not be noticeable).  Includes the fix for bug 456029:
https://build.mozilla.org/tryserver-builds/2008-09-24_18:56-sdwilsh@shawnwilsher.com-try-914d9222505/

Comment 17

9 years ago
I've been using your build, Shawn, and it was working great for a while, but recently it seems like it's worse than ever, prolonged writing to the disk even when I'm typing in text boxes. I'm not sure what would really cause the slow-down with prolonged use, maybe the history got full and is purging whenever it writes to places?
(In reply to comment #17)
> I've been using your build, Shawn, and it was working great for a while, but
> recently it seems like it's worse than ever, prolonged writing to the disk even
> when I'm typing in text boxes. I'm not sure what would really cause the
> slow-down with prolonged use, maybe the history got full and is purging
> whenever it writes to places?

can you check with process explorer (or an equivalent) that it's writing on places.sqlite and that's not a urlclassifier update? Alternative you could temporary set browser.safebrowsing.enabled and browser.safebrowsing.malware.enabled to false and try... remember to re-enable them after tests.

Comment 19

9 years ago
(In reply to comment #18)
> (In reply to comment #17)
> > I've been using your build, Shawn, and it was working great for a while, but
> > recently it seems like it's worse than ever, prolonged writing to the disk even
> > when I'm typing in text boxes. I'm not sure what would really cause the
> > slow-down with prolonged use, maybe the history got full and is purging
> > whenever it writes to places?
> 
> can you check with process explorer (or an equivalent) that it's writing on
> places.sqlite and that's not a urlclassifier update? Alternative you could
> temporary set browser.safebrowsing.enabled and
> browser.safebrowsing.malware.enabled to false and try... remember to re-enable
> them after tests.

I disabled the urlclassifier through the options and I'm monitoring through filemon. It's mostly places.sqlite that freezes my cursor as I'm typing. 

I changed the history settings to only keep 60 days of history, and now it's much faster than before. Can maxing out the number of sites to keep in history make it slower due to purging of obsolete data? Maybe I'm just thinking that I fixed something by changing that value. Placebo and what not.
What value was it at before you had it set to 60 days?

Comment 21

9 years ago
Default of 90.
Depends on: 459197

Updated

9 years ago
Depends on: 459235
Blocks: 459496
(Reporter)

Comment 22

9 years ago
This will block the 3.1 release. It's imperative to fix this to perform better for USB, mobile, and profiles on a networked drive.
Flags: blocking-firefox3.1? → blocking-firefox3.1+

Updated

9 years ago
Depends on: 460300

Updated

9 years ago
Depends on: 460301
Phil - do you have any livemarks (RSS feeds) that you load?  The changes we are doing do not address that, and we are seeing a fair number of writes from them.
Latest builds appearing here:
https://build.mozilla.org/tryserver-builds/2008-10-16_09:42-sdwilsh@shawnwilsher.com-try-10df9f2c3cc/
I've updated the test build of Mozilla Firefox, Portable Edition to use Shawn's new build from today (2008-10-16).  You can download it here:
http://portableapps.com/node/15474
(Reporter)

Updated

9 years ago
Blocks: 435642
(Reporter)

Updated

9 years ago
No longer depends on: 337512
(Reporter)

Comment 26

9 years ago
The comment from the portableapps URL above indicates that there's still work to be done before this is closed. Given the comment about "while a page is loading", I'd be surprised if this is not due to favicons.
(In reply to comment #26)
> The comment from the portableapps URL above indicates that there's still work
> to be done before this is closed. Given the comment about "while a page is
> loading", I'd be surprised if this is not due to favicons.
No, I bet it is us expiring on AddVisit.  Delete's do not benefit from our work :(

Updated

9 years ago
No longer depends on: 460301
Depends on: 453529
We really shouldn't block on a meta bug, but bugs that block this should be marked as blocking.  Renoming to make sure drivers agree.
Flags: blocking-firefox3.1+ → blocking-firefox3.1?
It should be noted that if this misses 3.1, Firefox could miss out on a possible opportunity to bundle on millions of flash drives a month as Firefox 3.0 in its current state is too slow for most flash drive users.  And flash drive use accounts for 1% of Firefox downloads on a monthly basis.
(In reply to comment #29)
> It should be noted that if this misses 3.1, Firefox could miss out on a
> possible opportunity to bundle on millions of flash drives a month as Firefox
> 3.0 in its current state is too slow for most flash drive users.  And flash
> drive use accounts for 1% of Firefox downloads on a monthly basis.
"this" is just a tracking bug.  If you look at the bugs blocking this, we've made a lot of progress with this in 3.1, so I'm not sure what your comment is based on...
(In reply to comment #29)
> It should be noted that if this misses 3.1, Firefox could miss out on a
> possible opportunity to bundle on millions of flash drives a month as Firefox
> 3.0 in its current state is too slow for most flash drive users.  And flash
> drive use accounts for 1% of Firefox downloads on a monthly basis.

John, I guess you should file a new bug on that (if there isn't one already) and make it dependent on this bug and make sure it has set the blocking-firefox3.1? flag (and possibly the blocking-gecko1.9.0.x? flag).
(Reporter)

Comment 32

9 years ago
Removing blocking request as it's a tracker. As mentioned above, please request blocking on dependent bugs.
Flags: blocking-firefox3.1?
(In reply to comment #31)
> John, I guess you should file a new bug on that (if there isn't one already)
> and make it dependent on this bug and make sure it has set the
> blocking-firefox3.1? flag (and possibly the blocking-gecko1.9.0.x? flag).

No detail would be in that bug other than what is in this bug, really.  I don't know enough about the inner workings to know what exactly is slowing things down.  The millions of other flash users and I just know that it is slowing down to the point of being unusable on average speed drives compared to Firefox 2 which works just fine on even slower flash drives.
(In reply to comment #33)
> The millions of other flash users and I just know that it is slowing
> down to the point of being unusable on average speed drives compared to Firefox
> 2 which works just fine on even slower flash drives.

which version? 3.0 or beta versions based on current 3.1 trunk?
Also, i think (my personal opinion) for Portable Firefox could make sense reduce default history expiration values, if possible.
(In reply to comment #34)

> which version? 3.0 or beta versions based on current 3.1 trunk?
> Also, i think (my personal opinion) for Portable Firefox could make sense
> reduce default history expiration values, if possible.

Only the stable releases achieve widespread use, so it's based on 3.0.x vs 2.x.  We do package the betas and will be doing 3.1B2 later today.  I'm not sure if adjusting history would help as it would just speed up or delay the expiration but just as much would eventually expire and need to be removed.
(In reply to comment #35)
> (In reply to comment #34)
>  I'm not sure if
> adjusting history would help as it would just speed up or delay the expiration
> but just as much would eventually expire and need to be removed.

This is a good point, but will also make the db smaller, so should be probably tested on the road. Probably later, when we will revise expiration to be async that could be better.
(Reporter)

Comment 37

9 years ago
(In reply to comment #36)
> This is a good point, but will also make the db smaller, so should be probably
> tested on the road. Probably later, when we will revise expiration to be async
> that could be better.

Is there a bug for that?
Depends on: 470025
(In reply to comment #37)
> Is there a bug for that?

I did not find a dupe so i filed bug 470025
No longer depends on: 470025
Depends on: 470429
(Reporter)

Updated

9 years ago
Depends on: 470025
3.1 work here is practically finished, we will look for further gains for 3.2
Target Milestone: Firefox 3.1 → ---

Updated

9 years ago
Depends on: 480211
(Reporter)

Comment 40

8 years ago
All dependent bugs are fixed. However, before closing this I'd like to see someone run Shawn's DTrace script that logs fsyncs, and catalog exactly what Places code is writing things on/after page-load.
(Reporter)

Updated

8 years ago
Whiteboard: [tsnap]
Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv
Component: Places → Bookmarks & History
QA Contact: places → bookmarks
(In reply to comment #40)
> All dependent bugs are fixed. However, before closing this I'd like to see
> someone run Shawn's DTrace script that logs fsyncs, and catalog exactly what
> Places code is writing things on/after page-load.

For Firefox 4.0 Shawn did various measures while working on async visits, that showed a really good shape regarding fsyncs (reduction of 65%), this was on further work done after the changes that hit 3.6.
So I think after 2 years this bug has done its time, we'll continue working on I/O in new bugs.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.