Closed Bug 432706 Opened 12 years ago Closed 9 years ago

Deleting/adding/updating a large quantity of bookmarks/history is very slow due to updating observers

Categories

(Toolkit :: Places, defect, P2)

defect

Tracking

()

RESOLVED WONTFIX

People

(Reporter: mkneubuehl, Unassigned)

References

Details

(Keywords: perf, Whiteboard: [tsnap])

Attachments

(3 files, 6 obsolete files)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de-CH; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de-CH; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5

In order to delete a LARGE (2500) quantity of bookmarks, it needs for a very long time (3-4 minutes).

Reproducible: Always

Steps to Reproduce:
1. With bookmark manager
2.
3.
Actual Results:  
It needs for a very long time to finish.

Expected Results:  
Bookmarks where deleted.
Would you say a dupe of bug 428459 ?
Component: Bookmarks → Places
QA Contact: bookmarks → places
Version: unspecified → Trunk
No, it is not exactly the same as in bug 428459. The browser doesn't hangs, it uses only a long time to delete all of the bookmarks (2500 with many subfolders). The behavior in the new RC1 is still the same (CPU usage 40%).
Keywords: perf
OS: Windows XP → All
Priority: -- → P2
Hardware: PC → All
Summary: Delete large quantity of bookmarks → Deleting a large quantity of bookmarks is very slow
Target Milestone: --- → Firefox 3.1
Status: UNCONFIRMED → NEW
Ever confirmed: true
bug 428459 says hang but probably was a really long time, i hope we can unify all those reports
Duplicate of this bug: 434535
setting _at_least_ a dependancy since is probably related
Depends on: 428459
Duplicate of this bug: 429480
Blocks: 437244
As i said yesterday in IRC with dietrich we should revise the transaction manager, it reloads all the data from the db for every delete, while usually we already know those informations when deleting (for example the title of a node, the uri of a node and so on), so we could make the transactions take an hash instead of a simple itemId and try to load infos from db only if we don't have them.
Assignee: nobody → mak77
No longer blocks: 380307
Whiteboard: [Related: 329534, 391836, 393497, 397993]
Attached patch wip (obsolete) — Splinter Review
this is based on patch from bug 430442, and includes that.
The main change is RemoveItem, that is the transaction method that makes more queries to the db, so, since often we already know infos from what we are removing, we could pass an hash with those infos, and use them if possible.
Creating s specific interface for this or making all interface taking nsIVariant appear too complex for the advantage we could have elsewhere, so after some trying i ended up with a single hash added where needed. 
I'm asking review only to get attention, still we should take bug 430442 first
Attachment #327514 - Flags: review?(dietrich)
Hate to rain on folks parades here, but we should really have a test land before this fix lands for deleting bookmarks so we can have automated coverage that ensures this doesn't break.  UI tests aren't that hard to make (chrome test in this case is what you'd want), and the download manager has lots of samples you can copy code from.
Flags: in-testsuite?
I'm writing a perf test for this, and I'm seeing a lot of the following assertion when I'm deleting 250 bookmarks:
###!!! ASSERTION: Removing item we don't have: 'Not Reached', file /Code/moz/central/mozilla/toolkit/components/places/src/nsNavHistoryResult.cpp, line 3469
Here's a handy performance test with hooks for shark to test this.  It adds 250 bookmarks, opens the library, and then deletes those bookmarks in one swoop (accel + a, then backspace).  The test includes instructions on how to run it, but it will not run as part of the normal chrome test harness.
Attachment #329723 - Flags: review?(dietrich)
(In reply to comment #11)
> I'm writing a perf test for this, and I'm seeing a lot of the following
> assertion when I'm deleting 250 bookmarks:
> ###!!! ASSERTION: Removing item we don't have: 'Not Reached', file
> /Code/moz/central/mozilla/toolkit/components/places/src/nsNavHistoryResult.cpp,
> line 3469
> 

I can't reproduce with 3.1a1pre 2008-07-15-04. What I see in error console is much of these errors:

  Error: [Exception... "'Illegal value' when calling method: 
  [nsITreeView::XXXXXX]"  nsresult: "0x80070057 (NS_ERROR_ILLEGAL_VALUE)"  
  location: "<unknown>"  data: no]

with XXXXXX equal to:

isContainer
getRowProperties
isSeparator
getCellProperties
getImageSrc
getCellText
getLevel

Anyway, is it a mine impression or the bookmark deletion is faster now - even if still much slower than Fx2?
May be dependent on bug 418643
You won't see the assertions in release builds, but I get those errors too.  Can you file a new bug on those please?
(In reply to comment #15)
> You won't see the assertions in release builds, but I get those errors too. 
> Can you file a new bug on those please?
Actually, don't.  It's bug 433370.
Attached patch shark profileSplinter Review
And a shark profile.

For starters, the call stack is, um, insane when viewing only the main thread.  We spend a bunch of time in JS, but in the end, that calls sqlite stuff.  We spend a lot of time in sqlite while deleting, which doesn't make any sense ;)
17.8% sqlite3_step

I'm gonna go ahead and apply this current patch and see how that helps.
Forget to mention - the underlying places function that we spend the most time is nsNavBookmarks::RemoveItem(long long) weighing in at 97.5% of our time!!!
Comment on attachment 329723 [details] [diff] [review]
performance test v1.0 (landed)

looks ok, r=me
Attachment #329723 - Flags: review?(dietrich) → review+
Comment on attachment 329723 [details] [diff] [review]
performance test v1.0 (landed)

Pushed to mozilla-central:
http://hg.mozilla.org/mozilla-central/index.cgi/rev/76f62ca28996
Attachment #329723 - Attachment description: performance test v1.0 → performance test v1.0 (landed)
More details as I figure out even more about shark.  We spend 6.6% of the time updating the bookmark indicies (nsNavBookmarks::AdjustIndicies), which isn't much, but is concerning since we are in a batch update.

The bulk of our time (87.8%) is spent notifying observers though.  Working on figuring out why...

Something that seems weird to me is that we call nsNavHistoryFolderResultNode::OnItemRemoved twice with mItemId being 5 (unfiled bookmarks as far as I know, since that's where I'm deleting).  The first time always gets that NOT REACHED assertion, but the second time works just fine.  If someone could explain why we call it twice, that'd be awesome.

What seems strange to me is that we have nsNavHistoryFolderResultNode, which adjusts bookmark indicies, while nsNavBookmarks (in nsNavBookmarks::AdjustIndicies) also does this.  Why are we duplicating work again and again?
Weirdness here.  I've added code so that OnBeginUpdateBatch sets a var so all operations are a no-op during batch.  However, that method never gets called.

I've gone in a debugger and set a breakpoint in nsNavBookmarks::BeginUpdateBatch, and we hit that, and it goes all the way and enumerates the observers to notify them about the the beginning batch update.  Our observer never gets called there, but does get called whenever nsNavBookmarks::RemoveItem is called.  This dispatches observers the same way, so I'm not really sure what is up.  If someone who knows this code better could provide insight here, that'd be awesome.
Attached patch wip batch 0.1 (obsolete) — Splinter Review
Attached patch v0.2 (obsolete) — Splinter Review
What an improvement.  Now running that perf test it takes less than one second to complete with this patch.  I'm failing only one test, and that looks to be tags not being restored.  Not 100% sure what's up with that yet.
Attachment #327514 - Attachment is obsolete: true
Attachment #330270 - Attachment is obsolete: true
Attachment #327514 - Flags: review?(dietrich)
I never indicated how to run the performance test, which may be helpful.  In your object directory, navigate to _tests/testing/mochitest, then run
python runtests.py --chrome --test-path=../chrome/browser/components/places/tests/perf/perf_large_delete.xul

Marco - I'm going to be gone for most of next week, so feel free to take this and run with it if you want.
adding bug 418643 as a dependancy, having a faster removeFolderChildren would be a good step forward
Depends on: 418643
Marco - do you plan on finishing this, or should I take it the rest of the way?
i'll probably finish the change with the transaction manager on comment #9 and finalizing the removeFolderChildren patch. i cannot use Shark pitifully so i cannot go on with your test, i've not yet compiled you patch in but i'll do soon to try following that thing (feel free to go on with that if you've slots)
Assignee: mak77 → dietrich
Blocks: 329534
No longer depends on: 428459
No longer depends on: 418643
Status: NEW → ASSIGNED
Requesting blocking 3.1 This change will have a broad impact: livemark reloads, bookmark restores, manual deletion of bookmarks, Weave and other extension sync.
Flags: blocking-firefox3.1?
Priority: P2 → P1
I support this as blocking. Response times in teh Library have become quite cumbersome.
Hey Paul,

Please see and read the bugzilla etiquette page [1].  Comments like comment 30 aren't at all constructive and do not help get the bug fixed (or blocking status in this case).

[1] https://bugzilla.mozilla.org/page.cgi?id=etiquette.html
(In reply to comment #31)
> Hey Paul,
> 
> Please see and read the bugzilla etiquette page [1].  Comments like comment 30
> aren't at all constructive and do not help get the bug fixed (or blocking
> status in this case).
> 
> [1] https://bugzilla.mozilla.org/page.cgi?id=etiquette.html
> 

Okay
Should this bug and bug 418176 be combined, as being a single issue with Places/Library, instead of 2 separate issues for bookmarks and history?
Duplicate of this bug: 418176
Now that 418176 has been made a duplicate of this bug, can we update the summary to something like:

Deleting a large quantity of Library items is very slow

or

Deleting a large quantity of bookmark/history items in the Library is very slow
Duplicate of this bug: 440101
I think bug 428459 is a dupe. 

#35:
From my experience with bookmark items, I'm finding slow performance when you added items in bulk and thus the title should encompass the whole performance issues concerning bookmarks/library items.  In particular, I noticed slowness when I changed a tag, and thus tagging may be the source of the bug.  

By the way - I recall seeing this bug in the past when the bookmark interface was changed. If that is an issue, do we tag this bug as a regression, or should a new bug be opened concerning creation of a new regression test case?
Blocks: 420199
Summary: Deleting a large quantity of bookmarks is very slow → Deleting a large quantity of bookmarks/history is very slow
The only sticking point left in this patch is that the tagging service depends on a live query result for basic functionality like "is a URI tagged", etc.

So with this patch, the tagging service can be working on false assumptions during a batch, since it's query node might not up to date.

For example, during a restore operation, all tags are removed and then restored from the backup *inside a batch*, resulting in a completely invalid restored set of tags.
Approving blocking of the 3.1 release, given a potential 80+% performance win for these types of activities.
Flags: blocking-firefox3.1? → blocking-firefox3.1+
Blocks: 393497
Blocks: 427521
Considering the symptoms that have been reported, 
I would hope for an improvement of 800% rather than 80%, 
otherwise the bug is not really fixed.

Please also do not forget that the problem is 
not constrained to deleting many bookmarks, 
but various other operations are incredibly slow as well, 
see my summary
https://bugzilla.mozilla.org/show_bug.cgi?id=427521#c10

----
Adding an off-topic note (sorry for this, I guess someone will bash me for it, 
but it's really a desperate issue):
Would some of the persons involved in this, while working on bookmarks issues, 
kindly have the mercy to care about major annoyance 
https://bugzilla.mozilla.org/show_bug.cgi?id=381767 too, please‽
(In reply to comment #40)
> Considering the symptoms that have been reported, 
> I would hope for an improvement of 800% rather than 80%, 
> otherwise the bug is not really fixed.

Let's get the wins we can, and re-evaluate from there. Any help you can provide in finding where the bottlenecks in the code are would be great.

> Please also do not forget that the problem is 
> not constrained to deleting many bookmarks, 
> but various other operations are incredibly slow as well, 
> see my summary
> https://bugzilla.mozilla.org/show_bug.cgi?id=427521#c10

The attached patch should affect all kinds of operations. I'll update the title.
Summary: Deleting a large quantity of bookmarks/history is very slow → Deleting/adding/updating a large quantity of bookmarks/history is very slow due to updating observers
Blocks: 428459
since the tagging service is also causing leaks (see Bug 462545) due to bad cleanup of open node we could convert it to hold a fixed cache, and use a bookmarks observer. During a batch we could double check our fixed list with the bookmarks service, checking if a tag still exists.
filed Bug 463513 to investigate tagging service change
Depends on: 463513
Blocks: 391836
Whiteboard: [Related: 329534, 391836, 393497, 397993]
Blocks: 431548
Whiteboard: [has wip patch][needs dependant bug]
all tests pass with this patch plus the patch in bug 463513.
Not having the opportunity of testing a patch version before the next release,
may I kindly ask whether those tests cover the cases I mentioned in
https://bugzilla.mozilla.org/show_bug.cgi?id=427521#c10
I'm explicitly asking because even after updating the bug title, it focusses 
on handling "many bookmarks", while I had observed slowness without many bookmarks involved:
> * Deleting one bookmark seems to be slower than deleting a group of bookmarks.
> * Even just opening a folder may take one second.
> * Deleting an empty folder takes a second, too. ...
Plus:
> * Bookmark import is incredibly slow, too, and almost blocks my PC.


May I also kindly again make this off-topic plea (feel free to bash me for it):
While there is development activity in the area of bookmarks, it would be 
great if finally someone had the mercy of caring about
https://bugzilla.mozilla.org/show_bug.cgi?id=381767
This is a great impact for many people who used to use that feature and many of them are still refraining from an update to firefox 3 in despair for that problem.
updating priority. needs to block 3.1, but not specifically b3.
Priority: P1 → P2
Flags: blocking-firefox3.1+
Product: Firefox → Toolkit
QA Contact: places → places
Target Milestone: Firefox 3.1 → mozilla1.9.1
This was blocking - fell of when it moved components
Flags: blocking1.9.1?
Flags: blocking1.9.1? → blocking1.9.1+
Whiteboard: [has wip patch][needs dependant bug] → [has wip patch]
Attached patch v0.2 updated (obsolete) — Splinter Review
mostly the same, needs test.
Attachment #330286 - Attachment is obsolete: true
Attachment #352447 - Flags: review?(mak77)
Attached patch v0.2 updated + test (obsolete) — Splinter Review
Attachment #352447 - Attachment is obsolete: true
Attachment #352502 - Flags: review?(mak77)
Attachment #352447 - Flags: review?(mak77)
Whiteboard: [has wip patch] → [has wip patch][needs review marco]
the strange thing i see here is that if i remove a bunch of bookmarks into a subfolder of bookmarks menu in the library, we end up refreshing bookmarks toolbar, unsorted bookmarks, all bookmarks, root, bookmarks menu, the folder itself, and for some of them we do that multiple times. I'll dig a bit more.
Comment on attachment 352502 [details] [diff] [review]
v0.2 updated + test

> NS_IMETHODIMP
> nsNavHistoryFolderResultNode::OnEndUpdateBatch()
> {
>+  if (mBatchInProgress) {
>+    mBatchInProgress = PR_FALSE;
>+    (void)Refresh();

any reason to ignore return result?

>diff --git a/toolkit/components/places/tests/bookmarks/test_432706_in_batch.js b/toolkit/components/places/tests/bookmarks/test_432706_in_batch.js
>+
>+var bmsvc = Cc["@mozilla.org/browser/nav-bookmarks-service;1"].getService(Ci.nsINavBookmarksService);
>+var histsvc = Cc["@mozilla.org/browser/nav-history-service;1"].getService(Ci.nsINavHistoryService);

nit: crop at 80

r=mak77

i think for the next version we need to take a deeper look at observers paths removing those that does not need to requery, for example if i change an item there's no need to update Tags queries unless that item is tagged.
Attachment #352502 - Flags: review?(mak77) → review+
Whiteboard: [has wip patch][needs review marco] → [has review]
Whiteboard: [has review] → [needs further investigation on detached nodes]
Attached patch v0.3 (obsolete) — Splinter Review
includes a minor change to force tag requery for bookmark uri nodes when the node has no parent.

i'll file a follow up for full support for self-updating-disconnected-nodes.
Attachment #352502 - Attachment is obsolete: true
Attachment #354863 - Flags: review?(mak77)
(In reply to comment #52)
> i'll file a follow up for full support for self-updating-disconnected-nodes.

filed bug 471563.
Whiteboard: [needs further investigation on detached nodes]
Comment on attachment 354863 [details] [diff] [review]
v0.3

is it possible to add a test to demonstrate the detached tags issue? maybe it could be a skeleton of the test that will make the followup, with only the tags case.
Attachment #354863 - Flags: review?(mak77) → review+
Attached patch v0.3.1Splinter Review
with the additional test, and failing tests commented out.
Attachment #354863 - Attachment is obsolete: true
http://hg.mozilla.org/mozilla-central/rev/0c0fb037904a
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Duplicate of this bug: 472343
Depends on: 473342
Backing this out due to numerous regressions. See bug 473058. The proper fix requires core changes too risky this late in the 3.1 cycle, so going to push to 3.2.

http://hg.mozilla.org/mozilla-central/rev/5977e010b3db
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/110916df5903
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Keywords: fixed1.9.1
Flags: blocking1.9.1+ → blocking1.9.1-
Target Milestone: mozilla1.9.1 → mozilla1.9.2a1
Bug 457622 (duplicate) mentioned a way to do delete a folder faster by deleting its content and then the folder itself, but haven't investigated this.
Found a typo: Comment 61 applies to Bug 475622 (duplicate).
Duplicate of this bug: 494790
Whiteboard: [tsnap]
Target Milestone: mozilla1.9.2a1 → ---
Blocks: 468307
Blocks: 510461
Here there is a real world example of an extension sending adding lots of  bookmarks. The extension replicates a dmoz category ( recursive ) on your browser bookmarks, and works very slow, the hard-disk stay all-ways red.

https://addons.mozilla.org/en-US/firefox/addon/6231

Is there any flag to set, or something, so we can add lots of bookmarks without breaking the hard-disk in two?
(In reply to comment #65)
> Is there any flag to set, or something, so we can add lots of bookmarks without
> breaking the hard-disk in two?

See bookmarksSvc.runInBatchMode(), that should help quite a lot, no miracles though.
Thanks Marco, 
But I'm sad to say that is pretty much the same.. the same.
I don't know what to do.
(In reply to comment #67)
> Thanks Marco, 
> But I'm sad to say that is pretty much the same.. the same.
> I don't know what to do.

how do you use that? it can't be the same, sure can be still slow, but the difference should be visible.
I was calling to runInBatchMode with just one item, with several items runInBatchMode working better.
Bookmarking Firefox category takes 70 seconds VS 40 seconds using runInBatchMode.

But the problem still remains, the hard-disk is red and more important the UI freezes until the bookmarking ends.

IMHO, it's better a very slow bookmarking and an UI that continues as if nothing happens.
Thanks.
Forget to say that I'm receiving two errors in the error console:

Error: 'JavaScript component does not have a method named: "onBeginUpdateBatch"' when calling method: [nsINavBookmarkObserver::onBeginUpdateBatch] = NS_ERROR_XPC_JSOBJECT_HAS_NO_FUNCTION_NAMED
Source file: chrome://odpbookmarks/content/lib/bookmarks.js
Line: 116
 ----------
Error: 'JavaScript component does not have a method named: "onEndUpdateBatch"' when calling method: [nsINavBookmarkObserver::onEndUpdateBatch] = NS_ERROR_XPC_JSOBJECT_HAS_NO_FUNCTION_NAMED
Source file: chrome://odpbookmarks/content/lib/bookmarks.js
Line: 116


Code is:

.BookmarksService.runInBatchMode
.({
.	runBatched: function(aUserData)//aUserData?
.	{
.		var data;
.		while(data = toRunBatched.shift())
.		{
.			var anURL = data[0];
.			var aTitle = data[1];
.			var parentFolder = data[2];
.			var anIndex = data[3];
.								
.			if(anIndex!==0 && !anIndex)
.				anIndex = BookmarksService.DEFAULT_INDEX;
.			 var URI = ios.newURI(anURL, null, null);
..			BookmarksService.insertBookmark(
.					 parentFolder, 
.					 URI, 
.					 anIndex, 
.					 aTitle
.					 );
.		}
.	},
.	onBeginUpdateBatch:function(aNada){},
.	onEndUpdateBatch:function(aNada){}
.	
.}, null);
.    onBeginUpdateBatch:function(aNada){},
.    onEndUpdateBatch:function(aNada){}

these should be in your bookmarks observer object, so that during batches you can maybe avoid doing some observing stuff. for example maybe you do something for each inserted bookmark, that will slow you down. since you know you are batching you can maybe replace that stuff with a single step when you receive onEndUpdateBatch
(In reply to comment #69)
> But the problem still remains, the hard-disk is red and more important the UI
> freezes until the bookmarking ends.

you can't completely avoid that for now, we are slowly moving APIs to be async, but that will require some time.
(In reply to comment #72)
> (In reply to comment #69)
> > But the problem still remains, the hard-disk is red and more important the UI
> > freezes until the bookmarking ends.
> 
> you can't completely avoid that for now, we are slowly moving APIs to be async,
> but that will require some time.

OK, Many thanks for your efforts :)

(In reply to comment #71)
> .    onBeginUpdateBatch:function(aNada){},
> .    onEndUpdateBatch:function(aNada){}
> 
> these should be in your bookmarks observer object, so that during batches you
> can maybe avoid doing some observing stuff. for example maybe you do something
> for each inserted bookmark, that will slow you down. since you know you are
> batching you can maybe replace that stuff with a single step when you receive
> onEndUpdateBatch

I think... I don't have an observer object!,  what I need to do to avoid these errors in the console? ( described previously )

Thank you.
(In reply to comment #73)
> (In reply to comment #72)
> I think... I don't have an observer object!,  what I need to do to avoid these
> errors in the console? ( described previously )

Don't you call BookmarksService.addObserver(yourObject)? that is adding a bookmarks observer. Could be that observer has been added by other extensions you have?
> Could be that observer has been added by other extensions
> you have?

True. Sorry for being distracting. Thank you.
(In reply to comment #75)
> True. Sorry for being distracting. Thank you.

i can suggest you to disable other add-ons while you test your own, since if they are observing bookmarks, it is possible that some of those 40s you see are due to other add-ons doing something on any bookmark you add.
With a clean profile it takes just 15s... that's really fast, but the UI and hard-drive...

I test on clean profiles... but I love test on real world situations, like my everyday profile with 71 extensions installed... because it gives me a more real feeling of what user will see. Also will show me how the other add-ons will react.

One more question: Is there any chance to temporally pause all observers?
You have been really helpful, many thanks again!
(In reply to comment #77)
> One more question: Is there any chance to temporally pause all observers?

No, it would be disruptive, we can't choice what you do in your observer, and we can't choice if you need to observe a change or not, that's up to extensions developers. If you identify the extension that is slowing you down i can suggest you to contact its author and tell him about the problem. Being 3 times slower is clearly a bad experience.
Assignee: dietrich → nobody
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) Gecko/20100917 CentOS/3.6-8 Firefox/3.6.10 - built locally from source from http://rpms.famillecollet.com/SRPMS/firefox-3.6.10-1.remi.src.rpm

History search box hangs for several minutes, cpu goes to 100% as soon as typing starts.

System hangs for minutes while trying to select searched entries with cpu at 100%.
search result is over 6000 items.
The original patch affects all consumers, whether core or extension.

Instead, what if we "turned off" just the core consumers that we control. So, when a batch starts, have our widgets ignore everything until the batch ends, and then do a full refresh.

Also, Marco did you ever dig farther into comment #50?
(In reply to comment #79)
> Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) Gecko/20100917
> CentOS/3.6-8 Firefox/3.6.10 - built locally from source from
> http://rpms.famillecollet.com/SRPMS/firefox-3.6.10-1.remi.src.rpm
> 
> History search box hangs for several minutes, cpu goes to 100% as soon as
> typing starts.

see also bug 595530, there is a possible regression in latest sqlite that makes those searches slower for some users.
(In reply to comment #80)
> Also, Marco did you ever dig farther into comment #50?

Nope, I guess with all the code changes for 4.0 we should start again logging observers and see where we are today.

Turning off internal consumers could work but we have to check them one by one to be sure they won't lose data, especially Sync. Batch updates unfortunately don't tell anything about what changed.
(In reply to comment #82)
> Turning off internal consumers could work but we have to check them one by one
> to be sure they won't lose data, especially Sync. Batch updates unfortunately
> don't tell anything about what changed.

Yeah, UI only might be enough (esp when Library is open). However, yeah should log observer traffic to be sure.
Status: REOPENED → NEW
I think this bug as it is is not anymore that useful:
- Firefox 4 improved the situation a lot, so the data analysis we have here is probably now invalid
- Some of the suggested changes were done in other bugs that landed in Firefox 4 or will land shortly
- Some of the changes could make things worse (in some view doing small updates is faster than serving only batches)
So I'm wontfixing this and we should file new bugs based on new current data.
Status: NEW → RESOLVED
Closed: 11 years ago9 years ago
Resolution: --- → WONTFIX
No! Firefox is still terribly slow on deleting history items, even the FF7 nightly builds!! I have uploaded a video showing it , it makes firefox go non responsive upon deletion , and takes a while to delete , after that its back to normal. I have a Athlon II 620 X4 (quad core) 2.6GHZ CPU ,2GB ram , which for sure doesnt make it look like ages old pc...


http://www.youtube.com/watch?v=0ZFHoaI_89U
I said "File new bugs based on new current data".
Btw in your vide you are removing 913 pages from history, with all their visits, that is still slow, but this bug can't help improving that.
For the record, they are 15 bugs about places performance already opened, two assigned: http://tiny.cc/wp4c2
erm , sorry about that , but chrome and opera do that job so well ,further the browser mustn't hang like that...
also , where shall i post my finding ? I dont think i should create another bug and increase the headache..
I got hit by this a couple of days ago and nearly filed a new bug. I'll paste it here for reference instead.

TL;DR: deleting 7000 Google Translate entries from my history resulted in 1 core being pegged for a number of hours and memory to exhibit a perpetual rise-then-drop pattern.

I had noticed that various pages that I expected to see in my history had been disappearing progressively. When I eventually went to investigate the issue I found that Google Translate had been building a large back catalogue due to 'Instant'. It seems like every keystroke and modification in Translate causes a new history entry. Incidentally, I found out that there seems to be no way to disable instant.

When I hit the delete key Firefox immediately started consuming one of my processor cores and began a process of raising its memory usage to around 600MB in a linear fashion and dropping back down to around 400MB before rising back to 600 again. I didn't manage to get more precise values I'm afraid.

Considering that it was showing memory and disk activity I decided to leave it for a while, since I didn't want to end up with a corrupted Places file. I'm unsure how many hours I left it, but I know it took the best part of the morning to finish the process.
It occurred on the latest stable version (5.0.1) of Firefox ?
I am sure it can occur on nightlies too..
(In reply to Virtual_ManPL from comment #91)
> It occurred on the latest stable version (5.0.1) of Firefox ?

Aurora (7.0a2)
I have an half-baked idea of how to improve that easily in my mind, I'll test it locally and file a bug for it if it gives back decent improvements.
Btw, please file new bugs rather than commenting in solved bugs, especially if you are unsure if your issue is exactly due to this specific bug.
I want to add only that try always testing it with new profile or Portable Fx to see it if still reproducible.
Because managing at last bookmarks is now far faster compared to how it was before.
I think I've been hit by this as well. Firefox 6.0.2, OSX 10.6.8. Reproduced by:

1) History -> Show all history
2) In the "search history" box, type in enough of a website name to generate the desired list of urls (say, "foo.com")
3) See returned list. Actual number of returned items, 2250.
4) Sort by location by clicking the "Location" column heading.
5) Command-A to select all, then scroll down to the bottom to deselect the last item via Command-click.
6) Hit delete key, start stopwatch.
7) Wait 22 minutes, force-quit Firefox in frustration.

I've run into this a few times in the past with this version. Load average never went above 1.03, plenty of memory is free, no system.log errors. I was going to file a bug against this, but then found this bug so I thought I'd update with new information. FWIW, this machine is shut down daily, so Firefox is getting started at least once a day. Also, I don't think the deselect in #5 has any bearing, but I've included it for sake of completeness.
> I think I've been hit by this as well. Firefox 6.0.2, OSX 10.6.8. Reproduced
> by:
> 
> 1) History -> Show all history
> 2) In the "search history" box, type in enough of a website name to generate
> the desired list of urls (say, "foo.com")
> 3) See returned list. Actual number of returned items, 2250.
> 4) Sort by location by clicking the "Location" column heading.
> 5) Command-A to select all, then scroll down to the bottom to deselect the
> last item via Command-click.
> 6) Hit delete key, start stopwatch.
> 7) Wait 22 minutes, force-quit Firefox in frustration.
> 
> I've run into this a few times in the past with this version. Load average
> never went above 1.03, plenty of memory is free, no system.log errors. I was
> going to file a bug against this, but then found this bug so I thought I'd
> update with new information. FWIW, this machine is shut down daily, so
> Firefox is getting started at least once a day. Also, I don't think the
> deselect in #5 has any bearing, but I've included it for sake of
> completeness.

Please try latest aurora or nightly build , i think there are some changed to fix it. But i don't know which fix has resulted in this , all i know is that my latest UX build is faster at deleting history records... It deletes in chunks now , earlier it used to hang and delete everything and become normal once all are deleted. However now it deletes in few chunks thus not freezing it up. Anyone knows the bug id?
(In reply to bogas04 from comment #97)

> Please try latest aurora or nightly build , i think there are some changed
> to fix it. But i don't know which fix has resulted in this , all i know is
> that my latest UX build is faster at deleting history records... It deletes
> in chunks now , earlier it used to hang and delete everything and become
> normal once all are deleted. However now it deletes in few chunks thus not
> freezing it up. Anyone knows the bug id?

Good to know! I'll wait for FF7 to see if it's been fixed, being as the release is only a week or so away. It'll take me at least that long to build up enough hits on the site in question to retry the time test anyway!
(In reply to bogas04 from comment #97)

> Please try latest aurora or nightly build , i think there are some changed
> to fix it.

I'm using Aurora and just did a test run. 1600 entries took 4 minutes 7 seconds to finish deleting and return to a usable state.

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0a2) Gecko/20110918 Firefox/8.0a2
Must be nightly then
Bug 681420 was fixed for Firefox 9, which is still in the Nightly stage and will be released 2011-12-20.
Depends on: 681420
You need to log in before you can comment on or make changes to this bug.