66907 - History sidebar/window is extremely slow

Status: RESOLVED FIXED

(Core Graveyard :: History: Global, defect, P2)

Description

[Stephan Niemz]

1. Open the history sidebar
-> It takes several seconds to load it
2. Try to do somthing (like click on a link on the displayed page or type
   something in a form input)
-> you can't do that until the sidebar finished loading

This together with the fact that the sidebar doesn't update while surfing around
(bug 66906) makes it IMHO unusable.

Build 2001012820 on NT4 and recent CVS pull on Linux

Comment 1

[Alec Flett]

we can't make it any faster until we have hierarchical sorting...

Comment 2

[ratman]

using win32 mtrunk installer build 2001031204.

with hierarchical sorting checked in, both history sidebar and the main history
window are significantly *slower* than they were without such sorting.  in
particular, it is difficult to discern between the delay involved in opening the
main branches (i.e., today, yesterday, 2 days ago, etc.) and an inactivity
timeout/freeze/crash in the browser itself.

Comment 3

[Alec Flett]

how fast is your machine? History is pretty responsive on a 400mhz PII, but
maybe it's particularily slow on slower machines?

Comment 4

[Jarek]

I have the same problem. It takes _very_ long to open folders in history
sidebar. For instance, it takes 20 seconds to open folder "Today" with approx.
60 entries. Makes history essentially unusable.

For reference, some specs: 
Celeron 400Mhz, 196 MB of RAM, running Windows'98
Mozilla 2001040404

Whenever I open any folder, processor usage goes up to 100% and drops to
essentially zero after the folder opens (indicated by windows system monitor).

Comment 5

[Alec Flett]

the two things that are going to help this are:
- indexing arbitrary columns in mork (not sure of a bug #)
- using the outliner in history (see bug 73857)

I suspect that the "opening the today folder" slowness will mostly be sped up by
the mork indexing. The problem right now is basically that we have to iterate
through every row in the database and ask "have we seen this host before?" and
if so "is this from today"?

if we could use mork to sort by date, we could quickly find the first row from
"today" and iterate through until we hit a row from "tomorrow" - essentially
limiting the scope of the rows that we need to match.

Comment 6

[robert]

In the meantime, just a suggestion here, not sure how much work would be
involved, thats for someone else to assess, how about invoking the hourglass
then clicking on History contents ?

I was just trying to search through the history in the sidebar.. but I was never
sure if i clicked on the icons because it takes so long to load,, having the
hourglass would let me know its actually doing something.

Comment 7

[Alec Flett]

good idea.
as always, accepting patches :)

Comment 8

[Alec Flett]

*** Bug 78746 has been marked as a duplicate of this bug. ***

Comment 9

[Todd Pringle]

Alec, from your comments in 78746, it sounds like this isn't going to happen for
beta.  We can probably live with that, though it makes the feature very hard to
demo when we go out to the press.  For rtm, I think we really need to implement
one or both of the suggestions in this bug, because as it stands now, history is
almost unusable on my machine (266 MHz, 128MB RAM - pretty close to our average
end-user) when there are a fair number of entries in a folder (as is almost
always the case for "6 Days or older").  

Also, can you mark nsbeta1 to get on the radar - it doesn't seem to work for me.

Comment 10

[Blake Ross]

Nom. nsbeta1; Todd, I updated your permissions so you can do this yourself.

Comment 11

[Alec Flett]

the solution when you go out to press is to have a small history :) - the time
it takes to open any one folder is directly proportional to the number of items
in history..

Comment 12

[Todd Pringle]

Thanks Blake.  Alec, yeah, that's what I was planning on telling folks for the demo.

Comment 13

[Paul Chen]

nav triage team:

Marking nsbeta1+ and mozilla0.9.2

Comment 14

[Alec Flett]

My comments from bug 78746:
------
I don't think there's any way I can make this faster for beta - we need two,
fairly large things:
- implement indexing in mork, and make use of it in histoy
- switch to rdfliner

each of these are a signifigant chunk of work (5+ days)

Comment 15

[Alec Flett]

removing nsbeta1+ for reconsideration - there's no way I can do this for
nsbeta1, the risk seems very, very high, and this does not sound like a
beta-stopper to me

Comment 16

[Todd Pringle]

Alec, the nsbeta1+ was just to show that we think this is important to keep on
the radar - the scheduling into 0.9.2 indicated that we think we can do this
post-beta for rtm.  I think because there is no planned rtm/rtm+ keyword, we're
using nsbeta1/nsbeta1+ still.  The milestone scheduling is then being used to
say when we want to do it by.  Vishy may be able to explain better.

Comment 17

[Viswanath Ramachandran]

Yes nsbeta1+ _and_ tm=mozilla0.9.1 means its a beta stopper. Otherwise its been
moved into the post beta list ie. potentially rtm. Its still not been decided
what the rtm keyword is so we're just moving the milestone for those while
reusing nsbeta1+. restoring nsbeta1+, but its not a beta stopper as its in
m0.9.2. thanks, Vishy

Comment 18

[Blake Ross]

I have global history limping along on rdfliner.  I don't know how much faster 
it is (will profile later), although I suspect that most of the history 
slowness was due to mork and the like, not the tree.

Comment 19

[Viswanath Ramachandran]

we need to scope this work out better and see if we can improve the performance 
without rewriting it. otherwise, we need to defer it to mozilla1.0. 

the only way we shd rewrite this to rdfliner is if we can have an experimental 
build that is well tested/qaed early in the mozilla0.9.2 cycle. 

Comment 20

[Alec Flett]

blake is working on the outliner conversion in another bug... it might happen in
0.9.2..

Comment 21

[Viswanath Ramachandran]

nav triage: moving to 1.0, this is not critical for rtm. 

Comment 22

[Aaron Kaluszka]

While perf is proportional to number of items in list, is it proportional in any
way to system specs?  Its almost unusable on my PIII-600/128MB RAM/Win2K.

Comment 23

[Jan Varga [:janv]]

Hi

I was talking with blake about slowness of history outliner.
I've made little research and found source of the slowness.

nsXULOutlinerBuilder::IsContainerEmpty()
  nsXULTemplateBuilder::CheckContainer()
    nsGlobalHistory::GetTarget()
nsGlobalHistory::GetTarget() is really way too slow. I takes between 100-300 ms
on my machine for *each row* of history outliner !!!

Comment 24

[Jan Varga [:janv]]

To be even more specific:

nsXULOutlinerBuilder::IsContainerEmpty()
  nsXULTemplateBuilder::CheckContainer()
    nsGlobalHistory::GetTarget()
       nsMdbTableEnumerator::HasMoreElements()

HasMoreElements() has to iterate through rows each time. It's about *1000* rows
sometimes. So nothing help it more as indexing.

Comment 25

[Jeremy M. Dolan]

Summary says this is only for history sidebar, but I get the same performace in
the sidebar and the seperate history window.

~20 seconds to open up 'yesterday'
~15 seconds to redraw after a 'page down', after yesterday is expanded
~85 seconds to expand a specific site

Comment 26

[Jeremy M. Dolan]

the scrolling seems to be covered by bug 91044

Comment 27

[Paul Chen]

nav triage team:

Need to look at this for mozilla0.9.4, marking mozilla0.9.4 and reassigning to
blaker

Comment 28

[Blake Ross]

This should be significantly improved since this was first reported (especially
in the past few days). Is it?

Comment 29

[Scott Tran (Wildcard)]

Seems there still is a little latency between the click and it actually showing
up, but it seems much better than before. Using 2001080203

Comment 30

[Blake Ross]

Jeremy Dolan, can you test again please?

Comment 31

[Jeremy M. Dolan]

With 0.9.3:

~8 seconds to expand yesterday (down from 20)
~16 to redraw after pagedown (no change)
~1 to expand a specific site (down very much from 85)

Of course, this is all subject to a lot of skew depending on how big my history
file is, how much I surfed yesterday, where the data is stored for the specific
site I expand.

Also note, for the 8 seconds to expand a day, if you close that day, then reopen
it again right away, you have to wait another 8 seconds. (No, I don't suggest we
cache MORE, just that it can be more annoying/longer then it seems)

Comment 32

[Paul Chen]

nav triage team:

At this point, we've gotten all the speedups we'll get for this milestone. Going
forward, I think we'll need to speed up the global history api itself. Pushing
out to mozilla1.0.1

Comment 33

[Ben Bucksch (:BenB)]

*Each* click on the day-twister (no matter, if open or close) takes the same time.

Nominating for mozilla1.0, modifying summary.

Comment 34

[Blake Ross]

-->[e-mail]

Comment 35

[Blake Ross]

This has greatly improved. I'm marking fixed, and specific performance
issues/optimizations can be filed separately as needed.

Comment 36

[Alexander Klimetschek]

I am running Win95 on a P233MMX with 128 MB of RAM and the history is very slow
on my computer. Moz build is 1.1, 20020826.

Opening the history window takes about 20 seconds, the time it takes for
opening/closing a folder is in a range of 10 to 20 seconds.
Closing the window needs about 5 seconds as well.

Are there any possibilities to improve the performance (relying on fast
computers is bad...)?