Closed Bug 871943 Opened 11 years ago Closed 10 years ago

When a lot of history is accumulated, the browser becomes very slow.

Categories

(Firefox :: Bookmarks & History, defect)

Product:
Firefox
Component:
Bookmarks & History
Version:
20 Branch
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED INCOMPLETE

People

(Reporter: t8av, Unassigned)

Details

(Keywords: hang, perf, stackwanted, Whiteboard: [needs profile])

Attachments

(1 file)

zipped HTML file that causes the browser to hang for almost a minute
353.33 KB, application/octet-stream
Details 
User Agent: Mozilla/5.0 (Windows NT 6.1; rv:20.0) Gecko/20100101 Firefox/20.0
Build ID: 20130409194949

Steps to reproduce:

I had a tab open on a website, then I clicked a URL. 

The browser responded very slowly (approx. 12 seconds, compared to Google Chrome which took only 2-3 seconds for the same URL).

Then I deleted all browser history and tried to click the URL again.


Actual results:

Browsing was much faster after deleting all history (however, still slower than Chrome).



Expected results:

There are many possible solutions for this problem:

1. The browser should use history much less during regular work.
2. Apply a different database approach when acceding old data.
3. Also see: https://bugzilla.mozilla.org/show_bug.cgi?id=867110
Keywords: perf
Component: Untriaged → Bookmarks & History
please get a profile with the integrated profiler and add it here, it's possible the problem was elsewhere, history access during browsing is extremely limited and not blocking, so that's surprising to me.
(In reply to Marco Bonardo [:mak] from comment #2)
> please get a profile with the integrated profiler and add it here, it's
> possible the problem was elsewhere, history access during browsing is
> extremely limited and not blocking, so that's surprising to me.

I insist that Firefox is very slow for me (much much slower than Chrome for the same URL).

Whenever it gets too slow, I create a new profile, then the browser responds more quickly, however after a while it gets slow again.

Please explain here how to create the data that you need to analyze the problem. I will create a new profile and gather the data there.
I checked this issue with a specific URL.

I used javascript to check HTML load time.

var timing = performance.timing;
var loadtime = timing.loadEventEnd - timing.responseEnd;
var loadtime_sec = loadtime/1000;

Firefox:
62.614sec  

Chrome:
5.504sec
(In reply to t8av from comment #4)
> I checked this issue with a specific URL.
> 
> I used javascript to check HTML load time.
> 
> var timing = performance.timing;
> var loadtime = timing.loadEventEnd - timing.responseEnd;
> var loadtime_sec = loadtime/1000;
> 
> Firefox:
> 62.614sec  
> 
> Chrome:
> 5.504sec

Thanks, I have few questions:
- what are the urls you were using to test?
- what type of hardware are you on?  Mainly HDD/SSD/something else, how much memory, processor (at least generation)


This can be in general anything as well as something itchy on a web page you are testing with it self.
URL - this is an internal URL behind localhost Apache server. If you want, I can copy the HTML (how do I upload files to this page?).

Hardware - CPU intel core quad Q82000 2.33GHz, RAM 2GB, OS - windows7 32 bits. Firefox is installed on internal disk C:/ but the profile is defined on an external hard drive (WD USB drive 1T).
(In reply to t8av from comment #6)
> but the profile is defined
> on an external hard drive (WD USB drive 1T).

Thanks for this info.  This is with high probability cause of the problem.  The USB drive is probably magnitude (or at least significantly) slower then the regular drive connected via internal SATA/EIDE.  When amount of data grows, we probably slow down.

Does also your cache (http cache) data are pointed to this drive?  

Are you running Firefox with -profile command line switch or did you create the profile on the USB drive using the profile manager?

Can you direct the Chrome's profile data also to this drive and test with chrome as well?  If there is still significant performance difference, then we definitely should prioritize out performance optimization efforts (some of them are tho on the way, e.g. our HTTP cache is just being rewriten from scratch to perform much better).

Anyway, we know Firefox performance suffers when profile resides on a slower drive, so this is not a new information.
Its not the disk:

1. Chrome's cache is also defined in the same external disk using HW link.

2. Firefox itself is installed in the internal disk C: , only the profile was defined in an external HD. I tried to access the same URL from a profile that is defined in C: and I got the same results.

I forgot to mention that not only that Firefox is very slow in that URL, but also the entire browser hangs and cannot be used until the page is completely loaded.

Any ideas?
(In reply to t8av from comment #6)
> URL - this is an internal URL behind localhost Apache server. If you want, I
> can copy the HTML (how do I upload files to this page?).

You can see "Add an attachment (proposed patch, testcase, etc.)" just above comment 0. You can then select Choose File, and if it does not auto-detect the Content Type you can manually select it to HTML.
I saved the HTML as "web page, complete", and zipped it.

This HTML page causes the browser to hang for almost a minute. When viewing this page in other browsers (IE, chrome) it loads very fast.
OK, I have uploaded the HTML file.

Please help me to check the cause of this problem. It's very Possible that there is a bug in Firefox since IE and chrome respond much much faster and do not hang.
(In reply to t8av from comment #11)
> Please help me to check the cause of this problem.
Provide a stack trace for the hang (see https://developer.mozilla.org/docs/How_to_get_a_stacktrace_with_WinDbg).
Keywords: hang, stackwanted
(In reply to Scoobidiver from comment #12)
> Provide a stack trace for the hang (see
> https://developer.mozilla.org/docs/How_to_get_a_stacktrace_with_WinDbg).

I will try to follow this long procedure of creating a stack trace for the hang.

Meanwhile can't you just open the attached HTML page and see if it hangs in your browser?
(In reply to t8av from comment #13) 
> Meanwhile can't you just open the attached HTML page and see if it hangs in
> your browser?
No, it doesn't in Nightly with a HDD.
after the slowdown finishes, could you please open about:telemetry page and paste here the "Slow SQL Statements on Main Thread" section contents?
by looking at the page I can't see a relation with history, unless by "history" we also consider any other private data (cookies, cache and so on)
When you said it's faster after you deleted history, did you mean you only checked "Browsing and Downloads History" in the clear history dialog, or did you rather cleaned up any private data in the dialog?
(In reply to Marco Bonardo [:mak] from comment #15)
> after the slowdown finishes, could you please open about:telemetry page and
> paste here the "Slow SQL Statements on Main Thread" section contents?

This field is empty. I tried to enable telemetry but it is still empty.
      
  Slow SQL Statements  (No data collected)
(In reply to Marco Bonardo [:mak] from comment #16)
> by looking at the page I can't see a relation with history, unless by
> "history" we also consider any other private data (cookies, cache and so on)
> When you said it's faster after you deleted history, did you mean you only
> checked "Browsing and Downloads History" in the clear history dialog, or did
> you rather cleaned up any private data in the dialog?

It is partially related. There are two different issues here: 

1. Firefox hangs in specific web pages, such as the one that I have uploaded.

2. After clearing history (I refer to: "Download and browsing history" and also the cache), firefox is faster for a while, until it gets slower again.

Maybe it is my fault for not separating these two issues to two different bugs, but also you can see this web page as a test case for #2 as well. If you want you can separate the two bugs. It is fine with me.
the problem is not separating into bugs, the problem is figuring if the issue is the cache or if the issue is "Browser and download history", since those are very different.
So I'd appreciate if you could figure if just clearing one of those and not both solves the speed problem.
I checked this issue again:

1. I created a new profile. Disabled all ODD-ONS and removed all extensions.

2. I tried to open the file that I uploaded (file2.html) in the new profile. 

3. I repeated #2 few times (closes the browser and opened it again in that page). In most cases Firefox hangs for 40-50 seconds, but in few cases it did you hang at all and was download quite fast. It is strange and is a random behavior of Firefox.

You wrote that your browser did not have a problem with this page, it is very different from my experience. Please try to load this page few times, maybe you will experience the slowness that I see.

Regarding your question - from my experience so far, cleaning "Download and browsing history" has more positive effect on the speed than clearing the cache. 

I did not say clearing history solves the problem, I said it improves the speed for a while.
(In reply to t8av from comment #20)
> Please try to load this page few times, maybe you will experience the slowness that I
> see.
With a new profile (HDD) on Nightly, I loaded the page and closed Firefox ten times without experiencing a hang.
since I think nobody posted this yet, you should be able to capture a profile with these instructions:
https://developer.mozilla.org/en-US/docs/Performance/Profiling_with_the_Built-in_Profiler
(In reply to Marco Bonardo [:mak] from comment #15)
> after the slowdown finishes, could you please open about:telemetry page and
> paste here the "Slow SQL Statements on Main Thread" section contents?

Ok, now I can see it in about:telemetry:

Slow SQL Statements on Main Thread
Hits	Avg. Time (ms)	Statement
1	62	UPDATE moz_downloads SET tempPath = :tempPath, startTime = :startTime, endTime = :endTime, state = :state, referrer = :referrer, entityID = :entityID, currBytes = :currBytes, maxBytes = :maxBytes, autoResume = :autoResume WHERE id = :id /* downloads.sqlite */
1	62	SELECT * FROM moz_logins WHERE (hostname = :hostname) AND (formSubmitURL = :formSubmitURL OR formSubmitURL = :private) AND (httpRealm isnull) /* signons.sqlite */
(In reply to t8av from comment #23)
> Ok, now I can see it in about:telemetry:
> 
> Slow SQL Statements on Main Thread
> Hits	Avg. Time (ms)	Statement
> 1	62	UPDATE moz_downloads SET tempPath = :tempPath, startTime = :startTime,
> endTime = :endTime, state = :state, referrer = :referrer, entityID =
> :entityID, currBytes = :currBytes, maxBytes = :maxBytes, autoResume =
> :autoResume WHERE id = :id /* downloads.sqlite */
> 1	62	SELECT * FROM moz_logins WHERE (hostname = :hostname) AND
> (formSubmitURL = :formSubmitURL OR formSubmitURL = :private) AND (httpRealm
> isnull) /* signons.sqlite */

62 milliseconds, not relevant enough for this case.
(In reply to Marco Bonardo [:mak] from comment #22)
> since I think nobody posted this yet, you should be able to capture a
> profile with these instructions:
> https://developer.mozilla.org/en-US/docs/Performance/
> Profiling_with_the_Built-in_Profiler

If I install Firefox nightly, can I still use my regular Firefox?
yes, just don't install in the same folder and create separate profiles (https://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles)
In any case I suggest before doing anything you backup your current profile, so you are safe in case of mistakes.
1. Ok, I installed nightly and created a new profile for it.

2. I tried to load files2.htm in nightly and it hangs for about a minute.

3. What do you want me to do next? Please choose the easiest way for me to send you the relevant data (I am not a firefox expert).
just follow the instructions to get a profile through the profiler, basically you start the profiler through the given button, execute the slow operation, stop it. It should generate a complete pseudo stack showing where most of the time was spent.
If I remember correctly, from there you should be able to upload the profile, it should give you a link so we can access it remotely. Or you may be able to just save it and attach it to the bug.
I can understand this is quite technical and scary, but in this case may be the only way to figure what's slow in your setup since we don't have access to it.
Cc-ing Vladan who recently made some testing on a very slow pendrive (iirc) and may be able to reproduce the issue through the same method.
t8av@yahoo.com" can you please try to disable all your plugins (not extenstions!) ?

- press Ctrl-Alt-A
- go to "Plugins" tab
- disable them all
- retest your case

Also, any antivirus software on your machine?  If so, which one and what version?  If not, try installing some (or just run some free scanner [1], [2]) and check for viruses.

[1] http://www.kaspersky.com/security-scan
[2] http://www.microsoft.com/security/scanner/en-us/default.aspx
(In reply to Honza Bambas (:mayhemer) from comment #30)
> t8av@yahoo.com" can you please try to disable all your plugins (not
> extenstions!) ?
> 
> - press Ctrl-Alt-A
> - go to "Plugins" tab
> - disable them all
> - retest your case
> 
> Also, any antivirus software on your machine?  If so, which one and what
> version?  If not, try installing some (or just run some free scanner [1],
> [2]) and check for viruses.
> 
> [1] http://www.kaspersky.com/security-scan
> [2] http://www.microsoft.com/security/scanner/en-us/default.aspx

I have AVG since I formatted the disk and installed the OS on this PC.

2. I don't think I should try also kaspersky or microsoft antivirus due to two reasons:

1. As far as I know a new Antivirus should be installed immediately after hard disk format and OS installation, otherwise it is a bit useless.

2. This problem of slowness and hang occurs only in very specific web pages, such as the one I have uploaded.
(In reply to t8av from comment #31)
> (In reply to Honza Bambas (:mayhemer) from comment #30)
> > t8av@yahoo.com" can you please try to disable all your plugins (not
> > extenstions!) ?
> > 
> > - press Ctrl-Alt-A
> > - go to "Plugins" tab
> > - disable them all
> > - retest your case


And what new info turning off plugins brought?



> 1. As far as I know a new Antivirus should be installed immediately after
> hard disk format and OS installation, otherwise it is a bit useless.

Sure, you are right, but those are just scanners.  No installation, no resident protection, just purely scan files on the disk.

However, I wanted to eliminate case you might experience a virus infection.  AVG should protect you.  

On the other hand, AVG could also cause the trouble...
(In reply to Honza Bambas (:mayhemer) from comment #32)
> 
> And what new info turning off plugins brought?

Disabling all plugings changes nothing - still I get hang and slowness of that page for about one minute.
(In reply to Honza Bambas (:mayhemer) from comment #32)
> 
> On the other hand, AVG could also cause the trouble...

I disabled AVG for 5 minutes and loaded the HTML page. Still same hang and slowness.
I used AVG internal disable feature. I guess there is a better way to completely stop AVG (such as killing all its processes), but I didn't try it.
http://people.mozilla.com/~bgirard/cleopatra/#report=3b9380a47da62f9b6adb3a0066ddcf5ca43789ae

Can you see the hang in this report? If not, then probably I created the report in a wrong way and should be repeated.
I see a 14s hang in that report. 2 seconds going away in layout painting, and 8 seconds in a kiFastSystemCallRet call to the OS that is likely waiting for something to finish, I assume IO. There is definitely some long operation blocking the main-thread.
Unfortunately doesn't seem to help pointing out which operation was pending. Maybe Honza has better luck than me reading this profile.
(In reply to Marco Bonardo [:mak] from comment #36)
> I see a 14s hang in that report. 2 seconds going away in layout painting,
> and 8 seconds in a kiFastSystemCallRet call to the OS that is likely waiting
> for something to finish, I assume IO. There is definitely some long
> operation blocking the main-thread.
> Unfortunately doesn't seem to help pointing out which operation was pending.
> Maybe Honza has better luck than me reading this profile.

1. The hang that I experienced for longer than 14 seconds. It was more than 40 seconds. I hope I created the report correctly.

2. I would like to remind you that the source HTML file is attached to this bug so maybe it can help to understand what went wrong.
t8av: I was looking at the report, there are definitely hangs reported.  However, w/o deeper analyzes I can't say what's going on.  I need some time to investigate.

Feel free to attach more reports!  It can always just help.

Please, stay tuned, this is an interesting bug.
I couldn't reproduce this hang with the attached html file and my regular Firefox profile.

I got one 35 second hang from loading a JS module from the startupCache in my SD-card profile: http://people.mozilla.com/~bgirard/cleopatra/#report=8dbeb03c9b691ff2742289a2c06a5fb52c245c1a

I don't think this is the same issue as t8av is reporting
(In reply to Vladan Djeric (:vladan) from comment #40)

> I got one 35 second hang from loading a JS module from the startupCache in
> my SD-card profile:

Many thanks, you are right!

I have disabled my menu modules and it is much faster now.

These menu modules are - anylinkmenu.css, menucontents.js, anylinkmenu.js.

I have downloaded them from: http://www.dynamicdrive.com/dynamicindex1/dropmenuindex.htm

From my side, the hang issue is solved, and I will probably change this JS menu to another one.

If you want, you can further investigate this JS module's hang, but it is your decision.

I still insist that browsing history causes the browser to be slow, and the solution should be automatic clearing of the oldest data, considering user configuration, so you cannot close this bug, yet.

Thanks again!
If you decide to further investigate this issue, then I would like to add:

1.  Calling the JS modules in the <head> section,  does not cause the hang. The hang occurs only if there are many instantiations of this JS menu in the HTML. In file2.htm I used the menu more than 200 times.

2. There is no hang in IE and Chrome for this file, so maybe there is some buggy behavior of Firefox with that file.
(In reply to Honza Bambas (:mayhemer) from comment #43)
> t8av, can you please test with a build at
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/gum-win32/
> 1376312141/ ?
> 
> Thanks.

Which issue do you want me to check? The specific JS menu modules slowness or the general issue of slowness given a lot of accumulated history?

If its the first - I don't use this code anymore, but you can check it easily by instantiating these menus 100 or 200 times in an HTML (you can use PHP to do this in a loop).

(As already mentioned, you can download the JS modules anylinkmenu.css, menucontents.js, anylinkmenu.js from: http://www.dynamicdrive.com/dynamicindex1/dropmenuindex.htm ).

If you want me to check slowness when history accumulated, then I don't have a specific testcase for it, but its not difficult to generate such a scenario.
I unfortunately don't have cycles.  I was hoping you will be able to do any slowness check you may perform.  The build is using a completely new non-blocking http cache implementation that is faster and doesn't cause UI hangs at all.
(In reply to Honza Bambas (:mayhemer) from comment #45)
> I unfortunately don't have cycles.  I was hoping you will be able to do any
> slowness check you may perform.  The build is using a completely new
> non-blocking http cache implementation that is faster and doesn't cause UI
> hangs at all.
comment 46 repost was for t8av (reporter)
Flags: needinfo?(t8av)
I apologize, but I failed to understand the last 2-3 technical comments.

Is there anything I can help to check this issue?

If you think that your tests show there is no more slowness due to history accumulation or due to the JS modules that mentioned above, then you can close the bug.
Flags: needinfo?(t8av)
(In reply to t8av from comment #48)
> I apologize, but I failed to understand the last 2-3 technical comments.
> 
> Is there anything I can help to check this issue?
> 
> If you think that your tests show there is no more slowness due to history
> accumulation or due to the JS modules that mentioned above, then you can
> close the bug.

I wanted to ask you to test your scenario you've originally reported this bug for with the latest Nightly and switch preference 'browser.cache.use_new_backend' to 1.  This will make it use the new cache backend that is completely UI-jank free.  I'd like to know if it helps to avoid the slowness.

Thanks.
Flags: needinfo?(t8av)
(In reply to Honza Bambas (:mayhemer) from comment #49)

> I wanted to ask you to test your scenario you've originally reported this
> bug for with the latest Nightly and switch preference
> 'browser.cache.use_new_backend' to 1.  This will make it use the new cache
> backend that is completely UI-jank free.  I'd like to know if it helps to
> avoid the slowness.

As I already wrote, the problem occurred when this JS module was instantiated 200 times in an HTML code.

My PHP code has changed a lot since then, and now I don't use that JS module as before.

If you want, I can create an HTML file which instantiates this JS module 200 times (or maybe more), and you can test it by yourself.

Is it good enough?
(In reply to t8av from comment #50)
> If you want, I can create an HTML file which instantiates this JS module 200
> times (or maybe more), and you can test it by yourself.
> 
> Is it good enough?

I am afraid this is dependent on your hardware.  My machine is relatively fast and I don't think I'll be able to reproduce.
1. I used in Firefox 24.0 to open the file attachment of this bug (which contains am HTML with the relevant JS menu + Firefox logo instantiated 200 times).

2. The HTML page loaded within approx. 3 seconds, without any hang.

3. I think it is a reasonable performance, but not an excellent performance.

4. Even if you machine is fast, you can still instantiate the JS menu many times (maybe 2000+) so you will see what I can see.

5. Is there anything else that is needed from my side regarding this issue?
Flags: needinfo?(t8av)
Thanks, I think that's all I need.  And when I here this summary, it more sound like a JS bug to me.  We know (and not do much about it... grrr...) that content scripts can very well kill the browser UI responsiveness.
1. I agree that JS can cause extreme slowness.

2. Changing the topic - Some web pages contain horrible JS. How can we protect the user? is there a way to limit the resources that a specific web page JS takes from the browser? 

3. Do you want to close this bug? If the answer is yes, then go ahead...
(In reply to t8av from comment #54)
> 3. Do you want to close this bug? If the answer is yes, then go ahead...

And if not, should it go to different component?
Flags: needinfo?(honzab.moz)
(In reply to Wayne Mery (:wsmwk) from comment #55)
> (In reply to t8av from comment #54)
> > 3. Do you want to close this bug? If the answer is yes, then go ahead...
> 
> And if not, should it go to different component?

Hmm... I think I'll try to find time and check the attached testcase (comment 52).

Please leave the bug as is.
Flags: needinfo?(honzab.moz)
(In reply to Honza Bambas (:mayhemer) from comment #56)

> Hmm... I think I'll try to find time and check the attached testcase
> (comment 52).
> 
> Please leave the bug as is.

I am not sure how this JS/Firefox bug behaves in the newer Firefox versions. It could be that you will have to check it with an older version of Firefox....
(In reply to t8av from comment #57)
> I am not sure how this JS/Firefox bug behaves in the newer Firefox versions.
> It could be that you will have to check it with an older version of
> Firefox....

Hmm.. it sounds like that is fixed then?
(In reply to Honza Bambas (:mayhemer) from comment #58)
> Hmm.. it sounds like that is fixed then?

1. Please read again previous remarks (around comment #52).

2. As I already wrote, closing this bug or not, is up to you, developers.
I don't see mention of whether there is high CPU.  If there was not high CPU, then isn't this then related to USB drive speed as Honza's observed earlier?  (and not js related)
Flags: needinfo?(honzab.moz)
I didn't find time to check on this bug so far.
Flags: needinfo?(honzab.moz)
(In reply to t8av from comment #52)
> 5. Is there anything else that is needed from my side regarding this issue?

do you recall roughly what CPU percent was used?
Flags: needinfo?(t8av)
(In reply to Honza Bambas (:mayhemer) from comment #53)
> Thanks, I think that's all I need.  And when I here this summary, it more
> sound like a JS bug to me.  We know (and not do much about it... grrr...)
> that content scripts can very well kill the browser UI responsiveness.

it will be interest to see why this issue correlates to the amount of history
This doesn't have detailed information to make this actionable
Status: UNCONFIRMED → RESOLVED
Closed: 10 years ago
Flags: needinfo?(t8av)
Resolution: --- → INCOMPLETE
Whiteboard: [needs profile]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: