slow performance concerning keyword/url usage

RESOLVED WORKSFORME

Status

()

defect
RESOLVED WORKSFORME
10 years ago
8 years ago

People

(Reporter: xtc4uall, Unassigned)

Tracking

({regression})

Trunk
x86
Windows XP
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

Posted file screencast
after landing of Bug 455555 i noticed a performance regression related to keyword usage and loading urls in the location bar (urls of already bookmarked items).

the regressions are indicated by the spinning icon, high I/O and higher-than-normal CPU time. trying to click other UI elements of Fx while the querying happens, makes the UI freeze/lock up/unusable.

i attached a Flash screencast of the issues; those cases are the rather fast ones, other queries take some seconds more.
i wonder if the querying is profile-able for normal human beings helping to resolve where the bottlenecks are ... .

my places stats script output:
 "livemark_container_cnt":1,
 "livemark_child_cnt":37,
 "visits_per_place_avg":1.8633233074440265,
 "visits_per_place_max":3094,
 "visits_per_place_min":1,
 "places_visited_unique_cnt":127337,
 "visit_date_oldest":"Sat 08 Mar 2008 18:59:31 GMT",
 "visit_date_newest":"Tue 11 Aug 2009 22:51:44 GMT",
 "visit_date_avg":"Tue 09 Dec 2008 13:59:24 GMT",
 "tag_cnt":56,
 "tag_size_avg":30.589285714285715,
 "favicon_data_size_avg":543.1869942196532,
 "favicon_nonnull_data_size_avg":543.1869942196532,
 "favicon_nonnull_cnt":3460,
 "session_unique_cnt":15017,
 "frecency_first_bucket_visit_cnt":2591,
 "frecency_second_bucket_visit_cnt":5812,
 "frecency_third_bucket_visit_cnt":14015,
 "frecency_fourth_bucket_visit_cnt":39449,
 "frecency_first_bucket_place_unique_cnt":1957,
 "frecency_second_bucket_place_unique_cnt":3999,
 "frecency_third_bucket_place_unique_cnt":8971,
 "frecency_fourth_bucket_place_unique_cnt":24443,
 "history_expire_days":999,
 "history_expire_days_min":999,
 "history_expire_sites":40000,

maybe the keyword related part is the already filed Bug 509607, but dupes are cheap ;).
(Reporter)

Comment 1

10 years ago
errrm, the missing part of the script output:

 "places_file_size":80031744,
 "moz_anno_attributes_cnt":21,
 "moz_annos_cnt":1587,
 "moz_bookmarks_cnt":6808,
 "moz_bookmarks_roots_cnt":5,
 "moz_favicons_cnt":3460,
 "moz_historyvisits_cnt":237270,
 "moz_historyvisits_temp_cnt":3,
 "moz_inputhistory_cnt":140,
 "moz_items_annos_cnt":6191,
 "moz_keywords_cnt":93,
 "moz_places_cnt":127535,
 "moz_places_temp_cnt":2,
 "bookmark_cnt":6346,
 "folder_cnt":431,
 "separator_cnt":31,
 "dynamic_container_cnt":0,
 "bookmark_nontag_cnt":4633,
I suspect this is highly related to bug 509048.
(Reporter)

Comment 3

10 years ago
i tried your tryserver build mentioned in bug 509048 comment 5 (-> https://build.mozilla.org/tryserver-builds/sdwilsh@shawnwilsher.com-try-f1cd14654e89/) but it does not fix the issues :-/.
any of the issues or all the issues?
(Reporter)

Comment 5

10 years ago
Posted file screencast 2
both of above mentioned issues (keyword + <string> & pasting url) still happen.

to demonstrate the impact on the UI (un-)snappiness i made another screencast:
- paste a url (what is a bookmark) into location bar via menu popup (ctrl+v makes no difference though)
- while the querying happens (spinning icon), i pressed ctrl+t 3x for opening new tabs

=> notice the behavior on the UI and the higher-than-acceptable CPU usage/time raising(!).

this all happens with the tryserver build as well as with today's MC nightly in opposite to the last MC build without the checkin of Bug 455555.

also i don't have any malware scanning software running what could have an impact on sqlite querying/reading.

this all is - of course - not noticable with e.g. fresh places.sqlite made of a new Fx profile.

fwiw, my places.sqlite file weighs 77,3 MB atm.
(Reporter)

Comment 6

10 years ago
Posted file screencast 3
furthermore i have to add a third aspect what regressed:
- typing (random) chars or actual URLs in the location bar

that triggers the same symptoms like the other two issues do.
just have a look at the two screencast examples i zipped up.
There has been lag related to typing in a url and hitting enter for 2-3 weeks now. I didn't bother filing a bug as I figured it was related to something already in the pipe. It is damn annoying though and I hope someone can find a fix for it. :)
(Reporter)

Comment 8

10 years ago
Brian, that would rather be Bug 509048 as Shawn mentioned. you might try the tryserver build mentioned in comment 3. my issues happen without (or before) triggering url loading using the enter key (or the "go" arrow) as it is shown in the screencasts.
Interestingly enough, I just loaded up the 8/19 build of Minefield 3.7a1pre and the problem I mentioned is currently not present. I don't see anything in the build that would have caused the problem to go away though. But I'm happy either way. Thanks.
(Reporter)

Comment 10

10 years ago
Shawn, i just tested a hourly build with the patch for Bug 509048 included [Minefield/3.7a1pre ID:20090827120110: http://hg.mozilla.org/mozilla-central/rev/82e2e7d9c18f].

=> pasting/entering urls/keyword seach terms got better, but yet not perfect. there's still kind of freeze/lock up in the UI due to (so i guess) the querying ("spinning icon") - even without hitting "enter" and triggering the url loading :-/
How does it compare to before?
(Reporter)

Comment 12

10 years ago
as i wrote: better, "snappier" ... but still far away from "i'd ship Fx 3.6 with this" ;)
(Reporter)

Comment 13

10 years ago
ok, i did some more testing:

* doing a backup of my (now) 78 MB heavy places.sqlite gives me a slim 2,4 MB JSON file. restoring this within a new profile gives me a new 4,2 MB places file. ok, because of Bug 423126 without any favicons, but this wouldn't explain the gap of ~74 MB.

so i had a look at the history in library and saw that almost all history entries are missing:

->
the original places file contains history from "today" back to entries with "Older than 6 months". as you can see from comment 0 i set
 "history_expire_days":999,
 "history_expire_days_min":999,
 "history_expire_sites":40000,
and the oldest entry dates back to
"visit_date_oldest":"Sat 08 Mar 2008 18:59:31 GMT",
and a total of
"moz_historyvisits_cnt":237270,
entries.

->
the newly created places file just contains a few entries splitted to june and may

=>
this leads me to the assumption that the history entries of my places file are kind of corrupt and therefor not restorable via the SQLite -> JSON -> SQLite way.

=>
and now the positive thing: above mentioned performance regression are no more - *none* - with the new places file.

=>
i guess that somehow querying a places file which has "corrupt" history entries makes the querying suck very hard, troubling the UI.

=>
a PRAGMA integrity_check; gives:
"rowid 163439 missing from index moz_places_url_uniqueindex"
"rowid 192834 missing from index moz_places_url_uniqueindex"
"wrong # of entries in index moz_places_url_uniqueindex"
"wrong # of entries in index moz_places_lastvisitdateindex"
"wrong # of entries in index moz_places_frecencyindex"
"wrong # of entries in index moz_places_visitcount"
"wrong # of entries in index moz_places_hostindex"
"wrong # of entries in index moz_places_faviconindex"

so is this worth any further digging or should i say good-bye to my beloved history and favicons, start all over again and close this as INVALID? :(
try running a REINDEX on your database (make a backup of it first in case we need it for debugging) then check again integrity.
(Reporter)

Comment 15

10 years ago
what i did now:

- run PRAGMA integrity_check; on the origignal places.sqlite file
	=> fails as mentioned in comment 13; issues from comment 0 etc. present
- run REINDEX and then recheck with PRAGMA integrity_check;
	=> integrity is now ok; issues from comment 0 etc. still present
- export to JSON file, reimport in a new profile
	=> history is lost; issues from comment 0 etc. are no more

checking the newly created places.sqlite file with SQLite manager tells me that
the tables
	moz_favicons
	moz_historyvisits
	moz_inputhistory
are empty.

my assumption is now that - somehow - the size/structure of the moz_historyvisits table causes the issues mentioned in comment 0 etc..
XtC4UaLL, can you still reproduce this?
Using keyword I see only a brief (<1/2 sec) spike in CPU of maybe 40%. my places is 41MB
I'm resolving this now since I moved on with a new Places Database and haven't encountered any Symptoms like above again.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.