Closed Bug 858127 Opened 11 years ago Closed 11 years ago

Some downloads.sqlite databases cause the downloads dialog to be untolerably slow and crashy

Categories

(Firefox :: Downloads Panel, defect)

Product:
Firefox
Component:
Downloads Panel
Version:
20 Branch
Platform:
x86_64
Windows 8
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED INVALID

People

(Reporter: emk, Unassigned)

Details

User Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0

I have to click the slow script dialog three times before opening the download manager.
Moreover, when I selected all items in the manager to copy URLs, Firefox crashed.
(Crash report: bp-8a015a50-40c2-4d99-81d5-1368f2130404)

The old UI at least opens immediately because it loads download items progressively. Why the new UI tries to load all items at once?
Also, the old UI copied all URLs without any problems (it was 6077 entries).

To say the least, the new download manager is unusable. I had to disable the new UI by setting browser.download.useToolkitUI to true.
(In reply to Masatoshi Kimura [:emk] from comment #0)
> Moreover, when I selected all items in the manager to copy URLs, Firefox
> crashed.
> (Crash report: bp-8a015a50-40c2-4d99-81d5-1368f2130404)

Unfortunately the crash report was unable to collect a stack, may be anything.
> 
> The old UI at least opens immediately because it loads download items
> progressively. Why the new UI tries to load all items at once?

It doesn't, only visible items are loaded. On the other side the old ui tried to load all at once, but it had less entries to read, since it didn't include downloads history.

> Also, the old UI copied all URLs without any problems (it was 6077 ent

Does this happen in safe mode?
We tested it with up to 15 thousands downloads without noticing any issue, there could be some specific reason to be investigated here.
Ah now that I think about it, is your profile on a network share or using a roaming profile?
We have a bug in OS.File that is fixed in 22 regarding that.
(In reply to Marco Bonardo [:mak] from comment #1)
> It doesn't, only visible items are loaded. On the other side the old ui
> tried to load all at once, but it had less entries to read, since it didn't
> include downloads history.

It is much worse to take a time so long only to load visible elements.

> Does this happen in safe mode?
> We tested it with up to 15 thousands downloads without noticing any issue,
> there could be some specific reason to be investigated here.

I tried safe mode and got the same result. Again, crash report didin't contain the stack:
bp-8626a966-00e8-4d28-8dd7-3916e2130404

(In reply to Marco Bonardo [:mak] from comment #2)
> Ah now that I think about it, is your profile on a network share or using a
> roaming profile?
> We have a bug in OS.File that is fixed in 22 regarding that.

My profile is on my local hard drive. But I'm using the profile since Firefox 2.0. Is it related?
I noticed the new UI is showing deleted (before updating Firefox 20) items after the oldest item.
(In reply to Masatoshi Kimura [:emk] from comment #3)
> (In reply to Marco Bonardo [:mak] from comment #1)
> > It doesn't, only visible items are loaded. On the other side the old ui
> > tried to load all at once, but it had less entries to read, since it didn't
> > include downloads history.
> 
> It is much worse to take a time so long only to load visible elements.

I agree, the problem is that what you see is not what we expect, we didn't have many complains about speed, and we explicitly worked on loading many thousands entries. So there could be something specific we don't know about in your profile.
Would you mind doing some experiment?
First would be good to know if creating a new profile, copying places.sqlite to it, and then opening Firefox with it and the Library changes anything.
Second would be good to try that profile on a different computer, just to check if there's a relation.
Third, would be nice to try doing a whole cleanup with this add-on, to be sure the database is clean: https://addons.mozilla.org/en-US/firefox/addon/places-maintenance/

If in case you'd be comfortable with sharing your places.sqlite with me, you may even compress it and mail it (or a mail link to a password protected archive) to me, so I can try to debug the problem.
The last possibility would be to try using the Firefox Profiler (from dev tools) to see where the time is spent.

> I noticed the new UI is showing deleted (before updating Firefox 20) items
> after the oldest item.

yes, downloads and download history have been merged, indeed in previous versions removing downloads was not properly removing all of their history data.
(In reply to Marco Bonardo [:mak] from comment #4)
> First would be good to know if creating a new profile, copying places.sqlite
> to it, and then opening Firefox with it and the Library changes anything.

If I copied only places.sqlite, Download Panel displayed only deleted garbages.
If I copied both places.sqlite and downloads.sqlite, the problem reproduces.
Actually it was sufficient to copy only downloads.sqlite to reproduce the problem.

> Second would be good to try that profile on a different computer, just to
> check if there's a relation.

I have no other machines at hand.

> Third, would be nice to try doing a whole cleanup with this add-on, to be
> sure the database is clean:
> https://addons.mozilla.org/en-US/firefox/addon/places-maintenance/

It looks like nothing changes.

Now, if I enable new UI and open Download Panel and restart Firefox, all download histories disappear...
Here's an execution log of Places Maintenance:
> Integrity check
+ The database is sane
> Reindex
+ The database has been reindexed
> Orphans expiration
+ Database cleaned up
> Coherence check
+ The database is coherent
> Vacuum
Initial database size is 143360 KiB
+ The database has been vacuumed
Final database size is 143360 KiB
> Statistics
Database size is 143360 KiB
user_version is 21
page_size is 4096
cache_size is 1024
journal_mode is wal
synchronous is 1
History can store a maximum of 999999 unique pages
Table moz_bookmarks has 6558 records
Table moz_bookmarks_roots has 5 records
Table moz_keywords has 4 records
Table sqlite_sequence has 1 records
Table moz_favicons has 10369 records
Table moz_annos has 6370 records
Table moz_anno_attributes has 13 records
Table moz_items_annos has 1353 records
Table moz_places has 246615 records
Table moz_historyvisits has 344562 records
Table moz_inputhistory has 1099 records
Table sqlite_stat1 has 15 records
Table moz_hosts has 18470 records
Index sqlite_autoindex_moz_bookmarks_roots_1
Index sqlite_autoindex_moz_keywords_1
Index sqlite_autoindex_moz_favicons_1
Index sqlite_autoindex_moz_anno_attributes_1
Index sqlite_autoindex_moz_inputhistory_1
Index sqlite_autoindex_moz_hosts_1
Index moz_bookmarks_itemindex
Index moz_bookmarks_parentindex
Index moz_bookmarks_itemlastmodifiedindex
Index moz_places_faviconindex
Index moz_places_hostindex
Index moz_places_visitcount
Index moz_places_frecencyindex
Index moz_places_lastvisitdateindex
Index moz_historyvisits_placedateindex
Index moz_historyvisits_fromindex
Index moz_historyvisits_dateindex
Index moz_annos_placeattributeindex
Index moz_items_annos_itemattributeindex
Index moz_places_url_uniqueindex
Index moz_bookmarks_guid_uniqueindex
Index moz_places_guid_uniqueindex
Index moz_favicons_guid_uniqueindex
(In reply to Masatoshi Kimura [:emk] from comment #5)
> If I copied only places.sqlite, Download Panel displayed only deleted
> garbages.

But the problem doesn't happen, right?

> If I copied both places.sqlite and downloads.sqlite, the problem reproduces.
> Actually it was sufficient to copy only downloads.sqlite to reproduce the
> problem.

OK, so the issue is the downloads themselves, not downloads history or Places.
It's strange, we should be clearing those at the first shutdown, maybe we can't complete a shutdown properly though.  Would you be comfortable with mailing me that downloads.sqlite file that causes the issue?

For now you could just remove downloads.sqlite from your profile and that should be enough to solve the hang issue.
Summary: New download dialog is untolerably slow and crashy → Some downloads.sqlite databases cause the downloads dialog to be untolerably slow and crashy
(In reply to Marco Bonardo [:mak] from comment #7)
> (In reply to Masatoshi Kimura [:emk] from comment #5)
> > If I copied only places.sqlite, Download Panel displayed only deleted
> > garbages.
> 
> But the problem doesn't happen, right?

Right.

> > If I copied both places.sqlite and downloads.sqlite, the problem reproduces.
> > Actually it was sufficient to copy only downloads.sqlite to reproduce the
> > problem.
> 
> OK, so the issue is the downloads themselves, not downloads history or
> Places.
> It's strange, we should be clearing those at the first shutdown, maybe we
> can't complete a shutdown properly though.  Would you be comfortable with
> mailing me that downloads.sqlite file that causes the issue?

Sent downloads.sqlite.

> For now you could just remove downloads.sqlite from your profile and that
> should be enough to solve the hang issue.

As I said in the last sentence of comment #5, download history are washed out when restarting Firefox.
(In reply to Masatoshi Kimura [:emk] from comment #8)
> > For now you could just remove downloads.sqlite from your profile and that
> > should be enough to solve the hang issue.
> 
> As I said in the last sentence of comment #5, download history are washed
> out when restarting Firefox.

Do you use permanent private browsing or clear history on shutdown?

To clarify, downloads are stored in 2 places:
- downloads.sqlite keeps the in-progress downloads, these are (should be) cleared on firefox shutdown (Apart the unfinished ones).
- places.sqlite keeps downloads history, these survive shutdown but work like any other kind of history, so private browsing or clear history on shutdown will affect its persistence.

Anyway, the contents of downloads.sqlite would be cleared by the new code, so I don't think anything would change. Even solving the cause of hang problem, I fear you may be losing some downloads history if it's all in downloads.sqlite.
> Do you use permanent private browsing or clear history on shutdown?

No.
It looks like that Firefox failed to migrate the download history from downloads.sqlite somehow. Is it possible to migrate the history manually?
Nope, there is no migration :(
Downloads history is stored in history (places.sqlite) from Firefox 8.0 in addition to downloads.sqlite. The 2 databases shoul basically have duplicated data and downloads.sqlite removal should not change anything.
I have no explanation why in your case the history part is missing, so far.
Looks like I misunderstood the backend change.
I restored places.sqlite and downloads.sqlite from a backup. Once entries are removed from downloads.sqlite by restarting Firefox, Download Panel becomes much faster.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → INVALID
I wonder why they were not removed on the first startup though.
Didn't you ever restart firefox after the first startup with the new downloads experience?

I will look at your database regardless, it's possible we can (and should) apply some performance improvements.
(In reply to Marco Bonardo [:mak] from comment #13)
> I wonder why they were not removed on the first startup though.
> Didn't you ever restart firefox after the first startup with the new
> downloads experience?

Maybe. I usually keep running Firefox unless extensions or Firefox itself ask me to restart.
that makes sense, thanks.
You need to log in before you can comment on or make changes to this bug.