Closed Bug 945120 Opened 11 years ago Closed 10 years ago

bookmarks - backups, exports are incomplete or empty

Categories

(Toolkit :: Places, defect)

Product:
Toolkit
Component:
Places
Version:
28 Branch
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: landis, Unassigned)

References

Details

(Keywords: dev-doc-needed, Whiteboard: [bugday-20131202]) 

User Agent: Mozilla/5.0 (X11; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0 (Beta/Release)
Build ID: 20131201030203

Steps to reproduce:

1: menu bar > bookmarks > Show All Bookmarks
2: menu bar > Import and Backup > [either] Backup [or] Export Bookmarks as HTML
3: 'save as' dialog opens and click Save 'button'


Actual results:

1: Bookmarks[date].json files initially show huge size 40-68+MB, then 0B (empty)
2: Bookmarks.html files are 3.x'ish MB, but Stop at line 8,743 +/-
3: Quit > Start FF > try again, different file name, same results.
*note: this is a brand new Nightly, central 'install' run from directory (Linux)


Expected results:

I should be able to open Bookmarks.html in kate (linux-kde) and edit unwanted content prior to uploading to my webserver and 'including' in a php page.
( this is the last page i edited and uploaded. this is how it should still work:
http://landisreed.com/pages/mozilla/MozillaBookmarks.php/ )
Landis.
from UX nightly, Same Problem. *.json 0 Bytes, *.html SLOWLY increases in size. It takes So Long to 'export' that if someone thinks they are 'backing up' their Bookmarks just before they Quit FF, they are being mislead. 
In 'older' firefox, the Backup Export drop down would not become active again, until the Backup or Export was done, which was a good indicator it was still working on the Backup / Export.

While writing this and other things I have to do, the Backup.html file is Still Not Done.. 10 mins.
Good thing I was writing this, or I would have Quit UX and not known Backup was NOT done.

15 mins now at 5 MB (system: openSusE 12.3 Linux, KDE 4.11.x, 4 GB RAM, 520 GB HDD 250 + free, 128 MB graphics card) Still not done.

20 mins now at 5.5 MB
25 mins now at 7.8 MB
23 mins now at 10 MB
33 mins now at 12.7 MB
35 mins now at 14 MB
36 mins now at 15.8 MB
37 mins now at 16 MB
38 mins seems finished at 16.3 MB 

Now, let's see if I can open file in kate (kde, utf-8 editor) - Yep.. 32,931 lines took 40 */- minutes. : (


the theme here is, the process gets faster (more data written as time goes on ie, less data written in beginning, writing more data later on)).

Landis
Severity: normal → major
26 Beta.

bookmarks-2013-12-02.json  3.2M
bookmarks.html             5.7M  10540 lines

It took minutes.  bug 490831, bug 824433.

The files end with a very recent bookmark, probably the last one.
Component: Untriaged → Places
Product: Firefox → Toolkit
Whiteboard: [bugday-20131202]
Is this OS/File System specific maybe?

FWIW, on Windows 7/NTFS/SSD I get

~20s for a ~10000 lines HTML export taking ~20-30 MB/s I/O Usage
~25s for a JSON backup taking ~10-18 MB/s I/O Usage

My places.sqlite stats (taken off https://addons.mozilla.org/en-US/firefox/addon/places-maintenance/ (statistics preset)):
History can store a maximum of 87202 unique pages
Table moz_places has 87346 records
Table moz_historyvisits has 162779 records
Table moz_inputhistory has 340 records
Table moz_bookmarks has 9340 records
Table moz_bookmarks_roots has 5 records
Table moz_keywords has 85 records
Table sqlite_sequence has 1 records
Table moz_favicons has 4072 records
Table moz_anno_attributes has 23 records
Table moz_annos has 3973 records
Table moz_items_annos has 2060 records
Table sqlite_stat1 has 15 records
Table moz_hosts has 8079 records
I can confirm this issue on Windows 7 64bit with Nightly 32bit.
Exporting about 20.000 Bookmarks to HTML file takes minutes,
when in other browsers it's done instantaneously or in worst cases in few seconds.



Statistics was taken with Places Maintenance 1.3 (https://addons.mozilla.org/addon/places-maintenance/)

> Integrity check
+ The database is sane
> Reindex
+ The database has been reindexed
> Orphans expiration
+ Database cleaned up
> Coherence check
+ The database is coherent
> Vacuum
Initial database size is 63424 KiB
+ The database has been vacuumed
Final database size is 63424 KiB
> Statistics
Database size is 63424 KiB
user_version is 23
page_size is 32768
cache_size is -2048
journal_mode is wal
synchronous is 1
History can store a maximum of 104338 unique pages
Table moz_places has 106366 records
Table moz_historyvisits has 134413 records
Table moz_inputhistory has 0 records
Table moz_hosts has 6105 records
Table moz_bookmarks has 29754 records
Table moz_bookmarks_roots has 5 records
Table moz_keywords has 0 records
Table sqlite_sequence has 0 records
Table moz_favicons has 4191 records
Table moz_anno_attributes has 9 records
Table moz_annos has 28618 records
Table moz_items_annos has 20695 records
Table sqlite_stat1 has 14 records
Index sqlite_autoindex_moz_inputhistory_1
Index sqlite_autoindex_moz_hosts_1
Index sqlite_autoindex_moz_bookmarks_roots_1
Index sqlite_autoindex_moz_keywords_1
Index sqlite_autoindex_moz_favicons_1
Index sqlite_autoindex_moz_anno_attributes_1
Index moz_places_faviconindex
Index moz_places_hostindex
Index moz_places_visitcount
Index moz_places_frecencyindex
Index moz_places_lastvisitdateindex
Index moz_historyvisits_placedateindex
Index moz_historyvisits_fromindex
Index moz_historyvisits_dateindex
Index moz_bookmarks_itemindex
Index moz_bookmarks_parentindex
Index moz_bookmarks_itemlastmodifiedindex
Index moz_places_url_uniqueindex
Index moz_places_guid_uniqueindex
Index moz_bookmarks_guid_uniqueindex
Index moz_annos_placeattributeindex
Index moz_items_annos_itemattributeindex
Flags: needinfo?(mak77)
Similar regarding 0B JSON backups: Bug 914229 - Cannot save bookmarks to json file.
I will investigate on slow backups in bug 824433, there is definitely something going wrong.
Assignee: nobody → mak77
Status: UNCONFIRMED → NEW
Depends on: 824433
Ever confirmed: true
Flags: needinfo?(mak77)
I think bug 824433 may have fixed the JSON cases, could you please check if you notice issues in the latest nightly?
for the HTML case, the patch in bug 968177 should help.
Depends on: 968177
I don't need to be assigned to this since there's no more work to do, I think this should be fixed btw (apart the html part that is pending review).
Assignee: mak77 → nobody
2014-02-11-03-02-01-mozilla-central-firefox-30.0a1.en-US.linux-x86_64

JSON backup happens in seconds (fast, at least compared to import, which takes minutes), but:
* directed into /tmp, it worked properly.
* directed into ~/Desktop, it only saved bookmarks-2014-02-12.json.tmp into /tmp (apparently of the right size).  Desktop is writable (export works, slowly).
sounds like an issue related to writeAtomic (that should first write the .tmp file to /tmp and them move it to the destination folder), I wonder if David has any clue about it
Flags: needinfo?(dteller)
I suspect that /tmp and the destination folder are not on the same file system, which makes it impossible to move from one directory to the other without copying. In this case, writeAtomic fails because it is not atomic anymore.
Flags: needinfo?(dteller)
is comment 11 correct for your case?
Flags: needinfo?(deletesoftware+moz)
Yes, /tmp is a tmpfs.
Flags: needinfo?(deletesoftware+moz)
I guess that may be bad not just for backups, but also for downloads and other components who first save to tmp and then try to atomic move to the destination folder...
I think I'm going to file a separate bug since it's not directly related to incomplete backups, even if I'm not sure what's the right solution. One side I don't think would be sane to provide a "fallback" inside writeAtomic, the other side it's not so nice to have to special case all of the consumers.
Well, the usual scenario is to call writeAtomic(path, data, {tmpPath: path + ".tmp"}), so as to ensure that the data is stored on the same file system. This has the drawback that it can leave .tmp files lying around in case of crash/power failure.
Filed Bug 971686

I hope this behavior across filesystems is documented in writeAtomic, cause it was a little bit unexpected from my side (I mean, it makes sense, but didn't actually think about it on first use)
You are right, it's not documented.
Keywords: dev-doc-needed
All of the known issues related to json and html exports should have been fixed, resolving as WFM.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WORKSFORME
I have this same issue but with the html export. It's going for around 30 minutes now, the html is usually 22mb, so the json would be about 9mb, but the json is fine i think ,the problem is only with html for now
please file a separate bug with the details about your issue (like the version of the browser)
You need to log in before you can comment on or make changes to this bug.