193567 - Automatically backup all profile data at beginning of each session in case of crash and offer method of restoring profile from backup data after crash.

Status: RESOLVED INCOMPLETE

(Core Graveyard :: Profile: BackEnd, enhancement)

Description

[Jason Bassford]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3b) Gecko/20030216
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3b) Gecko/20030216

I was exiting Mozilla when my XP system crashed.  Upon restarting my computer
and Mozilla, I discovered that I'd lost all of my bookmarks and (almost) all of
my preferences.  With the exception of some toolbars not being shown that I'd
disabled in my profile, it was as if I'd just created a fresh profile.

What I find interesting is that I hadn't modified any preferences or bookmarks
during that session.  Nor do I see any reason why Mozilla shouldn't open those
files for writing ONLY when they are modified.  There should be no reason for it
to have them open for writing upon a regular exit (or even one that causes a
crash) that I can see.  The only reason I can think of for the dataloss is that,
for some reason, they had been opened when the computer crashed.  (If there is
some reason why a Mozilla exit routinely opens bookmark and prefs files, then it
should do something like backing them up first in case something like this does
occur.)

This may well be the wrong component, and I have a suspicion that this should be
duped to any existing bug but I couldn't locate it.

I will refrain from making the severity here Critical until I can get some
feedback from somebody who's familiar with Mozilla's exit behaviour and can
comment knowledgeably.

Reproducible: Didn't try

Steps to Reproduce:
1. If somebody can come up with a method of ensuring a system crash (blue
screen) while I'm exiting Mozilla I'd be happy to try to reproduce this.

Comment 1

[Jason Bassford]

FYI: I don't believe that this is a duplicate of the MozDev project
http://recall.mozdev.org/ - since, as far as I know, that deals with just saving
open windows/tabs in the event of a crash not open bookmark and preference files.

Also, it would seem to me that creating an initial backup of these files would
be useful not just on exit (if that's the problem) but whenever there is any
write access.  Then, when Mozilla is run, if it detects the backup file it could
assume that some kind of crash has occurred (if it was written out properly the
backup file would have been deleted) and offer to restore the data...

Comment 2

[Jason Bassford]

I'm not sure if this bug should be marked as Critical.  I know that it involves
dataloss, but this may be (arguably) the fault of the OS (which could cause
dataloss to any file on the hard drive) rather than Mozilla itself doing
something incorrectly.

This bug could also be considered a Normal severity *enhancement* bug (for
backing up data prior to file writes), as a protection feature, depending on how
you look at it.

I'm going to change the Summary to something like "Backup profile data during
each session in case of crash." but only after hearing if we are supposed to be
doing file writes on exit or not.  (If so, then we might actually have two bugs
here - one for the profile data backup, one for not doing the file writes on
exit, unless there's some really good reason for this.)

Comment 3

[R.K.Aa.]

Isn't there already a prefs.bak made?

Comment 4

[Jason Bassford]

I do see a prefs.bak.  But there is no bookmarks.bak, nor a .bak for any of the
other profile data files.  Also, the restoration of these files after a crash
shouldn't depend on user intervention but should be automatic either with a
confirmation dialog ("A crash has been detected, do you wish to have your backed
up data restored?") or simply just happening in the background if it's obvious
that data has been corrupted (by having the backup files in existence, when they
should have been deleted after a successful exit/write).

Alternatively, if the .bak files are to exist on a "permanent" basis (so crash
detection as above wouldn't work) then perhaps there could be a "Tools ->
Restore Backup Data" menu item that the user could initiate.

Comment 5

[R.K.Aa.]

I think this is covered in bug 164244 and bug 170539.
For other profile related bugs, see tracker bug 123929.

Comment 6

[Jason Bassford]

Both of those bugs seem to be at best subsets of this one and, at worst, just
related.  One refers to bookmarks, the other to prefs, and both are talking
about ways of preventing corruption in the first place.  This one is about
bookmarks *and* prefs - plus all other dynamic files in the profile directory
(cookies, passwords, etc.) - in other words, any file that gets written to - and
deals with recovering from crashes/corruption, not necessarily preventing it in
the first place.

Moreover, those talk about seemingly random corruption, this bug is specifically
about keeping a (at least temporary) backup of specific data files whenever
Mozilla does a file open for writing purposes or just backing up ALL such files
at the start of each session.

For clarity, I'm going to change the Summary and Severity.

I'll also state what I see to be the simplest method of ensuring data recoverabilty:

1. At start of each Mozilla session copy the contents of the profile's data
directory (all files) to a /backup directory.  (Each new session will overwrite
previous data stored there.)

2. Create a Tools -> Restore Backup Data type of menu item that can be run manually.

3. It shouldn't be too hard to create a file every time Mozilla is started which
is deleted upon a successful exit.  If Mozilla is run and the file is THERE it's
a good indicator there's been a system crash and Mozilla can automatically
prompt the user as per 2.: "There appears to have been a system crash.  Would
you like to recover your profile data?"

Comment 7

[Jason Bassford]

Since focus has changed from making backup copies of files during file writes
(and questioning why bookmark and preferences files are written to during exit),
changing component to Profile Manager BackEnd.

Comment 8

[Nicholas Allen]

I could have sworn there was already a bug for this but I can't find it.

Comment 9

[Grace Bush]

see bug 22689- duplicate if you agree

Comment 10

[Jason Bassford]

This bug is about it happening without any manual intervention on the part of
the user.  I shouldn't have to launch Mozilla then immediately export on the
assumption that a crash might occur.

So, no, it's not a duplicate.  However, I WILL mark is as blocking this bug,
since a fix for this bug might well incorporate that bug as at least a portion
of the solution.  (Also, the "export" would exist only until a successful exit
was detected - if Mozilla exits *without* a crash, the last thing it should do
is delete the exported (backup) data, since it's only a temporary save - unless
the user *specifically* requests an export.)  The remaining portion would be to
detect that a crash has occurred and to automatically prompt the user for
recovery (essentially import, although it shouldn't be called that at the time)
at the start of the next session.  (So crash recover would happen prior to the
next automatic export/backup.)

Comment 11

[Bill Mason]

*** Bug 201606 has been marked as a duplicate of this bug. ***

Comment 12

[Dave Dyer]

Making a backup once per session probably wouldn't help, since
one wouldn't normally know there was any damage until you start
the next session, which would destroy the backup.  In fact, in
the case where I got zapped, I restarted the browser several times
before I guessed that my bookmarks had been destroyed.

-- so the best strategy would be to make an explicit backup that
would be preserved only if something catastrophic happened before
the operation finished, and which explicitly would NOT superceed
an existing backup.   Eudora does something like this when rewriting 
mail files.

Comment 13

[Jason Bassford]

> one wouldn't normally know there was any damage

No, you misunderstand this bug. (And it's about a system, behind the scenes,
backup that serves the same kind of function as automated data integrity in RAID
systems - it's NOT the same bug as being able to make a backup and keep it for
future reference that would require manual/user action.)

On browser startup, check for the existence of a system backup.  If a backup
exists, then a crash HAS occurred, so prompt the user - or not, this could still
be automatic behaviour.  (A crash has occurred and you've most likely lost data.
 Should data from your previous session be restored?) The user doesn't need to
be aware if there was damage or not - the browser itself will assume it to be
the case. (if there were no system backup, no crash could have occurred).  The
backup remains on disk, in case the current session crashes again.

(If there was no backup at startup, then the FIRST thing that's done is to
create one of the current "good" state.)

The very LAST thing the new session does on exit is to delete the backup.

So - the only time there will ever be a backup in existence on browser startup
is if the previous session was not exited properly, and corruption can be
assumed.  (The computer crashed, the browser crashed, etc.)  You'll know at the
very next session - you won't be running the browser multiple times before you
discover it because it won't leave it up to you to figure it out.

Comment 14

[safeco]

history.dat appends URLs contemporaneously, but if Moz is not exited gracefully,
the history file's 'footer' info will not be appended.  The next time Moz is
launched it will be able to read the history file's data, but lacking the
'footer', Moz will consider the file invalid and scrap it, destroying months'
worth of potentially valuable, path-finding 'breadcrumbs' when Moz is closed!

This is an unacceptably high level of volatility, perhaps best addressed via (in
order of preference):

1. Endow Moz with the ability to synthesize/recreate the missing 'footer' data?
2. Change history.data format to something less volatile (ie no footer
used/needed), stop trying to aggregate the data by day, and don't bother storing
trivial ancillary data (ie long URLs of doubleclick.net images the pages contained)
3. Make a backup copy of history.dat (verifying that it is whole/intact) each
time Moz is launched, with transparent ability to revert to last good
history.dat in case history.dat is found to be corrupted.

Thank you for your attention to this VERY costly/frustrating deficiency.

Comment 15

[Mike Connor [:mconnor]]

*** Bug 209216 has been marked as a duplicate of this bug. ***

Comment 16

[Oleg Sidletskiy]

*** Bug 214586 has been marked as a duplicate of this bug. ***

Comment 17

[Jo Hermans]

*** Bug 226720 has been marked as a duplicate of this bug. ***

Comment 18

[Christian Flatscher]

Hi,

I suggest to add a feature to Mozilla / Firebird that allows to create backups
of files such as bookmarks.html, Personal toolbar, mail prefs, news prefs, etc.

How about adding a menu entry to the "Edit / Preferences..." menu underneath the
"Navigator" branch called "Settings backup"?

This should contain tick boxes of what somebody wants to backup, to where and
how often (time interval) this should happen. Also a restore feature should be
introduced with a selection option what and which backup should be restored.

This may solve some of the issues where an OS freezes or crashes unrecoverably.

--Christian

Comment 19

[Jason Bassford]

> This should contain tick boxes of what somebody wants to backup, to where
> and how often (time interval) this should happen.

Perhaps useful, but it would have no impact on this bug other than, maybe, as a
dependency.  The backup that this bug describes should happen automatically
without any user intervention, whenever Mozilla is launched.  (And a restore
after a crash would automatically be prompted on the first launch after that.)

Comment 20

[Christian Flatscher]

Hmmmm. I don't agree. My Mozilla did not store anywhere backups of the bookmark
file. After the crash half of the entries were lost. Luckily I had a not so
old backup of the file.

It is nice to have some backups when Mozilla is launched. Yet I do not believe
that anybody closes Mozilla and starts it again if a new URL has been added to
the bookmark file - just to be on the save side. Would you do this?

I also have not been asked to load any backups once I got my system back into
working state. I assume this feature is there and enabled by default - but it
did not show up on my system since only the bookmarks.html file was affected.

I think it is better to have such a functionality in place since this makes
life for users much more easier. Anything else means that I have to manually
safe everything with a third party program such as WinZIP / WinRAR / whatever.

And I believe I have read already in some other places about similar incidents
and complaints.

Comment 21

[Jason Bassford]

What are you talking about? <grin>  Your comments don't seem to have any 
bearing on this bug.  Re-read comment 0 again, as well as the follow up 
comments.  This bug is ONLY about the system automatically backing up all 
profile data at the beginning of each session - and then deleting it again upon 
proper exit.  If, when starting a subsequent session, the backup data is found, 
that means that the previous session was not exited properly (there was a 
crash) and you are prompted to restore it.  This should all happen in the 
background and it doesn't have anything to do with manual backups / restores 
(which are covered in other bugs).  (If you never saw anything like this 
before, it's because it doesn't exist - this bug wants to create it.)

Comment 22

[Christian Flatscher]

Jason,

thanks for your update. I can see that you completely missed the point here -
see also the beginning of this bug report. Quote from the first entry:

"...I'd lost all of my bookmarks and (almost) all of my preferences..."

Are you able to read and understand?

I just wanted to have an extended a feature to make Mozilla / Firebird safer
against failures.

I interpret your last update in the fashion that there is no will to fix this
problem or implement such a requested feature.

With this response / outlook I will now manually backup all vital Mozilla files
and file this problem under "will never be fixed and is not taken serious".

Thaks - although "thanks" is in such a case a totally inappropriate word.

--Christian

Comment 23

[Jason Bassford]

> Quote from the first entry

Yes, I filed this bug.  I'm the reporter.  I'm perfectly aware of what I said. 
<grin>

> I interpret your last update in the fashion that there is no will to
> fix this problem or implement such a requested feature.

Er, no.  My last statement simply said that the functionality that this bug 
wants to get implemented doesn't exist yet.  (Which seems pretty self-evident, 
or else this bug would be closed.)

Your comments have not been on topic for the scope of this bug (more in line 
with bug 22689) - in that they have dealt with manual backups and restores, 
something that this bug isn't directly concerned with.  Any further comments of 
this type should be taken offline (feel free to email me directly) rather than 
confusing the issues trying to be resolved here.

As for the technicalities of this issue, I can think of at least one 
partial "workaround" - create a batch file that backs up your profile 
directory, launches Mozilla, then deletes the backup directory on Mozilla exit 
if Mozilla gives the appropriate exit code (implying that it exited properly 
rather than crashing).  The batch file could be further extended to look for 
the existence of this backup data at its start - informing the user that a 
potential crash occurred (since the folder was NOT deleted) and giving them the 
option of aborting the rest of its processing so that a manual restore of the 
backed data can be made.  However, I would only know how to code something like 
this in a Win32 environment, am not entirely sure about exit codes and if the 
Windows process would actually return something back to the batch file (which 
would have to remain running in the background until Mozilla exited or crashed) 
or not.

Comment 24

[:aceman]

Comment 0 and comment 14 have good points regarding file handling. The open
files thing is bug 231606.

Comment 25

[Stewart Gordon]

*** Bug 232236 has been marked as a duplicate of this bug. ***

Comment 26

[Elmar Ludwig]

*** Bug 298215 has been marked as a duplicate of this bug. ***

Comment 27

[Elmar Ludwig]

*** Bug 299861 has been marked as a duplicate of this bug. ***

Comment 28

[edward ash]

I would think if FF/moz opens ALL the files being used, that it would SAVE the
parameters and KEEP the configuration.
Not keep it active for a crash to kill that portion of the program.
Opening the Bookmarks, and keeping them as ACTIVE is not the best idea I have seen.
Better to Copy them to ram, and Leave the original ones alone.
Upon makeing new ones, Write it to file, and reopen a copy in ram.
Edit them, do the job wirting/deleteing, and copy the new list to ram.
this is STILL a bad Bug, and I wont Use FF/Moz as my main browser for that reason.

Comment 29

[Steve England [:stevee]]

*** Bug 315613 has been marked as a duplicate of this bug. ***

Comment 30

[edward ash]

OK,
Im running 1.5.
and the thing has lost my configuration, and my bookmarks.
There have been no REAL problems except with MY.YAHOO, it almost always has a problem there is in yahoo.mail.
WELL, I found my bookmarks HISTORY, buried in my C: drive.  WISH it wasnt there, as I have multi drive system and TRY to keep C: for the OS only.  If it Messes up, I only need to recover the OS.
This is a problem that MANY programs have, that they dont keep there PARTS inside there own location.  A safe system could be made with a Partitioned or Multi drive system, that is easy to recover.  But IF, the OS, or C: drive is corrupted and it is needed to FORMAT the drive to recover, ALL the drivers and config are LOST.  Leaving things on the C: drive also makes them EASY to find for Bots, Virus, and other BAD THINGS..

Comment 31

[Ria Klaassen (not reading all bugmail)]

*** Bug 328234 has been marked as a duplicate of this bug. ***

Comment 32

[Jason Bassford]

Firefox 2.0 beta 1 now has this.  For the Firefox implementation, see bug 328154.

Comment 33

[Jason Bassford]

Actually, I may be talking out of my hat - in looking at it more closely, it seems to be about browser session restore (sites visited, etc.) not bookmarks, preferences, and so on.  Although, I haven't actually looked in detail at all of the bits...

Comment 34

[Benjamin Smedberg]

This bug is filed in a bugzilla component related to pre-Firefox code which no longer exists. I believe it is no longer relevant and I am therefore closing it INCOMPLETE.

If you believe that this bug is still valid and needs to be fixed, please reopen it and move it to the Toolkit:Startup and Profile System product/component.