145048 - offline actions should run during sync

Status: RESOLVED FIXED

(SeaMonkey :: MailNews: Backend, defect)

Description

[Alec Flett]

So after weeks of trial and error, I finally figured out how to playback the
offline actions - but I've found that the whole system is quite precarious. The
actual functionality is tremendously useful and powerful, but it is so hard to
use that its almost worthless.

As I see it (and maybe I've got this wrong too!) - you do a download/sync, and
then go into offline mode (or click on the offline icon and then download/sync
from the dialog) - that part works fine. the problem is basically that the next
time you click the online button, the playback begins - whether or not you can
actually talk to the server or not.. not to mention that if you go online from
the browser window, the playback will never occur (or at least I sure as heck
can't figure out if its actually happening) - in any case, you only seem to get
one shot at the playback, then its all lost and all your folders go back to the
state they were on the server. You only get one chance, then you're screwed!

Here are a few situations that have bit me:

Today, I came in after doing a whole bunch of offline work, and plugged in my
laptop. I clicked the "offline" button in the mail status bar - but I forgot to
plug it into the network - I got about 5 "failed to connect to
nsmail-1.mcom.com" dialog boxes, before I realized what was happening - so I
plugged in the network cable, and the rest of the playback seemed to go ok - so
about 20 of my 25 offline folders seemed to actually sync correctly.. the other
folders seem to have gone back to the way they were before I went offline, 2
hours earlier!

Another way this has affected me is if I connect my laptop to the Internet, but
don't have access to my company's mail server (i.e. outside the firewall) I
don't want to loose all the offline work I did for that server!

Yet another odd case is when I'm using Netscape's WebMail and their
authentication system fails to connect to the server (which happens about 20% of
the time, but that's netscape's bug :)) then I loose all offline sync info for
that server, and I'm basically screwed for that sync.

And finally, once or twice I've accidentally clicked the "offline" icon while
I'm  actually offline, sending me back into online mode - once again, all my
offline changes are lost! 


So anyhow, here's what I think should happen, which should pretty much solve all
these problems:
1) every time you go online, from any window, the mail window should ask "Would
you like to sync your offline work with the server now?" so at least the user
knows that the sync is happening! (or optionally, if you go online from the
browser window, the mail window would just "remember" that you had just gone
online, and when it got focus, it could ask you)
2) when the offline sync begins, if the first connection to a server fails, it
should ask something along the lines of "I can't seem to connect to <server> -
would you like to continue syncing with this server?" instead of this
ininformative "can't connect to <server>" - then it could continue trying to
sync with the other servers. 

cc'ing jglick for UI/behavior advice. I don't think this feature has been
completely thought out, or at least nobody has tried to use it in real-world
situations :(

Comment 1

[grylchan]

Hey Alec,
Just trying to understand your situation better here.

you're right that if you go online via the browser,
it doesn't sync with mail. See bug 92939.

There is another bug, that if you have unsent mesgs, you quit,restart,
login to messenger, you should be prompted to send unsent mesgs
if you have that pref enabled, see bug 115024

What offline prefs do you have set either in Main pref area?
(when going online: ask me, auto send unsent mesgs, dont send
 when going offline: ask me, auto download, dont download)

or Mail/news account settings offline & disk space pref section?:
    -make mesgs in my inbox available when I'm offline 
     [I'm wondering if you're running into this bug 103400 ]
    -create new folders, select them for offline  

I'm curious as to what offline changes were lost/lose sync info/
loose all offline work for that server when you say

>Another way this has affected me is if I connect my laptop to the Internet, but
>don't have access to my company's mail server (i.e. outside the firewall) I
>don't want to loose all the offline work I did for that server!

>Yet another odd case is when I'm using Netscape's WebMail and their
>authentication system fails to connect to the server (which happens about 20% 
>of the time, but that's netscape's bug :)) then I loose all offline 
>sync info for that server, and I'm basically screwed for that sync.

>And finally, once or twice I've accidentally clicked the "offline" icon while
>I'm  actually offline, sending me back into online mode - once again, all my
>offline changes are lost! 

Mesgs you thought were downloaded aren't? Mesgs deleted get undeleted?
Labels? Message status goes from New to read?

Mesgs that get downloaded stay downloaded whether you connect
to the server or not. I think if you are sending unsent mesg but
not connected to mail server and/or network, they won't disspear
from unsent mesg folder.

I understand about maybe better informative error mesgs.

thnx

Comment 2

[Alec Flett]

sorry, what I lose are all the actions that I perform on messages - mostly
deleting them or moving them from one folder to another, as well as reading a
message and having it get marked as read.

When I lose this information, I have to go re-read all the messages that I
already read and re-delete all the messages that I already deleted.

I highly suggest someone try this, for at least 2 weeks: try to use offline mode
exclusively, all day - reconnect at in the morning when you arrive, then
disconnect. reconnect at lunch, and disconnect. Pretend you don't know the magic
incantation that makes the offline playback happen - try different situations -
go online while your network cable is unplugged, go offline and then online
again , etc... basically my experience is that the user experience simply isn't
robust - there is a very, very specific set of actions that make everything
happen just right, and if you mess up one step, you lose any offline "work" that
you might have done.

also, my prefs are to ask me every time I go offline/online, and to make my
messages available in my Inbox

Comment 3

[David :Bienvenu]

Offline playback is supposed to happen both when you go online in mail (I don't
know why it was decided not do to this when you go online from the browser,
though I suspect it was because we've been keeping the browser separate from
e-mail) and when you select a folder that has offline events. So, for example,
once you select your inbox, events in the inbox should get played back. That
doesn't seem like a magic formula to me - presumably, you're going to select
your inbox at some point, and the playback should just happen, pretty much
transparently without a lot of special steps on your part. I'll check to make
sure that's still true, but it's been that way since 4.0.

Re the lost playback when you go online when you are in fact, not online, that's
also been that way since 4.0 - unfortunately, the offline playback code is a zoo
of iterations, recursion, and asynchronousness so I think the original
implementer thought it was safer to remove the offline op on the attempt rather
than the completion of the event. Obviously, this is bad, but you should just
lose one offline action, not every offline action. E.g., we might lose the fact
that you read the first message, not the fact that you read 20 messages and
deleted 20 others. I can try to rewrite that code not to lose the first event
but it's scary. It's very possible that instead of losing events, we will now
try to do attempts multiple times, with the resulting errors, alerts, etc. I
think the first thing to do is figure out why you're losing all the events and
not just the first event.

Comment 4

[Alec Flett]

hrm... I don't know about 4.x, because I didn't use the offline feature much (I
worked on the unix client, we had no offline :))
but in the current world, I almost ALWAYS loose all my playback info (i.e. where
always means about 90% of the time).. or so it would appear.

my one other thought is this - say somehow I keep loosing the playback/state
information for the INBOX - maybe mozilla is re-downloading all my headers, and
re-filtering them? maybe there's a bug somewhere where the INBOX is losing all
its playback data? 

in any case, it seems like I am constantly going back and re-reading/deleting my
messages once I reconnect to the network. Honestly, I feel like I must be the
only user of the offline features, and I wonder if someone (QA :)) just banged
on it for a week, and tried to use it as a real client (Rather than just going
through regression tests on known behavior) then the real usability issues could
be nailed down.

I'm sure that under the hood, things are working as designed, but its simply not
usable in its current state. :(

Comment 5

[David :Bienvenu]

trust me, you're not the only offline user - I know from the bugs filed and
fixed, and lengthy e-mail correspondance with some of them that other people are
using it extensively (one of them on Solaris, even!) I'm not sure if there's
something different about your useage or profile, or if there's been some sort
of regression.

Comment 6

[David :Bienvenu]

I did a little research - what's happening when you go online w/o a network
connection is that we try to playback the first offline op, which fails, and is
removed. But what's also happening is that we're clearing the flag on the folder
that says there are offline events for that folder, and that's why the other
events don't get played back when you connect to the network. The offline events
are still there, and in fact, if you go offline, make a change, and go back
online, your most recent change, and all except the lost change from the first
offline session, will get played back (I'm not suggesting that as a workaround,
but just as evidence that it's the losing the "folder has offline events" flag
and not the offline events themselves).

I suspect I should just go for the complete solution and try not to delete
offline events until they complete. It will take a while, however.

Comment 7

[Alec Flett]

see, there is a magic formula :)
but thanks for doing the research! I can't wait to have an easy to use offline
system :)

Comment 8

[grylchan]

Alec, what build are you using? Trunk/branch/pr1?
I assume on linux.

This a old 4.x migrated profile or new profile?
I assume all imap mail accounts?

David, the only bug about offlie/no network connection is
this one (which Alec maybe running into) bug 103400

Comment 9

[David :Bienvenu]

One possible easier solution would be to have the offline playback first attempt
to select the folder it's playing back offline ops in, and only if that
succeeds, start playing back and removing ops. It's no extra overhead on the
server, since the folder's most likely going to need to get selected anyway to
playback the offline ops.

Comment 10

[Kathleen :Brade]

This is probably what happened to me yesterday (causing me to lose about 5 hours
of offline e-mail actions).  At about 5am today I was thinking about this
problem and was going to suggest something like David suggests in comment 9 (do
a dummy action that can essentially be ignored except to flag network problems).

I think this is an important issue to address for the next release due to
likelihood of failure and criticalness of issue (dataloss == loss of organization)

Comment 11

[David :Bienvenu]

I'd prefer that we not cheapen the dataloss keyword and save it for situations
where we really do lose data - otherwise, it becomes less meaningful. And I can
promise you that it's not getting into this release since much safer fixes for
other serious offline bugs have been shot down w/o a second thought.

Comment 12

[Michael Buckland]

Discussed in mail news bug meeting.  Decided to minus this bug.

Comment 13

[David :Bienvenu]

*** Bug 180484 has been marked as a duplicate of this bug. ***

Comment 14

[jquinn]

I was mis-classifying my experience with this bug as bug 67172 - if you read the
comments there you can see me growing more and more exasperated and annoyed with
what I saw as an important bug being ignored. 

I would agree that this is data loss. I have, on at least one occasion,
responded twice to the same message because I thought it was new. Since my
internet connection is unreliable, I consistently lose my offline changes, and
so offline is categorically unusable.

If you spend hours using a word processor to write a document and it doesn't
save, that's data loss. If you spend hours using a mail program to do the most
fundamental part of what a mail program does - classify and respond to
communication - then have to do it all over again, that's data loss too. If my
hard disk file table got munged, and I still had all my files but they lost
their datestamp and were in the wrong folders, that's data loss. If I still have
all the same mail messages but they lose their state and are in the wrong
folders, that's data loss. Status and location might be only a few dozen bytes
of actual information, but it is the most important dozen bytes, and it is
totally lost.

Comment 15

[jquinn]

ps. Comment 9 would fix most of my problems, but not the case where the
connection is broken in the middle of playback. I would personally be happy if,
in that case, the entire playback for that folder started from the beginning
next time I connected. ie, the "has events" flag should not get cleared from the
folder unless the server at least responds (errors are fine) to every event in
the playback. Then, you wouldn't need a dummy select action, either. Still, a
fix as in comment 9 would make me ecstatic, offline would at least be usable again. 

Big picture: a lot of mail clients suck in their offline capabilities. IMO
fixing this one bug would take Mozilla from far-behind-the-pack to
ahead-of-the-pack.

Comment 16

[David :Bienvenu]

Attachment: proposed fix

This fix makes it so we advance to the next folder in case of errors playing
back operations, which means in the case of selecting a folder, the lite select
will fail and we won't lose any offline operations. In the case of the
connection failing in the middle of playing back operations, we might still
lose one operation, but we shouldn't lose any more than that.

Comment 17

[David :Bienvenu]

Cavin, can I get a review? thx.

Comment 18

[(not reading, please use [e-mail] instead)]

Comment on attachment 106616 [details] [diff] [review]
proposed fix

sr=sspitzer

Comment 19

[Cavin Song]

Comment on attachment 106616 [details] [diff] [review]
proposed fix

r=cavin.

Comment 20

[David :Bienvenu]

fix checked in.