142278 - Offline synchronization silently fails to complete or seems to but does not download what it should.

Status: RESOLVED INCOMPLETE

(SeaMonkey :: MailNews: Backend, defect)

Description

[Arnaud Le Hors]

When I do 'Download/Sync Now' the result is completely unpredictable. I keep 
calling it but it sometimes doesn't even seem to start. When it starts it seems 
to stop in the "middle" of downloading the messages. Calling it again, 
sometimes makes it go a little further.
This isn't a new bug. It was already there in Netscape Messenger (Netscape 4). 
I switched to the newer version in hope this would be solved but it's not.

I use a Sun IMAP server.

It looks like if a network operation fails (timeout or whatever), the whole 
process is stopped. I would expect it to retry, if it doesn't work move to the 
next msg, the next folder, keep trying to get as much as possible, and in any 
case display some error message. I get nothing. It just stops, failing to go 
offline at the end as requested.

Comment 1

[David :Bienvenu]

what build are you using? if it's not the latest mozilla trunk build, then
please try the latest trunk build?

Comment 2

[grylchan]

also how many mesgs are you trying to synchronize?
couple 100 or couple 1000 mesgs?
I assume over a high speed network rather than a 56 dial up?
And are these mesgs rather large in size?

Comment 3

[Arnaud Le Hors]

I just downloaded the latest version:
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0.0+) Gecko/20020506
and tried it again. It seems a little better in the sense that it seems to keep 
trying by only giving up on the current folder and moving to the next one. It 
also ends switching to offline as requested.

I tried to figure if the messages it gives up on had any special 
characteristics but found none. Sometimes it's a big message with an 
attachment, sometimes it's a very short message with just a few lines of text. 
And the next time around it typically succeeds downloading the very same 
messages it gave up on earlier.

I'm using a fast network connection. I do have quite a lot of data involved 
though. I'm talking about 20+ folders, containing several thousands of messages 
and a total amount of data of over 100 Megabytes.

Thanks for your consideration.

Comment 4

[David :Bienvenu]

An imap protocol log might help:

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

The logs will probably be quite large, and You'll probably want to edit out a
lot of it before attaching it. It would probably be most useful to attach two
logs, one showing it giving up on a message/folder, and one showing it actually
downloading the message (or you could put both download synchronizations in one
log).

It would also be helpful if you indicated which message the download failed on. Thx.

Comment 5

[Arnaud Le Hors]

Ok, I got a log and it shows something interesting. Here is a short extract 
which I think shows a relevant part, I'll be happy to provide more if needed.

...
2012[2af4480]: sun1.sanjose.ibm.com:S-mail/w3c-xml-core-
wg:CreateNewLineFromSocket: )

2012[2af4480]: sun1.sanjose.ibm.com:S-mail/w3c-xml-core-
wg:CreateNewLineFromSocket: * 1195 FETCH (UID 3748 RFC822.SIZE 2749 RFC822 
{2749}

2012[2af4480]: sun1.sanjose.ibm.com:S-mail/w3c-xml-core-wg:STREAM:CLOSE: Normal 
Message End Download Stream

2012[2af4480]: sun1.sanjose.ibm.com:S-mail/w3c-xml-core-wg:PARSER:Internal 
Syntax Error: %s: * 1195 FETCH (UID 3748 RFC822.SIZE 2749 RFC822 {2749}
...

Comment 6

[Andrew Treloar]

I'm also seeing this behaviour under OSX. I see it in RC2 and in the Nightly
Builds. Typically folders that are marked for download do not get downloaded
(but not consistently) and after the sync finishes, the client is actually
offline (even if I didn't check that box in the dialog).

Comment 7

[David :Bienvenu]

I think a little bit more of the log might be useful, Arnaud. A little bit more
at the top, and at the bottom would help. I'd like to see what command we're
sending, and what data we're getting back. Thx.

Comment 8

[Henry Jia]

CCing to Calvin & me

Comment 9

[David :Bienvenu]

Arnaud, do you have more of the log you can attach?

Comment 10

[Arnaud Le Hors]

Attachment: Imap log

Sorry for the delay. Here is a more complete log. I limited it to what
corresponds to the synchronization of a single folder, and collapsed sections
with many similar lines to just a few lines surrounding the changes in the log.
I hope this helps.

Comment 11

[Antonio.Xu]

I have researched the bug. I found some very interesting things. I found this 
problem will happen when mozilla try to parser the response from imap server 
unsuccessfully. At the first, mailnews will send a command for fetch all mails 
and get those response from imap server, then mailnews will pasering every the 
reponse and try to download every mail's body in a circulatory funtion. But 
when mailnews parsering the response from imap server unsuccessfully,mailnews 
will stop download all mails and quit from the circulatory function. I think we 
can find a way to skip the wrong response, and try to parser next the reponse 
from imap server and try to download its body. At the finally we can show the 
sync result to user. Another way is only add a alert for remind user that the 
sync is failed,when parsering failed. I think we can not avoid this problem due 
to some netwerk operation fails, but we should decrease damnify. I will try to 
find a way to skip wrong response and download next mail, then I will try to 
inform user email sync finish but incompletely.

Comment 12

[David :Bienvenu]

my guess is that this is a server problem - perhaps we're not getting the
correct number of bytes back from the server when we download the message.
Arnaud, if you can set me up with a test account that exhibits this problem, I'd
be happy to try it out. Antonio, if you're going to suggest that we just ignore
syntax errors, that would be incorrect.

1848[2937168]: sun1.sanjose.ibm.com:S-mail/apache-xerces-c-dev:STREAM:OPEN Size:
160229: Begin Message Download Stream

1848[2937168]:
sun1.sanjose.ibm.com:S-mail/apache-xerces-c-dev:CreateNewLineFromSocket:
Received: from localhost (maki.sanjose.ibm.com [9.43.222.48])


<... huge number of lines truncated ...>

1848[2937168]:
sun1.sanjose.ibm.com:S-mail/apache-xerces-c-dev:CreateNewLineFromSocket: )


1848[2937168]:
sun1.sanjose.ibm.com:S-mail/apache-xerces-c-dev:CreateNewLineFromSocket: * 1387
FETCH (UID 9949 RFC822.SIZE 46568 RFC822 {46568}


1848[2937168]: sun1.sanjose.ibm.com:S-mail/apache-xerces-c-dev:STREAM:CLOSE:
Normal Message End Download Stream

1848[2937168]: sun1.sanjose.ibm.com:S-mail/apache-xerces-c-dev:PARSER:Internal
Syntax Error: %s: * 1387 FETCH (UID 9949 RFC822.SIZE 46568 RFC822 {46568}

Comment 13

[Antonio.Xu]

Hi David,I know if we ignore syntax errors,it would be incorrect. But I need 
your advice about what behavior will correct when parsering failing. I will 
follow your advice for fix the bug.

Comment 14

[David :Bienvenu]

parsing should never fail - the server should generate correct protocol, and we
should parse protocol correctly.

Comment 15

[Antonio.Xu]

I think the bug is due to network operation fails, we should find a way to 
decrease damnify. I think parsering can not runing successfully due to 
network's problem. so we should find a way remind user what was happen.

Comment 16

[David :Bienvenu]

That is not what's going on here with this log. Network errors don't cause
syntax errors in the imap protocol code.

Comment 17

[Antonio.Xu]

Arnaud,Could you add comments about your imap server's detailed information, 
please verify the bug again with mozilla1.0. I need your infomation for fix the 
bug.

Comment 18

[James Rome]

I think this bug is still a problem on 1.5a, Win XP. I have a huge IMAP mailbox
that I need to download and ALWAYS use offline because the powers that be
decreed that we have to move to Exchange. 

It is unclear where mozilla actually puts the offline folders (which should be
settable IMHO), but in my profile there is an ImapMail folder with two folders
for my server, one with a .sbd extension. Both have four of my hundred or so
folders in them, even though I selected the whole set for offline download. I
probably have 1 GB of mail + attachments for 20 years stored there!

So, how do I get it downloaded?

And, is there a way to always use ONE of my 6 different mail servers offline?
Since I cannot get any new messages there there is no reason to actually log on
once I can download my old mail.

Comment 19

[David :Bienvenu]

James,
  The offline messages are stored in the same location as the .msf files for
your imap folders, which is settable. Did you do a File | Offline |
Download/Sync Now? That should download the contents of all your imap folders on
your server that are configured for offline use.
  The only way to use the server offline is to actually tell Mozilla that you
are offline - otherwise, we'll try to update the folder when you select it, to
see if there are new messages. It sounds to me like you really want to use your
Exchange Server as a pop3 server. I believe that's possible, though I don't know
for sure.

Comment 20

[denis]

Attachment: mozilla imap log

I have the same unpredictable behavior here. I have multiple levels of folders.
Seems that Inbox gets downloaded fine, but not the rest of the folders. They
are all marked for offline.

Clicking individual folders' "Download now" in the properties/offline does
work. Its only the File->Offline->Download/Sync Now that does not seem to work.


Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5a) Gecko/20030619

Sorry if my problem is unrelated. Symptoms are very similar (other then the
log)

Comment 21

[Matthew Elvey]

*** Bug 223115 has been marked as a duplicate of this bug. ***

Comment 22

[Matthew Elvey]

Bug 223115 mentions a new log I sent in, and that not only does offline sync
sometimes silently not complete, sometimes it SEEMS TO COMPLETE but did not
download what it should have. Re comment 2: ~50 folders, ~180MB mailbox.  ADSL,
sometimes with WiFi, though it's not needed to get errors, I suspect packet loss
contributes to the problem.  Enhancing Summary to reduce dupes.

Comment 23

[Michael Zimmermann]

*** Bug 236532 has been marked as a duplicate of this bug. ***

Comment 24

[David :Bienvenu]

I suspect the problem with this server is that it's not giving back the right
number of bytes for the message:

1386 FETCH (UID 9948 RFC822.SIZE 160229 RFC822 {160199}

It says the msg size is 160229 but then says it's going to return 160199 bytes.
Our parser should be trusting the 160199 value - I'm not sure if it is or not.
I'll look into it.

Comment 25

[Josef Rosenlehner]

This bug happens with Linux also.
Therefore "OS" should be changed to all.

Comment 26

[Matthew Elvey]

Changed.

Since the last 2-3 versions of Mozilla have been out, I've been seeing a new
error frequently, which says (I'm paraphrasing): 'that folder is in use, try
again later'.  I'm not sure what causes it. (Perhaps it's running Mozilla on two
machines, but I doubt that's it.) Download does seem to be getting less flaky
for me.  

I'd imagine that since Bug 221023 was fixed, a new log (i.e. from a current
build) would be helpful. Any coders (e.g. Bienvenu) want a new log?

Josef, in what build/version are you seeing what specific behaviour? 

Comment 27

[Josef Rosenlehner]

Sorry for answering your question so late.
I have the problems with the Linux-versions of Mozilla 1.7.3 and 1.8.a4.
(In reply to comment #26)
> Changed.
> 
> Since the last 2-3 versions of Mozilla have been out, I've been seeing a new
> error frequently, which says (I'm paraphrasing): 'that folder is in use, try
> again later'.  I'm not sure what causes it. (Perhaps it's running Mozilla on two
> machines, but I doubt that's it.) Download does seem to be getting less flaky
> for me.  
> 
> I'd imagine that since Bug 221023 was fixed, a new log (i.e. from a current
> build) would be helpful. Any coders (e.g. Bienvenu) want a new log?
> 
> Josef, in what build/version are you seeing what specific behaviour? 

Comment 28

[Josef Rosenlehner]

Maybe there is a memory leak responsible for this misbehaviour, I looked at some
talkbacks created by Mozilla 1.8a4 and seems that all ends in the statement
outErr = morkEnv_kOutOfMemoryError;
See also TB1333297V, TB1328458K, TB1326322K, TB1326322K, TB1310472M, TB1305876K,
TB1288892G, TB1284647K, TB1251817G, TB1250889G, TB1249444Z and some more

Comment 29

[David :Bienvenu]

Josef, those logs all have to do with downloading for newsgroups (or at least,
all the ones I looked at). Do you have a lot of newsgroups configured for
offline use?

Comment 30

[Josef Rosenlehner]

(In reply to comment #29)
> Josef, those logs all have to do with downloading for newsgroups (or at least,
> all the ones I looked at). Do you have a lot of newsgroups configured for
> offline use?

Yes i've subscribed to about 600 newsgroups on 2 news server.

Comment 31

[David :Bienvenu]

my guess is that we're not closing the news db's during offline news download.

Comment 32

[David :Bienvenu]

Attachment: proposed fix for newsgroups

need to close db's as we go, like we do for imap.

Comment 33

[David :Bienvenu]

I've checked in a fix so that we'll close newsgroups as we go, into the trunk
and aviary branch.

Comment 34

[Wayne Mery (:wsmwk)]

David, did this fix only the leak and the original problem is outstanding?  

(In reply to comment #12)
> ... perhaps we're not getting the  correct number of bytes back from the server when we download the message.
> Arnaud, if you can set me up with a test account that exhibits this problem, I'd
> be happy to try it out. 

reporter is gone afaict

Comment 35

[David :Bienvenu]

the fix only changed the news memory bloat - the original problem is outstanding, though I think it was a server problem. (there are a bunch of issues described in this bug, actually)

Comment 36

[Wayne Mery (:wsmwk)]

Amaud, can you still reproduce with trunk build?
 backup your profile first
 http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-trunk/

if you see it, please attach imap protocol log

Comment 37

[Wayne Mery (:wsmwk)]

=> incomplete