Last Comment Bug 352998 - Mail never deleted from pop server(due to fail to keep consistency of popstate.dat when connection loss); Then eventually Server fills after months and email rejected
: Mail never deleted from pop server(due to fail to keep consistency of popstat...
Status: RESOLVED FIXED
[sg:nse dos]
: fixed1.8.1.5
Product: MailNews Core
Classification: Components
Component: Networking: POP (show other bugs)
: Trunk
: x86 Windows XP
: -- critical with 4 votes (vote)
: ---
Assigned To: David :Bienvenu
:
Mentors:
: 352999 (view as bug list)
Depends on:
Blocks: 263142
  Show dependency treegraph
 
Reported: 2006-09-16 14:01 PDT by geopilot
Modified: 2010-11-16 15:09 PST (History)
12 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
[checked in]possible fix (10.99 KB, patch)
2007-05-12 12:16 PDT, David :Bienvenu
mscott: superreview+
Details | Diff | Review
[checked in]proposed fix (with some other cleanup) (15.48 KB, patch)
2007-06-04 15:49 PDT, David :Bienvenu
mscott: superreview+
Details | Diff | Review
minimal diff (3.29 KB, patch)
2007-06-04 15:57 PDT, David :Bienvenu
mscott: approval1.8.1.5+
Details | Diff | Review

Description geopilot 2006-09-16 14:01:27 PDT
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041217
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041217

CRITICAL problem with "leave messages on server for x days"
using the following option
under menu Tools: Account settings
check box for
Account settings
in "server settings"
check "leave messages on server" and "for at most x days" substituting say 30 days for x

1. When using thunderbird on mindspring or earthlink (which are standard pop3 servers) thunderbird never keeps track of how long the emails it has read have been on the server and it never deletes them. This means your email quota eventually fills and you stop getting email.

At some point your email box fills and you must manually delete all emails (which for most people is technically impossible without tech support so they will just switch away from thunderbird) and results in loss of emails not yet downloaded between last thunderbird use, deletion trama, and next thunderbird use (often requires you to log in to webmail).

People must be allowed to leave messages on their server for several days so they can synchronize all received emails betwen several computers or phones etc which may not log on to the owners pop server everyday . (at least that is the main reason I use it and for short term backup in case of email loss on home computers between backup schedules).

This has been an ongoing issue through the past several versions and maybe for ever. I have seen many posts about it on the internet but never the bug boards becuase like me I guess a lot of people don't want to have to register just to report a bug.

This is a major problem in core general email client functionality. It's a deal killer for an other wise incredible product.
It is a critical core funtion of any email program and is pretty much a deal killer for email powerusers.

You only notice this problem after your emailquota fills and your box begins to reject emails so many people may not notice it for months. It's the AIDS bug for email clients. Infected now - only learn later.

2. different problem same cause?
Additionally sometimes thunderbird redownloads all emails that have already previously downloaded.
This seems to happen if I am running thunderbird on two computers with leve email on server set and both access the server.

possible cause?

I beleive Thunderbird is not correctly and independently of the server keeping track of emails it has already downloaded from a server and is not tracking how long they have been left on the server. Each manifestation of thunderbird on each computer must keep track independently of the emails it has downloaded.

Most email programs have the option to have your email client download either the last message you have read from the server regardless of whether thyou did it on the computer you are using at the moment (which I presume is indicated by the server itself somehow)
or the other option is downloading starting formt he last email you have read on the current computer (which the currect computer email client must be tracking) The latter choice insures that you will have all emails on each computer you operate from which is essential for business when you need access to all previous emails someone has written you no matter which computer you are on.

Reproducible: Always

Steps to Reproduce:
1.under menu Tools: Account settings
check box for
Account settings
in "server settings"
check "leave messages on server" and "for at most x days" substituting say 30 days for x
check your email.
after months your email server will fill or you can see via webmail that your emails are never getting deleted from the server after the specificed days you set for them to be deleted.
This eventually happens to EVERYBODY who uses leave messages on server for x days. I tmay take months for peopels boxes to fill especially gmail but it is a critical certain problem!

3.

Actual Results:  
email server fills. emails begin to be rejected. people dont know it and they lose critical emails!
messing with the leave messages on server settings in thunderbird may cause thunderbird to Retreive duplicates of ALL emails still on server AGAIn leaving you with thousands of duplicates as well.

SO obviusly thunderbird is 
1. not keeping track by itslef of which messages it downloaded and how long ago and then telling the server to delete the ones past keep time.
also obviously when you chenge thuderbird leav emessages on erver settings thunderbird is redownloading everything on the server not recognizing it already downloaded thiose that were left on the server.

HUGE MASSIVE MESS!

Expected Results:  
messages should be deleted froms erver after the speicfied days

this is huge.
it should be ina  category of "primary basic functionaltiy" but you don't have one. however this means Thunderbird can't do one of the most basic email retrieval functions. I can't reccomend it to my company clients if it can't.

I love thunderbird but had i known it had this hidden problem that only appears after a long period I problaby wouldn't have swicthed to it.

The bug reporting process is so onerous that i have had to wait to report this but it is reported all over newsgorups on the internet and thunderbird is getting a bad rep from it.

Please fix!!


PLEASE
Comment 1 Bob Clary [:bc:] 2006-09-16 14:23:46 PDT
*** Bug 352999 has been marked as a duplicate of this bug. ***
Comment 2 Bob Clary [:bc:] 2006-09-16 14:25:36 PDT
not security sensitive
Comment 3 WADA 2006-09-16 23:59:49 PDT
(Q1) What is your Thunderbird version? Or mail&news of Mozilla 1.7.x?

When "leave messages on server" is used, mail's status is kept in popstate.dat file in mail directory for each mail account. 
  - status, uidl, dateReceived
    Where status is KEEP=='k', DELETE='d'
                    TOO_BIG='b'    (when download size limit is used)
                    FETCH_BODY='f' (when header only download)
    UIDL : Unique IDentifier Local
And mechanism of deletion is simple and roughly next.
 (A) When downloading, get all UIDLs for mails held in server's mail box
     by LIST and UIDL command.
 (A-1) If a UIDL is not found in popstate.dat, download it, and save the UIDL
       in popstate.dat with status of "K", and save the UIDL in X-UIDL header
       of the mail.
 (A-2) If UIDL kept in popstate.dat is not found in response(list of UIDLs)
       from server (this means already deleted at server), the UIDL is
       removed from popstate.dat because no need to track any more.
 (A-3) If status of a UIDL in popstate.dat is "D", ask server to delete it
       by "DELE" command, then remove the UIDL from popstate.dat.
 (B) When deletion of a mail is requested (by filter, by option settings for
     deletion from server such as date base option or action base option),
     change status of the UIDL in popstate.dat to "D".

Do next.
(1) Keep backup of popstate.dat file.
(2) Start Thunderbird, and Get NSPR log for POP3, 
    and terminate Thunderbird, then save NSPR log file.
      See http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#pop
      Don't forget to replace "protocol:5" by "POP3:5".
(3) Keep backup of popstate.dat file.
 
(Q2) Is there any entry of status="D" at step (1)?
(Q3) Check server response for LIST & UIDL command in NSPR log and actions
     after getting list of UIDL.
     "DELE" command is issued? Or no "DELE" for entry of status="D"?
(Q4) Is there any difference between contents of (1) and contents of (3)?

(Q5) Do you use Global Inbox?
     If yes, account for "Global Inbox" is "Local Folders"? 
(Q6) Do you check "Until I delete or move them from Inbox" option?
(Q7) Check folder property of Inbox, and see "Retention Policy" tab.
     What is set? (It may be a feature from Tb 2.0)

Check result of following test.
 (1) Create a filter rule of "IF subject is XX??XX, Delete from POP server",
     and put it at top of message filters for the mail account.
 (2) Create a filter rule of "IF subject is YY??YY, Delete from POP server",      
     and put it at top of message filters for "Local Folders".
 (3) Send two mails to your self, mails of Subject=XX??XX and Subject=YY??YY
 (4) Connect to the POP3 server
(Q8) Will the mails be deleted from server?

By the way, bugzilla.mozilla.org is not cutomer support center nor customer help center nor customer claim accepting center, long long explanation about what will happen after problem you are experiencing is not required.
Please describe what is happening/what is occuring clearly, instead of writing complaints, to make problem analysis efficient.
Comment 4 Mike Cowperthwaite 2006-09-17 11:01:28 PDT
Dupe of bug 231541?
Comment 5 geopilot 2006-09-17 17:06:08 PDT
(In reply to comment #3)
here are my answers


(Q1) What is your Thunderbird version?

thunderbird

answer
version 1.5.0.5 (20060719)



When "leave messages on server" is used, mail's status is kept in
popstate.dat


answer: I found it. with thunderbird open and running all entries are K (some should be D right since some should have expired or would deletion
action already have taken place once thunderbird automatically accessed
my server and downloaded email.

in the log file i generated it gets  all 18000 uidls. it never issues a delete.

but in the popstate.dat there are only k anyway so i guess i wouldn't enter a dele to the server.


I have "leave on server at most 30 days" checked . i do NOT have "don't delete until i delete from inbox" checked




file in mail directory for each mail account.
  - status, uidl, dateReceived
    Where status is KEEP=='k', DELETE='d'
                    TOO_BIG='b'    (when download size limit is used)
                    FETCH_BODY='f' (when header only download)
    UIDL : Unique IDentifier Local
And mechanism of deletion is simple and roughly next.
 (A) When downloading, get all UIDLs for mails held in server's mail box
     by LIST and UIDL command.
 (A-1) If a UIDL is not found in popstate.dat, download it, and save
the UIDL
       in popstate.dat with status of "K", and save the UIDL in X-UIDL
header
       of the mail.
 (A-2) If UIDL kept in popstate.dat is not found in response(list of UIDLs)
       from server (this means already deleted at server), the UIDL is
       removed from popstate.dat because no need to track any more.
 (A-3) If status of a UIDL in popstate.dat is "D", ask server to delete it
       by "DELE" command, then remove the UIDL from popstate.dat.
 (B) When deletion of a mail is requested (by filter, by option
settings for
     deletion from server such as date base option or action base option),
     change status of the UIDL in popstate.dat to "D".

Do next.
(1) Keep backup of popstate.dat file.
done
(2) Start Thunderbird, and Get NSPR log for POP3,
    and terminate Thunderbird, then save NSPR log file.
      See
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#pop
      Don't forget to replace "protocol:5" by "POP3:5".
done
(3) Keep backup of popstate.dat file.
done
(Q2) Is there any entry of status="D" at step (1)?
no
(Q3) Check server response for LIST & UIDL command in NSPR log and actions
     after getting list of UIDL.

   "DELE" command is issued? Or no "DELE" for entry of status="D"?

no DELE issued
(Q4) Is there any difference between contents of (1) and contents of (3)?
no

(Q5) Do you use Global Inbox?
no i created named account for mindspring call geopilot@mindspring.com . non e of it is under global inbox but local still exists
I do have another email from eudora i imported into local folders. but that only backs up old emials imported from eudora. there is no account server access from it.

     If yes, account for "Global Inbox" is "Local Folders"?
(Q6) Do you check "Until I delete or move them from Inbox" option?
no

(Q7) Check folder property of Inbox, and see "Retention Policy" tab.
     What is set? (It may be a feature from Tb 2.0)
dont delete any messages is checked. Server defaults is NOT checked. Is this the reason? does folder properties really refer to server properties accessed by that folder account?


Will have to try below this later if what i said already doesn't help you.




Check result of following test.
 (1) Create a filter rule of "IF subject is XX??XX, Delete from POP
server",
     and put it at top of message filters for the mail account.
 (2) Create a filter rule of "IF subject is YY??YY, Delete from POP
server",
     and put it at top of message filters for "Local Folders".
 (3) Send two mails to your self, mails of Subject=XX??XX and
Subject=YY??YY
 (4) Connect to the POP3 server
(Q8) Will the mails be deleted from server?

By the way, bugzilla.mozilla.org is not cutomer support center nor customer
help center nor customer claim accepting center, long long explanation about
what will happen after problem you are experiencing is not required.
Please describe what is happening/what is occuring clearly, instead of
writing
complaints, to make problem analysis efficient.

Comment 6 geopilot 2006-09-17 17:12:15 PDT
in my account inbox folder (not global i set up named one)

does folder properties really refer to server properties accessed by that folder 
account rather than the emails contained in that inbox folder?
If so it is poorly named. Should "use server defaults" for that folders properties be checked if I set "leave message on server at most 30 days" in thunderbird options?
 
Comment 7 WADA 2006-09-17 18:28:09 PDT
(In reply to comment #5)

> > (Q7) Check folder property of Inbox, and see "Retention Policy" tab.
> >      What is set? (It may be a feature from Tb 2.0)
> dont delete any messages is checked. Server defaults is NOT checked. Is this
> the reason? does folder properties really refer to server properties accessed
> by that folder account?

Perhaps, but I'm not sure. Can you test with "Use server default" checked?

Another question. 
Earthlink has Web mail service too, in addtion to usual POP3 server service.
Is the Web mail service you are using is "additional Web mail access to already using POP3 mail box?" type?
Or "POP3 access to Web mail thru proxy software or service like proxy"?
(You said "POP3 server", so I think former.)
Comment 8 David :Bienvenu 2006-09-17 18:47:42 PDT
the disk space and offline settings have nothing to do with deleting message from the server. In order to delete messages from the pop3 server, you need to go into the account settings for the server (tools | account settings) , pick server settings, and under leaves messages on server, check the box that says "for at most XX days", and pick the number of days you want to leave messages on the server for. 
Comment 9 David :Bienvenu 2006-09-17 19:16:04 PDT
can you e-mail me your prefs.js file? This works fine for me, and for everyone else, as far as I know, so I suspect there's something going on in your setup that's different. If your prefs.js looks OK, we probably will need you to generate a pop3 protocol log.

You wouldn't expect to see a deleted entry in popstate.dat very often - once we delete the message from the server, it doesn't show up in popstate.dat for very long because next time we list messages on the server, the server doesn't tell us about the deleted messages, right?
Comment 10 David :Bienvenu 2006-09-17 19:18:48 PDT
One last point, which wouldn't explain your problem, but just for completeness sake, we start counting the age of the message from when it is downloaded from the server to TB, *not* when it is received on the pop3 server. In other words, we start counting the days we've left a message on the server from the day when we first downloaded it.  This avoids the situation where you say leave messages on server for 10 days, and then go on vacation for 11 days, and then startup Thunderbird, and it deletes all the messages on the server.
Comment 11 WADA 2006-09-17 19:47:45 PDT
Additional question to geopilot :
What is your every day operation?
If you do "start Tb, download mails, then shutdown Tb immediately", and if popstate.dat checking for "for at most x days" option is kicked after 5 minutes of starting of Tb or download, similar situation on "Junk Purge" in the past can occur...
Comment 12 geopilot 2006-09-17 22:53:48 PDT
several answers to seveeral peoples different questions

first some basics 
I have 200 MB in email server storage. I get about 1000 or more emails a day so i can have 18000 emails sitting on my server like i do right now if i leave them on for 30 days. 

Webmail and email are different for earthlink i beleive. There is no connection and i never use webmail except to go online and empty all my emails when the 200 mb gets full becuase thunderbird never emptys it. so maybe once a month or two. 

Quick thought!
Is a range variable being overflowed by the 18000 number or the 30000 emails i have in my thunderbird? ? is it an 16 bit number variable by any chance that can only hold 16000 and is getting overflowed? (I archive all my emails from years ago How big are your variables handling email counts?)

next: if I reset my days on server I think what will happen is thunderbird will redownmload all messages left on server. That has happened before when I did that and i need to avoid it becuase i can't get rid of the doubles. frankly i don't want to trigger another one right now.

That is an associated bug I'll bet. maybe the  program does not preserve the popstat.dat file after you change the leave message on server preferences and close the prefernces box and that causes everything to be redownloaded becuase thunderbord doesn't know they were already downloaded once?

back to the original problem.

It's not just me. I searched the net and sevral people are having this problem at mozillazine. My father is also having the same problem and just had to get mindspring support to empty his email box becuase his were bouncing. he has thunderbird on his computer at another location with a completely different mindspring email ( and in no way tied to mine).

it is possible that at this momnet my emails are not yet older than the 30 day period i have set THIS TIME. But it has happened before and i cleared my emial box so it may not be past 30 days yet. I will check. If that was the case that would explain no "D"'s in the popstat.dat file.

anyybody need anything else?



I emailed prefs.js to requester

re:
One last point, which wouldn't explain your problem, but just for completeness
sake, we start counting the age of the message from when it is downloaded from
the server to TB, *not* when it is received on the pop3 server. 
good thinking but it messes up how long youre stuff stays on the server if you are gone for a month. with a large volumes of email that can lead to rejected emails which is the worst possible situation (becuase you may lose critical communications and not know it)

I usually anticipate this issue anyway and reset my time to leave messages for much longer than i have been away.
A better solution would be if time since last email downloads >= leave messages on server days a window pops up asking if you would like to leave meassges on server for a few days.



Comment 13 geopilot 2006-09-17 23:13:42 PDT
checking via webmail I have emails on the server starting at july 3 2006
i have 400 mb storage
my leave messages on server time is set for 30days.

i did not understand the comment below about fast open and close of tb however i usually leave thunderbird open for a good while once opened. Are you saying it doens't have time to fully process the popstate.dat file for lots of messages if you shut it down right away?

When does the popstate processing start? when it opens? when it checks for new mail? when it is closed?

Additional question to geopilot :
What is your every day operation?
If you do "start Tb, download mails, then shutdown Tb immediately", and if
popstate.dat checking for "for at most x days" option is kicked after 5 minutes
of starting of Tb or download, similar situation on "Junk Purge" in the past
can occur...


Comment 14 geopilot 2006-09-17 23:42:46 PDT
The results of your test m-wade (quoted below)  are that XX??XX is deleted from server but YY??YY is not deleted from the server.


Check result of following test.
 (1) Create a filter rule of "IF subject is XX??XX, Delete from POP server",
     and put it at top of message filters for the mail account.
 (2) Create a filter rule of "IF subject is YY??YY, Delete from POP server",    
     and put it at top of message filters for "Local Folders".
 (3) Send two mails to your self, mails of Subject=XX??XX and Subject=YY??YY
 (4) Connect to the POP3 server
(Q8) Will the mails be deleted from server?
Comment 15 WADA 2006-09-18 00:17:56 PDT
(In reply to comment #13)
> Are you saying it doens't have time to fully process the popstate.dat file...
Nope. Question to know whether same situation to Bug 228512(Junk Purge case) can occur or not, on "for at most XX days" of "Leave message on server".
Comment 16 geopilot 2006-09-18 00:31:04 PDT
I don't use junk purge.  I do have filters that move things from one inbox to a subfolder of the inbox depending on who they come from or subject line. In some cases they are moved and in others they are copied.

(In reply to comment #15)
> (In reply to comment #13)
> > Are you saying it doens't have time to fully process the popstate.dat file...
> Nope. Question to know whether same situation to Bug 228512(Junk Purge case)
> can occur or not, on "for at most XX days" of "Leave message on server".
> 

Comment 17 David :Bienvenu 2006-09-18 09:36:43 PDT
popstate.dat processing happens when we do get new mail from the pop3 server - prior to running the url, we iterate over popstate.dat and mark messages older than 30 days for deletion.

It's true that there are bugs that cause us to lose popstate.dat, and to generate a new one, which also causes us to re-download all the messages left on the server (this generally happens when we lose the connection while downloading the list of messages from the pop3 server, iirc). I don't think changing the days we leave on server will cause us to lose popstate.dat and redownload messages, but I haven't tried it. If you run into one of these bugs more than every 30 days, I can see how messages would never get deleted from the server, because we'll regenerate popstate.dat, and re-start the download date on the messages. If you have 30,000 messages on the server, I hope you're not running into that bug more than every 30 days! You might try setting a shorter age limit.

There are a couple other possibilities - one is that mindspring has a problem with their pop3 server so that deletes aren't honoured. It seems unlikely, but if your dad is having the same problem with mindspring, that would explain it. Or perhaps you set up your Dad's account and set up the same 30 day expiration time, and he's running into the re-download problem as well?

The other possibility is some confusion happening because you originally set up your server as nntp.mindspring.com instead of pop3.mindspring.com, and renamed the server (according to your prefs.js). But I doubt that's an issue, because popstate.dat is generally working for you (i.e., we don't redownload every message every time you connect to the server).
Comment 18 WADA 2006-09-18 20:40:31 PDT
(In reply to comment #17)
> popstate.dat processing happens when we do get new mail from the pop3 server -
> prior to running the url, we iterate over popstate.dat and mark messages older
> than 30 days for deletion.
Thanks for explanation. I can sleep well :-)

> It's true that there are bugs that cause us to lose popstate.dat
Bug 237131?
geopilot, see Bug 263142 for detailed explanation than mine about popstate.dat. 

> you originally set up your server as nntp.mindspring.com instead of
> pop3.mindspring.com, and renamed the server (according to your prefs.js).

Is this mean followings are set? 
 - mail.server.serverN.hostname     = nntp.mindspring.com
 - mail.server.serverN.realhostname = pop3.mindspring.com
If so, this never causes confusion by Thunderbird, but tends to produce confusion by human beings...

This setup generates directory of nntp.mindspring.com for this account. 
If another account with server name(hostname) of pop3.mindspring.com is defined, directory of pop3.mindspring.com is created. And if user deletes the another account, this directory is kept.
This can cause inconsistency in prefs.js when user edits prefs.js manually.
This causes unsuccessful restore of backup if user restored backup data to unused directory of pop3.mindspring.com, and data overlay if directory of pop3.mindspring.com is used by other account.

geopilot, please be careful when checking files under mail directory.
(I don't think above has relation to your problem)
Comment 19 geopilot 2006-09-19 01:32:48 PDT
In answer to someones question my father runs thunderbird on his computer too thanks to me and he also has his email fill up. He also has a 30 day time on his emails but they never get deleted.

Lastly and unfortunately now for some reason since I did the tests that were asked thunderbird is now no longer displaying my inbox. It opens and takes a long time to build an inbox summary but never displays the messages text in either the preview window or when I click on the message in the new window. No text just subject and other data like sender and date. When I reply to a message it just start the message witho 	who I am replying to bur quptes no text.

I left a copy of popstat.dat in the application data folder from prior tests. It was named "copy of popstat.dat"
I noticed thunderbird had created a new folder called copy of popstat.dat under the account name I guess because the file existed in the folder when I opened thunderbird. I have deleted that copy of popstat.dat file and now the folder doesn't show up but the inbox still won't display the text of my inbox messages.
Anybody got any ideas how I can get them to display again? Did the testing and leaving a file called copy of postat.dat in the folder completely screw up my inbox?

Please don't ask me if I had a backup of my inbox... Two years of email ugh.
Is there a master file I can recover and get thunderbird to reread?

Ugh.
Would the program really confuse a file named copy of postat.dat with the actually popstat.dat?


All the other problems are the same. Still not deleting from server (there's my back up maybe).maybe someone is looking out for me....

Appreciate the help by the way. I'd love to help get this server deletion error debugged and fixed. Thunderbird is a great program.
Comment 20 Bob Clary [:bc:] 2006-09-19 02:06:07 PDT
confirming for tb2/winxp. I set up a pop account for testing, set leave messages on server and set for at most 1 days. I sent a mail to the account on 9/16 and another on 9/18 but the 9/16 mail is still in the account.
Comment 21 Majken Connor [:Kensie] 2006-09-19 02:36:17 PDT
geopilot, you've hit another profile corruption issue, the emails are still there, but the indexes are probably corrupt. If I'm not mistaken, compacting the folder you're seeing this issue on can help.  If not, or if you'd rather sort this out in real time feel free to join #thunderbird on irc.mozilla.org and we can post the relevant results back here, rather than adding to the bug with unrelated support comments.
Comment 22 David :Bienvenu 2006-09-19 18:01:32 PDT
Bob, we start counting from when you download the messages, not from when they're sent. So if you just set up the test profile, I wouldn't expect messages to be aged away for two days.
Comment 23 David :Bienvenu 2006-09-19 18:05:12 PDT
Geopilot, Have you been automatically upgraded to TB 1.5.0.7 yet? I think that should have fixed the rebuilding summary file problem. If not, you can try deleting your Inbox.msf file (Inbox.msf, not Inbox!)

Comment 24 geopilot 2006-09-19 22:14:27 PDT
A couple of ansewers.
My fathers separatte computer runs thunderbird and has the same problem of his email box never being cleared when he sets keeps messages on server for maximum of 30 days.

I will delete the inbox.msf thanks.
Nope haven't upgraded to 1.07 yet.didn't want to in the middle of this but I will I guess. I'l handle any other support issue as you mentioned. Back to the bug.

 the main bug is still not deleting email on the server after access date has passed . That's the primary urgent functionality issue. 


However discussions have pointed out
a newly indentified bug -The loss of popstats.dat which sometimes causes all email still on server to be download for a second time.
Regarding the latter it seems the solution would be before opening popstats.dat make a backup of it replacing any old backup.
Begin email download.
When email download completes without error mark a variable and add all new emails.
Then replace popstats.dat with new one and retain old popstats.dat as backup.
Only replace backup popstats.dat on next email download.

So you never end up with an empty popstats.dat so you never redownload all emails already downloaded.

Comment 25 Bob Clary [:bc:] 2006-09-20 05:52:03 PDT
(In reply to comment #22)
> Bob, we start counting from when you download the messages, not from when
> they're sent. So if you just set up the test profile, I wouldn't expect
> messages to be aged away for two days.
> 

I sent and received the email on 9/16. It is now 9/20 and it hasn't yet been deleted.
Comment 26 David :Bienvenu 2006-09-20 07:48:45 PDT
Bob, can you send me your prefs.js, popstate.dat (from the pop3 account directory in your user profile dir), and a pop3 protocol log? http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#pop3

thx!
Comment 27 Bob Clary [:bc:] 2006-09-20 08:31:26 PDT
(In reply to comment #26)

I might have screwed up with respect to different profiles that pointed to the same pop account. I set up a new profile and deleted the account from the other profiles and have started a new test with mail sent and received today. I check again tomorrow and the next day and keep the logs to send to you.

One thing I have noticed is that the server settings of leave messages on server for at most x days kind of conflicts with the retention policy. If you right click the Inbox and choose properties, retention policy. A new profile has 
"Use Server defaults" unchecked, and has "Don't delete any message" selected.
Comment 28 David :Bienvenu 2006-09-20 08:34:13 PDT
in that context, I think the retention properties are strictly for the local copies of the messages...
Comment 29 David :Bienvenu 2006-09-28 09:30:22 PDT
Any updates, Bob or geopilot? Geopilot, see comment #26 for additional info I could use to try to debug this...
Comment 30 geopilot 2006-09-28 10:59:00 PDT
The bug still exists.
No mail is being deleted from the server.
It exists on every computer I have installed thunderbird on (2 computers in two locationa with separate email accounts for  different people.)

On a separate but related bug note I have also experinced thunderbird losing its popstats.dat file two times again after failing to connect to the mail server.
I noticed it when on reconnection thunderbird was attempting to redownload 20,000 emails it had already downloaded. Now i keep a backup of popstat  and if I see that 20,000 download happening I stop thunderbird and overwrite the old popstat with a backup.
Thunderbird needs to keep a backup of popstat itself for times when it fails to connect so that popstat info isn't wiped out.
Comment 31 Bob Clary [:bc:] 2006-09-28 12:17:49 PDT
(In reply to comment #26)

done. sorry, iamlame
Comment 32 David :Bienvenu 2006-09-28 16:06:53 PDT
Bob, your pop3 protocol log shows us issuing the delete to the server, so that we did delete one message from the server...so you should only have one left (which looks like a message from today, judging by the timestamp)

geopilot, I'm trying to fix the case where we lose the connection - the fix is not to have a backup popstate.dat - the fix should be not to crunch the existing on in the first place...
Comment 33 geopilot 2006-11-10 00:03:28 PST
did you fix the deletion of popstate.dat in cases where you lose the connection?



(In reply to comment #32)
> Bob, your pop3 protocol log shows us issuing the delete to the server, so that
> we did delete one message from the server...so you should only have one left
> (which looks like a message from today, judging by the timestamp)
> 
> geopilot, I'm trying to fix the case where we lose the connection - the fix is
> not to have a backup popstate.dat - the fix should be not to crunch the
> existing on in the first place...
> 
Comment 34 geopilot 2007-01-28 14:15:01 PST
I am disappointed to see this critical error has not been fixed in the latest release.

I just logged on to my server and when the mail file was being downloaded again there was an error and when I relogged on to get my mail thunderbird began to redown load over 25,000 emails stored on my server that it had already downloaded the months before.

Is anyone working on this?
We took a  lot of time and effort to determine that thunderbird was deleting the downloaded email status file before beginnining to download more email and if that new download process somehow went wrong all records of which emails had alrerady been downloaded in the weeks before was lost becuase the email downloaded status info file had been deleted already.
We know the problem.
Can anyone code a fix?


This is a huge critical error.

thunderbird simply can't, after it chokes, go back on line and redown every single email it has already downloaded.

it creates total duplicate inbox chaos for those of us who store email on the server for backup and multiple geographic point access of those emails.

PLEASE PLEASE PLEASE will someone fix this.
and could you write a "remove duplicate emails in same folder" routine at the same time to fix all the ones that have already been duplicated?

PPPLLLEEEAAASSSSEEEE
(In reply to comment #0)
> User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5)
> Gecko/20041217
> Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5)
> Gecko/20041217
> 
> CRITICAL problem with "leave messages on server for x days"
> using the following option
> under menu Tools: Account settings
> check box for
> Account settings
> in "server settings"
> check "leave messages on server" and "for at most x days" substituting say 30
> days for x
> 
> 1. When using thunderbird on mindspring or earthlink (which are standard pop3
> servers) thunderbird never keeps track of how long the emails it has read have
> been on the server and it never deletes them. This means your email quota
> eventually fills and you stop getting email.
> 
> At some point your email box fills and you must manually delete all emails
> (which for most people is technically impossible without tech support so they
> will just switch away from thunderbird) and results in loss of emails not yet
> downloaded between last thunderbird use, deletion trama, and next thunderbird
> use (often requires you to log in to webmail).
> 
> People must be allowed to leave messages on their server for several days so
> they can synchronize all received emails betwen several computers or phones etc
> which may not log on to the owners pop server everyday . (at least that is the
> main reason I use it and for short term backup in case of email loss on home
> computers between backup schedules).
> 
> This has been an ongoing issue through the past several versions and maybe for
> ever. I have seen many posts about it on the internet but never the bug boards
> becuase like me I guess a lot of people don't want to have to register just to
> report a bug.
> 
> This is a major problem in core general email client functionality. It's a deal
> killer for an other wise incredible product.
> It is a critical core funtion of any email program and is pretty much a deal
> killer for email powerusers.
> 
> You only notice this problem after your emailquota fills and your box begins to
> reject emails so many people may not notice it for months. It's the AIDS bug
> for email clients. Infected now - only learn later.
> 
> 2. different problem same cause?
> Additionally sometimes thunderbird redownloads all emails that have already
> previously downloaded.
> This seems to happen if I am running thunderbird on two computers with leve
> email on server set and both access the server.
> 
> possible cause?
> 
> I beleive Thunderbird is not correctly and independently of the server keeping
> track of emails it has already downloaded from a server and is not tracking how
> long they have been left on the server. Each manifestation of thunderbird on
> each computer must keep track independently of the emails it has downloaded.
> 
> Most email programs have the option to have your email client download either
> the last message you have read from the server regardless of whether thyou did
> it on the computer you are using at the moment (which I presume is indicated by
> the server itself somehow)
> or the other option is downloading starting formt he last email you have read
> on the current computer (which the currect computer email client must be
> tracking) The latter choice insures that you will have all emails on each
> computer you operate from which is essential for business when you need access
> to all previous emails someone has written you no matter which computer you are
> on.
> 
> Reproducible: Always
> 
> Steps to Reproduce:
> 1.under menu Tools: Account settings
> check box for
> Account settings
> in "server settings"
> check "leave messages on server" and "for at most x days" substituting say 30
> days for x
> check your email.
> after months your email server will fill or you can see via webmail that your
> emails are never getting deleted from the server after the specificed days you
> set for them to be deleted.
> This eventually happens to EVERYBODY who uses leave messages on server for x
> days. I tmay take months for peopels boxes to fill especially gmail but it is a
> critical certain problem!
> 
> 3.
> 
> Actual Results:  
> email server fills. emails begin to be rejected. people dont know it and they
> lose critical emails!
> messing with the leave messages on server settings in thunderbird may cause
> thunderbird to Retreive duplicates of ALL emails still on server AGAIn leaving
> you with thousands of duplicates as well.
> 
> SO obviusly thunderbird is 
> 1. not keeping track by itslef of which messages it downloaded and how long ago
> and then telling the server to delete the ones past keep time.
> also obviously when you chenge thuderbird leav emessages on erver settings
> thunderbird is redownloading everything on the server not recognizing it
> already downloaded thiose that were left on the server.
> 
> HUGE MASSIVE MESS!
> 
> Expected Results:  
> messages should be deleted froms erver after the speicfied days
> 
> this is huge.
> it should be ina  category of "primary basic functionaltiy" but you don't have
> one. however this means Thunderbird can't do one of the most basic email
> retrieval functions. I can't reccomend it to my company clients if it can't.
> 
> I love thunderbird but had i known it had this hidden problem that only appears
> after a long period I problaby wouldn't have swicthed to it.
> 
> The bug reporting process is so onerous that i have had to wait to report this
> but it is reported all over newsgorups on the internet and thunderbird is
> getting a bad rep from it.
> 
> Please fix!!
> 
> 
> PLEASE
> 

Comment 35 geopilot 2007-01-28 14:17:00 PST
this is the issue as we determine before if you read the thread.

"However discussions have pointed out
a newly indentified bug -The loss of popstats.dat which sometimes causes all
email still on server to be download for a second time.
Regarding the latter it seems the solution would be before opening popstats.dat
make a backup of it replacing any old backup.
Begin email download.
When email download completes without error mark a variable and add all new
emails.
Then replace popstats.dat with new one and retain old popstats.dat as backup.
Only replace backup popstats.dat on next email download.

So you never end up with an empty popstats.dat so you never redownload all
emails already downloaded."

Comment 36 geopilot 2007-01-28 14:24:22 PST
Also the loss of this file popstats.dat also means your retention policy is completey screwed up becuase thunderbird only counts retention days from when it thinks it picked up the email - not as it should from how long ago the email is dated.

so if you want to leave emails on the server for say 30 days and you lose your popstats.dat file on the 28 th day well all thos eemaill from days 1 through 28 are going to sit on the server for  ANOTHER 30 days  ( or forever if the popstats.dat file keeps getting lost as it seems to do)
Comment 37 David :Bienvenu 2007-01-28 15:52:00 PST
Did I miss the part where someone had reliable reproducible steps for reproducing the problem? I don't know of any, and I've tried for many hours to reproduce this problem.
Comment 38 Majken Connor [:Kensie] 2007-01-28 17:49:28 PST
geopilot - how are you exiting thunderbird? are exiting before you shut down the pc, or are you just shutting down the pc and letting windows close all your programs?  

Also, as someone asked before, are you closing thunderbird very soon after opening it? For instance will you run thunderbird, see that it has not reported any new mail and then exit thunderbird right away, or do you leave it running?
Comment 39 geopilot 2007-03-12 16:14:22 PDT
This again jsut happened to me

2900 messages downlowded twice after error on message 2003

I connect nomrlaly via a static internet cable connection.
it will happen to anyone who has their download session disconnected for some error etc.

just turn off removing emailf from server and let a lot build up. then stop your download session in the middle of a large download of emails. then when you go back on you will get all the emails already downloaded  all being downloaded again.
this has been confirmed ove rand over.

it is hard to log on here to rerplay to email repsoonses i get to this. (Why cant i just repsond in email to the email notes. do you relaize how har dit is for normal peopel to find this site and relogon with a password to say it happened again?)

I gues smost users just give up whent he bug never gets fixed. It is  ahuge basioc problem here though. 
it should not redownload emails previously downloaded just becuase the popstat.dat file is disctroyed before all the downloaded message are downlo9add. postat.dat needs to be updated for each email that is successfully donwloaded form the server.


Comment 40 Wayne Mery (:wsmwk, NI for questions) 2007-05-10 07:11:44 PDT
move to networking pop?
Comment 41 David :Bienvenu 2007-05-12 12:16:41 PDT
Created attachment 264617 [details] [diff] [review]
[checked in]possible fix

This change makes it so we handle and propagate stream errors back to the state machine. I really don't know if this is going to fix the problem, but it does fix a problem I was seeing where we were not clearing the running protocol object on the pop3 server, and not clearing the open folder, in the case of stream errors, so the fix obviously makes us go through the cleanup code when we weren't before.
Comment 42 David :Bienvenu 2007-05-12 19:14:35 PDT
We'll see if this fixes any problems for people running on the trunk.
Comment 43 WADA 2007-05-13 08:47:26 PDT
(In reply to comment #17)
> It's true that there are bugs that cause us to lose popstate.dat, and to
> generate a new one, which also causes us to re-download all the messages left
> on the server (this generally happens when we lose the connection while
> downloading the list of messages from the pop3 server, iirc).
Adding description about "loss of popstate.dat when connection loss" to summary, for ease of search and for ease of DUPing other bugs to this bug.
David, please correct it if my change is not appropriate.  
Comment 44 David :Bienvenu 2007-05-16 09:01:25 PDT
Comment on attachment 264617 [details] [diff] [review]
[checked in]possible fix

this is a "correctness" fix - we should be paying attention to stream errors.  This problem is hard to reproduce, so I'd like to get the fix into a 2.0 branch build for others to try.
Comment 45 geopilot 2007-06-02 10:33:30 PDT
Did anything regarding this get into the 1.5.0.12 update?
cheers.


Comment 46 David :Bienvenu 2007-06-03 16:43:47 PDT
No - and I'm still trying to figure out some fixes for this. I have found a reproducible way to get some of the popstate.dat cleared out, using the debugger and tweaking return codes, but I'm still working on a fix.
Comment 47 David :Bienvenu 2007-06-04 15:49:53 PDT
Created attachment 267209 [details] [diff] [review]
[checked in]proposed fix (with some other cleanup)

This fixes the case I could reproduce, of a particular fetch failing.  I'll attach a patch which shows the really relevant parts of the fix, for posterity, and the 2.0.0.x branch sake. I had to clean up the formatting of the code so that I could better understand what the heck was going on.
Comment 48 David :Bienvenu 2007-06-04 15:57:52 PDT
Created attachment 267211 [details] [diff] [review]
minimal diff

this is what the actual fix is - in particular, only committing the newuidl to popstate.dat if it is the right size, and making sure that one of the changes gets added to the original uidl hash table, in case we end up committing that one.

I've cc'ed Christian because I know he's looked at this code and might have suggestions.
Comment 49 David :Bienvenu 2007-06-05 09:21:51 PDT
Comment on attachment 267211 [details] [diff] [review]
minimal diff

if this bakes well on the trunk, I'd like to get it into a 2.0.0.x release.
Comment 50 David :Bienvenu 2007-06-06 14:35:22 PDT
really cc'ing Christian this time. I'm going to mark this fixed. If anyone sees this with new trunk builds, please let me know, and re-open with details.
Comment 51 David :Bienvenu 2007-06-25 12:38:56 PDT
Comment on attachment 267211 [details] [diff] [review]
minimal diff

this patch should be combined with the last patch in bug 240049 if either of them are allowed into 2.0.0.5
Comment 52 Scott MacGregor 2007-06-29 15:54:32 PDT
Comment on attachment 267211 [details] [diff] [review]
minimal diff

a=mscott for mailnews only change to 1.8.1.5.
Comment 53 David :Bienvenu 2007-07-01 20:29:12 PDT
fixed for 1.8.1.5
Comment 54 Daniel Veditz [:dveditz] 2007-07-02 11:00:32 PDT
Comment on attachment 264617 [details] [diff] [review]
[checked in]possible fix

assuming this is an leftover request given the fixed keyword
Comment 55 Al Billings [:abillings] 2007-07-17 10:38:41 PDT
Now that it is fixed, can someone give reliable repro steps for testing the fix? After reading all of the comments, I'm not sure of the scope of the fix or the steps to verify it in the branch.
Comment 56 geopilot 2007-07-26 23:02:40 PDT
goog news! I just had sucha crash in the middle of downloading 8818 messages.
on number 8803 the dreaded popup window came up

"The RETR command did not succeed erro retreiving a message. Mail server pop.mindspring.com responded: unable to open that message.


SO HERE"S THE GOOD NEWS.

After i clicked ok to close the pop up error i shut down and reconnected to get the rest adn it only downloaded the last few messages it hadn't finished!

YEAH!
Thanks for the great work guys.
seems the popstat.dat error is history.


now unfortunately i have found a reapppearance of the 2 gig inbox file limt so maybe we should look at the popstat.dat fix code and make sure no old short variable gots reintroduced to the message handling code.

also is panacea.dat a temporary postat.dat file?

if i want to downmload all message from my server becuase some were skipped ealry on with the  new 2 gig erro problem do i have to erase the postat.dat and pancea.dat to get tbird to download EVERYTHING on the server just this once?.

Note You need to log in before you can comment on or make changes to this bug.