[POP3] Implement FETCHLIMIT and EXPUNGE preferences (in the manner of FETCHMAIL)



10 years ago
10 years ago


(Reporter: David A. Cobb, Unassigned)


Firefox Tracking Flags

(Not tracked)




10 years ago
User-Agent:       Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Dealio Toolbar 3.1.1)
Build Identifier: 2008-04-14-03-trunk

	Please implement the following two preferences, modeled on the behavior of the *nix 'fetchmail' 

program:  Fetchlimit is a number of messages after which, no matter what may be left on the server, the 

program sends QUIT, tears down the connection, and waits for the "query server every N minutes" clock.  

Expunge is a number of messages after which the program sends QUIT -- allowing the server to actually 

delete the messages -- them more-or-less immediately cycles back to re-authorize.  That would also be a 

good time to make sure everything is 'synced' and all the information that's useful has actually been 

written to disk.

	Because of workstation down-time, it is quite possible for a POP server to accumulate far more 

messages than would be expected in a "normal" batch.  A number of factors seem to render this situation 

quite pathological.  While I have not yet been able to detect whether the resultant failures are caused 

by the server (ISP) deliberately "throttling" the amount of information transmitted in a session or in a 

unit time, or whether the client (T-Bird) somehow becomes overwhelmed by the volume, the scenario winds 

up in the same horrible place:

C: AUTH, etc to initialize the conversation
S: +OK
S: +OK
S: 1 1000
S: 2 2000 ... continuing thru
S: 15000 2049
S: .
S: [I won't show the UID list here as it is not at issue]
S: .
C: DELE 1 ## T-Bird immediately deletes messages which the UID shows were retrieved previously
S: +OK
S: +OK 2000 octets
S: header-line
S: header-line
S: body-line
S: body-line
S: .
[and so forth, until the fatal limit manifests and
C: ****** Deems the connection to have timed-out

Provided the original retrieval was initiated by the timer, rather than by the user (another bug), after 

a waiting period specified by preference, the process starts again.

Problem! if the backlog gets "high enough" -- and I'm now working through 16000 mails -- the LIST and 

UIDL transmissions virtually exhaust the limitations on the connection/session, or maybe we can retrieve 

two or three mails.  That's slower than the arrival of more stuff at the server, and the problem grows 

and grows.

Problem! Even a great program like T-Bird can fail.  Maybe the power goes out, maybe my wife pushes the 

wrong switch, maybe an illegal BASE64 message arrives and locks up the computer (still another bug).  If 

this happens, QUIT is never (never, never) sent to the server.  All those messages we've been sending 

DELE'tes against are not removed and we will have to deal with them again (and again, and again).  In the 

worst case, some or all of the previously-retrieved UID's do not get written to disk and the message has 

to be retrieved again.

The fetchmail program offers a mechanism which has successfully enabled be to get past this horrible 

situation.  "Fetchlimit" specifies a number of messages, after which the client sends QUIT, tears down 

the connection, and takes its timeout -- even though many messages may remain on the server.  And 

"Expunge" specifies a number of messages, after which the client sends QUIT, takes a quick breath, and 

immediately cycles back to re-authorize.  Both these permit the server to actually drop the messages 

which have been successfully downloaded.  Of course, fetchmail doesn't know whether the message has been 

successfully dealt with locally -- just that the local sendmail has accepted responsibility for it -- but 

that isn't an issue here, in fact any decent sendmail will either deposit or save anything and not lose 

the message.

After my most-recent week of horror, the only way in which I've been able to drain the swamp (my ISP's 

pop server) is by setting the "expunge" all the way down to 200!  If I try to retrieve much more, the 

timeouts start biting me.  Fetchmail doesn't really cope well with the timeouts, it winds up getting 

duplicate messages -- not your problem -- but with the lower setting I have actually plunked the whole 

terrible volume into T-Bird's movemail Inbox.

Reproducible: Always

Steps to Reproduce:

Comment 1

10 years ago
Oh, RAT! please pardon the wierd line breaks.  I composed this in a textfile so I could put it up quickly ('cause I'm at work).

Comment 2

10 years ago
Looks good, and I'm not seeing a duplicate, so confirming.
Ever confirmed: true
OS: Linux → All
Hardware: PC → All
You need to log in before you can comment on or make changes to this bug.