151056 - MDN: imap only: No return receipt prompt with a mail mesg that is approx bigger than 50kb

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Backend, defect)

Description

[grylchan]

If a user requests a return receipt on a mesg that is bigger than
50 kb(either having an attachment or just huge amount of text in body
of the mesg) the addressee won't be prompted for Read Receipt request.

This occurs in both branch/trunk/pr1.

Tested this on commercial branch 2002-06-11-08-1.0.0 NT 4.0,
20020607 branch/trunk builds on linux 2.2
& PR1 on Mac os 10.1.4

When I tested this for pr1, I used attachments that weren't that big.
So I didn't run into this before.

I'm not sure what the size cutoff the mesg has to be in order
for Return Receipt prompt to not work. I thought it was anything
bigger than 30kb. But to be safe just make sure mesg is approx
bigger than 100kb. Small mail mesgs say anything under 20kb works fine.
I get inconsistent results as to what the true 'cut off' size for a
mail mesg is.


Again it's the size of the mesg that's the problem not attachments
themselves.

This works fine in pop acnts so I don't know why imap fails.

I DO NOT have the pref 'do not download mesgs greater than 50kb' set.

I tried 2 different imap mail servers and got same results.

steps to reproduce:
1.create a imap mail acnt
2.Login
3.Set up either Global/Account based Return receipt prefs
  -when sending mesgs, always request a return receipt
  -when I receive a request for a return receipt
    -allow return receipts for some mesgs
     -ask me on all 3 cases
4.Compose a mesg to yourself
5.Either attach a attachment bigger than say 100kb or
  write/paste huge amount of text into the body of the mesg
6.Send mesg

result: you get the mesg, but no return receipt request prompt
expected: to get return receipt request prompt

Comment 1

[grylchan]

adding nsbeta1 but realize probably to late

Comment 2

[David :Bienvenu]

see also bug 140477 - my guess is that when mime parts on demand gets invoked,
again, we don't go through the code that mdn expects us to go through. My guess
is that the fix I describe in bug 14077 might fix this as well.

Comment 3

[David :Bienvenu]

the problem here is that when we get to
nsImapMailFolder::NormalEndMsgWriteStream, the hdr has already been marked read,
so we don't execute the mdn code.

Comment 4

[David :Bienvenu]

the problem is that we're marking the msg read in the db because we're getting a
response from the server that the message has been seen when we fetch the body
headers - in a race condition (I guess), this marks the msg read in the db
before we try to check for an mdn request. This happens in
nsImapServerResponseParser::PostProcessEndOfLine(). I'll have to look at a log
to check this out.

Comment 5

[David :Bienvenu]

fetching the body structure sets the seen flag:

1684[4d26458]: nsmail-1:S-Personal:SendData: 15 UID fetch 15374 (BODYSTRUCTURE)

1684[4d26458]: nsmail-1:S-Personal:CreateNewLineFromSocket: * 773 FETCH (UID
15374 BODYSTRUCTURE (("TEXT" "HTML" ("CHARSET" "us-ascii") NIL NIL "7BIT" 1059
40 NIL NIL NIL)("APPLICATION" "PDF" ("NAME" "ash.pdf") NIL NIL "BASE64" 37172
NIL ("INLINE" ("FILENAME" "ash.pdf")) NIL) "MIXED" ("BOUNDARY"
"------------020004030008050904070300") NIL NIL))

1684[4d26458]: nsmail-1:S-Personal:CreateNewLineFromSocket: 15 OK Completed

1684[4d26458]: nsmail-1:S-Personal:SendData: 16 UID fetch 15374 (BODY[HEADER]
BODY[1.MIME] BODY[2.MIME])

1684[4d26458]: nsmail-1:S-Personal:CreateNewLineFromSocket: * 773 FETCH (FLAGS
(\Recent \Seen) UID 15374 BODY[HEADER] {880}

so ignore my comments about fixing the other problem fixing this problem.

Comment 6

[scottputterman]

We should try to fix this. David, did you want to keep looking into this or
should we reassign it to Navin?

Scott

Comment 7

[David :Bienvenu]

I can keep looking at it. I just need to figure out a way to handle this.

Comment 8

[grylchan]

It affects both 'ask me'and 'always send'

I think mesg size for RR not working is 30kb but sometimes it varies.

No problems in 4.79 as RR works for big mesgs. But I realize
code in mozilla was completely rewritten for RR.

Comment 9

[David :Bienvenu]

I can see one way of fixing this which isn't very attractive, but I think it
will  work.

1. Add a boolean attribute to nsIImapUrl, msgUnreadBeforeFetching, intialize it
to PR_FALSE (so we don't get any false positives). In the
nsImapService::DisplayMessage code, get the msg hdr for the message we're trying
to display, and check if it's read or not. If it's not read, set
msgUnreadBeforeFetching to PR_TRUE. Then, in
nsImapMailFolder::NormalEndMsgWriteStream, instead of checking 
msgHdr->GetIsRead(&isRead), we would check the
nsIImapUrl::msgUnreadBeforeFetching, and if it was unread, then we go into the
mdn code. This will fix the problem where the msg gets marked read during the
process of fetching the body structure for the message.

there are a couple other possibilities:

2. Override nsIMsgMailNewsUrl::SetMimeHeaders in nsImapUrl to check if the msg
being read is unread and if has an mdn request header (basically, move some of
the logic in nsImapMailFolder::NormalEndMsgWriteStream to SetMimeHeaders) and if
the msg is unread and there's an mdn request, set a new attribute on the imap
url - msgHasMDNREquest, and check that attribute in
nsImapMailFolder::NormalEndMsgWriteStream. This would only work if we
SetMimeHeaders before fetching the body structure, and we probably don't do that
- I'd need to check. If it worked, it would be because we basically moved the
logic for checking if the msg was read to a point before the server tells us the
msg is read.

3. Since the problem is that the server is telling us the msg is read in an
unsolicited flags response from the body structure, we could tweak the imap
protocol code to ignore this info in the case where we're fetching a msg for
display (we know that when we're fetching a message, the server is going to mark
it read, and we'll mark it read in the db anyway, so it's safe to ignore this
information in this one case). So, that fix would be in
nsImapProtocol::NotifyMessageFlags - something like checking the imapAction for
the current url - if it's a message fetch, and we're just being told that the
msg is read, then we can ignore the notification and not pass it on to the
message sink. We'd also need to ignore the /RECENT flag since the
nsImapMailFolder code ignores it. So, we'd do something like: if currentAction
!= nsImapMsgFetch || (flags & ~kImapMsgRecentFlag) != kImapMsgSeenFlag) then
call notifymessageflags on the message sink. Or, better, we could check what we
think the flags were, and what the flag change is, and if it's just the read
flag changing, ignore it. So, we'd use the protocol flag state and call
nsImapFlagAndUidState::GetMessageFlagsFromUID() for the uid, and compare that
with the notify flags. If the change is just the SEEN flag, and we're reading a
message, ignore it. This is the most direct way of fixing this, and doesn't
involve adding any new attributes to interfaces. It's unattractive in the sense
that we're ignoring something the server is telling us, but it's something we
already knew...

Do either Cavin or Navin want a shot at this?

Comment 10

[Navin Gupta]

I can take a look at this. 

Comment 11

[Cavin Song]

Thanks Navin. I just finished reading David's comment #9 and thought that
suggestion #1 is very straightforward (without looking at the code).

Comment 12

[Navin Gupta]

I am seeing other problem besides msg read. I sent a msg size 80 kb but it
doesn't have Dispoistion-Notification-To header. If I send a msg with no subject
or body then I can see this hdr View | headers | All.

looks like something is going wrong when we send msg of large size. taking so that
it is on my radar

Comment 13

[grylchan]

Hey Navin,
using commercial branch:
Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0) Gecko/20020619

I tried sending myself mesgs with mesg sizes of 95kb & 131kb
and though I didn't recieve a return receipt prompt I did
see in the headers (using View|headers|all) this:
  Under mesg id & above User agent string I see:
     Disposition-Notification-To: [e-mail]

I assume this is what you thought was missing? I did have subject
and body text filled out when i sent the mesg. Just fyi...

Comment 14

[Navin Gupta]

I was doing Edit Msg as new, that doesn't insert that hdr. 

Comment 15

[grylchan]

hmm I tried edit a mesg as new (w/big attach) and I did
see the disposition notification header. 

Just fyi, there is another bug, bug 144296, that lists
other ways we can 'compose' a message and not have return receipt
set on that mesg even though we have the preference set. I guess
we could add 'edit as new' to that bug..

Comment 16

[Navin Gupta]

Actually I ran into some other bug with mdn that has to do with fwd msgs. 
It seems MIME is overwrites first set of headers with next set of headers. It is
not able to clearly determine when the headers are over. So the last set of
headers are used. 

I am able to reproduce your case if I do a fresh compose and cut & paste big 
attachment. still investigating...

Comment 17

[David :Bienvenu]

to recreate my case, all you have to do is attach an exe or .dll larger than 50K
or  so to a message and turn on request return receipt - no cutting or pasting
is needed.

Comment 18

[Navin Gupta]

why don't we use the RECENT flag? RECENT flag is the correct way to fix this 
bug.

Comment 19

[David :Bienvenu]

sorry, no, that's wrong. Imagine this scenario - you open your inbox, get a
bunch of new messages, but don't read them, and shut down. Then you open your
inbox again, read the unread (but not /RECENT) messages. If we used the /RECENT
flag, we wouldn't try to send an MDN receipt for the message.

Comment 20

[David :Bienvenu]

if you're looking for the minimal fix, Navin, option 3 is about two lines of code.

Comment 21

[Navin Gupta]

Yes, you are right. I'm testing option 3 right now. 

Comment 22

[Navin Gupta]

Attachment: fix - idea no. 3

This is to ignore the server telling us about flags --> that msg is read while 

we are downloading the body.

Comment 23

[Navin Gupta]

Attachment: fix - idea no. 3

right patch.

Comment 24

[Navin Gupta]

Can I get reviews ? thx. 

Comment 25

[Navin Gupta]

Actually we can completely ignore flag update response from server when
we are fetching msg. I can't think of any major flag change while doing msg 
fetch other than this read (\seen) flag getting changed. 

Comment 26

[Navin Gupta]

So this 
+      if (m_imapAction != nsIImapUrl::nsImapMsgFetch && (flags & 
~kImapMsgRecentFlag) != kImapMsgSeenFlag)

can be just

if (m_imapAction != nsIImapUrl::nsImapMsgFetch )

 

Comment 27

[David :Bienvenu]

no, more like this:

if m_imapAction != nsIImapUrl::nsImapMsgFetch || (flags & ~kImapMsgRecentFlag)
!= kImapMsgSeenFlag)

your patch makes it so we ignore all flag notifications if we're fetching a
message - I'd rather not do that.

Comment 28

[Navin Gupta]

Attachment: fix

ok, let us notify other flag changes for msgfetch.

Comment 29

[David :Bienvenu]

Comment on attachment 88548 [details] [diff] [review]
fix

sr=bienvenu

Comment 30

[Cavin Song]

Comment on attachment 88548 [details] [diff] [review]
fix

r=cavin.

Comment 31

[Navin Gupta]

fix checked in. 

Comment 32

[Jaime Rodriguez, Jr.]

adt1.0.1+ (on ADT's behalf) approval for checkin to the 1.0 branch, pending
drivers' approval and QA verification on the trunk. pls check this in asap, then
add the "fixed1.0.1" keyword.

Comment 33

[grylchan]

using commercial trunk:
2002-06-21-09-trunk win 2k
2002-06-21-10-trunk/ linux 2.2
2002-06-21-09-trunk/ mac 9.2.2, mac 10.1.4

Tried sending mesgs w/return receipts:
  -to imap,pop, or webmail acnts
  -normal mesgs w/no attachs
  -mesgs that varied in size (via attach or lots of text in the body)
   sizes tested were approx: 10kb, 30kb, 75kb, 100kb,674kb

In all cases, I was prompted 'to send a Return receipt' or
if I had 'auto send of RR' pref set, when I opened the msg, it automatically
sent a RR back to the sender.

Tried both global & account based Return Receipt prefs and both worked
as expected.

marking verified on trunk.

Comment 34

[Judson Valeski]

please checkin to the 1.0.1 branch. once there, remove the "mozilla1.0.1+"
keyword and add the "fixed1.0.1" keyword.

Comment 35

[Navin Gupta]

fixed on branch

Comment 36

[grylchan]

Commercial branch:
2002-06-25-08-1.0/ nt 4.0
2002-06-25-06-1.0/l linux 2.2
2002-06-25-13-1.0/ Mac 10.1.4
2002-06-25-05-1.0/ mac 9.2.2

Using various sized mail messages:
15kb, 30kb, 75kb, 167kb,1800kb,674kb, 108kb

I was able to get a Return Receipt Prompt or
Automatically send one everytime I read a different
sized mesg. 

Tried both imap and pop accounts.
Tested both Global & Account Based Return Receipt prefs.

Verified on Branch. 

Changing keyword fixed1.0.1n to verified1.0.1