482879 - crash [@ MimeMultipart_parse_line ]

Status: VERIFIED FIXED

(MailNews Core :: MIME, defect)

Description

[Wayne Mery (:wsmwk)]

topcrash for 3.0b2 [@ MimeMultipart_parse_line ]

earliest crash found is build 20090217025755
 bp-b0cbe94d-cefe-4981-8565-30d0a2090305
 "I was trying to look at an old meeting invite email that originated from a Lotus Notes client

bp-ef99d6b3-7f15-44db-86b9-f053c2090307
0	thunderbird.exe	MimeMultipart_parse_line	mimemult.cpp:201
1	thunderbird.exe	convert_and_send_buffer	mimebuf.cpp:184
2	thunderbird.exe	mime_LineBuffer	mimebuf.cpp:272
3	thunderbird.exe	MimeObject_parse_buffer	mimeobj.cpp:275
4	thunderbird.exe	MimeMultipartRelated_parse_eof	mimemrel.cpp:1075
5	thunderbird.exe	MimeMultipart_close_child	mimemult.cpp:616
6	thunderbird.exe	MimeMultipart_parse_line	mimemult.cpp:185
7	thunderbird.exe	convert_and_send_buffer	mimebuf.cpp:184
8	thunderbird.exe	mime_LineBuffer	mimebuf.cpp:272
9	thunderbird.exe	MimeObject_parse_buffer	mimeobj.cpp:275
10	thunderbird.exe	MimeMessage_parse_line	mimemsg.cpp:232
11	thunderbird.exe	convert_and_send_buffer	mimebuf.cpp:184
12	thunderbird.exe	mime_LineBuffer	mimebuf.cpp:272
13	thunderbird.exe	MimeObject_parse_buffer	mimeobj.cpp:275
14	thunderbird.exe	mime_display_stream_write	mimemoz2.cpp:946
15	thunderbird.exe	nsStreamConverter::OnDataAvailable	nsStreamConverter.cpp:938
16	thunderbird.exe	nsMailboxProtocol::ReadMessageResponse	nsMailboxProtocol.cpp:593
17	thunderbird.exe	nsMailboxProtocol::ProcessProtocolState	nsMailboxProtocol.cpp:690
18	thunderbird.exe	nsMsgProtocol::OnDataAvailable	nsMsgProtocol.cpp:347
19	thunderbird.exe	nsInputStreamPump::OnStateTransfer	netwerk/base/src/nsInputStreamPump.cpp:508
20	thunderbird.exe	nsInputStreamPump::OnInputStreamReady	netwerk/base/src/nsInputStreamPump.cpp:398
21	xpcom_core.dll	nsInputStreamReadyEvent::Run	xpcom/io/nsStreamUtils.cpp:111
22	xpcom_core.dll	nsThread::ProcessNextEvent	xpcom/threads/nsThread.cpp:510
23	xpcom_core.dll	NS_ProcessNextEvent_P	nsThreadUtils.cpp:227
24	thunderbird.exe	nsBaseAppShell::Run	widget/src/xpwidgets/nsBaseAppShell.cpp:170
25	thunderbird.exe	nsAppStartup::Run	toolkit/components/startup/src/nsAppStartup.cpp:192

Comment 1

[Magnus Melin [:mkmelin]]

Crashes at 
  if (obj->options->state->partsToStrip.Length() > 0)

http://hg.mozilla.org/comm-central/file/6fcaf32e4036/mailnews/mime/src/mimemult.cpp#l201

partsToStrip null?

Comment 2

[Ludovic Hirlimann [:Usul]]

Setting blocker ? to get attention.

Comment 3

[David :Bienvenu]

partsToStrip is an object, not a pointer - it can't be null. Since the user isn't detaching/deleting attachments, it should have a length of 0.

Comment 4

[David :Bienvenu]

much more likely that state is null - I'll put up a patch.

Comment 5

[David :Bienvenu]

Attachment: proposed fix - protect against null options and options->state.

by inspection, this is my best guess as to what's crashing...

Comment 6

[David :Bienvenu]

marking blocking+, since it's a topcrasher.

Comment 7

[Andrew Sutherland [:asuth] (he/him)]

The best explanation I can come up with for what is happening here is that MimeMultipartRelated_parse_eof is getting called twice on a single instance.

My patch at http://hg.mozilla.org/comm-central/rev/f0bdc29e9c1c changed the behavior so that if a second pass through happens, we will not fix-up body->parent or body->options to be equal to obj and obj->options, respectively.
(The logic replaces relobj->headobj with the newly created node; on the second
pass through, there relobj->headobj is no longer there to be replaced!)

This would likely reslt in both the crash observed in this crasher, as well as this one in MimeInlineTextHTML_parse_eof that we have 5 of in 3.0b3pre:
http://crash-stats.mozilla.com/report/index/03c18437-350c-4d31-94ab-d644d2090510
and this slightly different crash in the same place:
http://crash-stats.mozilla.com/report/index/615903d3-9330-4b10-ae4e-129b02090510

In all cases it looks like the situation is caused by the multipart/related being embedded inside a multipart itself.  Presumably mimemult.cpp has some scenario in which it does the call multiple times.  I was unable to quickly cause gloda's test_mime_emitter.js to cause the problem but the stack traces
aren't giving me much to go on and I haven't written a fuzzer.

Hoping for bienvenu thoughts?

Comment 8

[David :Bienvenu]

oh, the fact that it was a meeting invite made me suspect Lightning was involved, and I thought some of Lightning's mime use might end up with null state - that was just a vague recollection. 

Your theory is certainly possible, though nested mime multipart messages should be pretty common, so we'd need to figure out what's special about the crash messages. What we need is a message that causes the crash...

Comment 9

[Mark Banner (:standard8)]

(In reply to comment #8)
> Your theory is certainly possible, though nested mime multipart messages should
> be pretty common, so we'd need to figure out what's special about the crash
> messages. What we need is a message that causes the crash...

So do we want to try the patch or try and get a test case?

Comment 10

[David :Bienvenu]

I'd like to see if the patch makes the crashes go away from crash-stats - is there any more info in crash-stats about possible causes, especially if Lightning is involved? Generic libmime crashes are bad w/ gloda because gloda runs all messages through libmime, and I'm not sure that it backs off in the case of crashes indexing a particular message.

Comment 11

[Wayne Mery (:wsmwk)]

from last 3 months, 3 of 6 crashes with comments
 bp-da649106-7520-4be5-8c6a-3dc8c2090518 "...Every other invitation sent from Lotus Notes causes it."
 bp-3fc5525b-2a1a-4a47-93b8-eb6592090410 "Latest versions (from trunk) of TB3 and Lightning. Crashed when a particular invitation from Lotus Notes is displayed."
 bp-ef99d6b3-7f15-44db-86b9-f053c2090307 "Left computer. I went to sleep. Computer went to sleep. Woke up. Came back. Woke it up. Saw this."


No longer topcrash for 3.0b2 - not even top 100. 27 crashes in 28 days
However, #31 for 3.0b3pre.
Not sure what can be deduced from that.

Comment 12

[Ludovic Hirlimann [:Usul]]

(In reply to comment #11)
> from last 3 months, 3 of 6 crashes with comments
>  bp-da649106-7520-4be5-8c6a-3dc8c2090518 "...Every other invitation sent from
> Lotus Notes causes it."

Might be worth getting a note invite and attacjing it to this bug - do we know anybody at IBM we could ask this from ?

Comment 13

[David :Bienvenu]

maybe post on the newsgroup and forums for someone with Lotus notes installed?

Comment 14

[Ludovic Hirlimann [:Usul]]

Will do.

Comment 15

[Ludovic Hirlimann [:Usul]]

Attachment: Lotus notes invitation

This is what I received after I posted to the newsgroups. It did not crash my Thunderbird, but I don't use lightning.

Comment 16

[Mike Kaply [:mkaply]]

I forwarded the info to my IBM contacts. Hopefully someone might take a look. I'm not at IBM anymore, so not much I can do.

Comment 17

[Günter]

@15 attachment id=381573

The attached ICS data is not compliant to the RFC:
Sorry, your calendar could not be parsed.
Error: 	Error was: Error at line 32: Illegal character in opaque part at index 4: CID:
Cause: 	Caused by: Illegal character in opaque part at index 4: CID:<FFFF__=[e-mail]>

Test-output with: http://severinghaus.org/projects/icv/

Same error has been seen before with previous LN 7.0 and we have had problems with ReminderFox also.

Beside the non-compliance to RFC it seems to be unclear why TB (using LG) crashes.
Any os/version/release available here??

CC to 'Fallen' also.

Comment 18

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 376713 [details] [diff] [review]
proposed fix - protect against null options and options->state.

agreed that our lack of forward progress on actually identifying the underlying problem justifies just landing the patch.

However, I would request that at the very least we spin off a new bug if we mark this one fixed without actually addressing the underlying problem I believe exists.

Comment 19

[David :Bienvenu]

I tried that ics file with lightning and it did not crash (w/o this patch). So we are still w/o a test case.

Comment 20

[Günter]

(In reply to comment #19)
David I think testing that ICS data file with LG will just give an error -- as with the Validator (see #17). AFASIK both (V & LG) are based on the same sw-sources.

So a test against this bug needs to receive a LN invitation directly into TB/LG. This is because of the MimeMultipart of such messages.

From our ReminderFox iCal support I have an old msg which was send using LN7 and it seems to have the same error as the Validator shows. Not sure about the mimetype, but it's an S/MIME signed message with Content-Type: multipart/mixed

What is the test environment required?
Could give it a test, using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.21) Gecko/20090302 Thunderbird/2.0.0.21 --- but have to install LG 0.9.

Comment 21

[David :Bienvenu]

Lightning displays it just fine, as near as I could tell. I attached the ics file and sent a message to myself - I could try sending it as a multipart alternative part, actually, which may be what Lotus Notes does (as opposed to multipart related).

Comment 22

[David :Bienvenu]

multipart alternative didn't crash either. So, Ludovic, I think it would be good to see the whole message you got from Notes, instead of just the ics attachment.

Comment 23

[Ludovic Hirlimann [:Usul]]

Attachment: Complete message

> multipart alternative didn't crash either. So, Ludovic, I think it would be
> good to see the whole message you got from Notes, instead of just the ics
> attachment.

Here you go.

Comment 24

[Günter]

Test w att id=381904:

- the .eml format seems not to help here much, can open the eml as an external msg, read the text and ICS attachment/data, but seems not to op like a normal msg received via pop/imap

- copied the source of the .eml to a local folder and needed to add a header line like:
From - Tue Apr 08 13:45:35 2008

to get it working as a 'normal' msg.
But is that the same behavior?? Maybe David can answer.

- no problem opening that modifier msg from a TB folder also with the ICS attachment. LG sees the invitation and processes it normally, as 
ReminderFox does [But RmFx handles the ICS as an 'foreign' invitation -- not allowing to Accept/Decline etc, which lacks with LG0.9 -- separate bug!].

Resumee:  not a real test case; tester(s) should receive an invitation DIRECTLY from a LN account

Tested with: 
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.21) Gecko/20090302 Lightning/0.9 Thunderbird/2.0.0.21; ReminderFox 1.9.1

Comment 25

[Ludovic Hirlimann [:Usul]]

It did not crash for me when I received it, but I don't have lightning installed.

Comment 26

[Andrew Sutherland [:asuth] (he/him)]

Let's just write a fuzzer.  And by we, I mean ideally a volunteer with some spare cycles.

Message generation library:

http://mxr.mozilla.org/comm-central/source/mailnews/test/resources/messageGenerator.js


Test code that uses it, demonstrating usage:

http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/test/unit/test_mime_emitter.js


The fuzzer doesn't actually need to verify the output of course.  In fact, since part of the goal would be to gibberish-y structures, we arguably can't do that unless the fuzzer has a "legal" mode.

For simplicity of initial development, I would propose having the fuzzer masquerade as a test that simply does not define any tests.  If you do a "make SOLO_FILE=test_mime_fuzzer.js check-interactive" (in the objdir that corresponds to where the test lives, sourcew-wise) the make system takes care of the paths for you.  Then at the prompt you could type "run_fuzzer(100)" or something instead of _execute_test() which would simply be a no-op.

Comment 27

[David :Bienvenu]

Günter, are you saying you can reproduce this problem with a direct message, or that you can't reproduce it w/ a .eml file/local file message? I think Lightning streams the message when you click on it, so I don't think it distinguishes between a newly arrived message and an existing message, but if there's any doubt, copying an unread message like this to an imap inbox should fool anybody.

I think a fuzzer is a fine idea, but we might have to know what's special about Lightning that's causing the crash in order to make the fuzzer exercise that permutation (e.g., the way Lightning streams a content part). Assuming that Lightning is involved at all.

Comment 28

[Günter]

(In reply to comment #27)
> Günter, are you saying you can reproduce this problem with a direct message, or
> that you can't reproduce it w/ a .eml file/local file message? 

Sorry if it's not clear. I never had a crash! Never had any problems with it! Not with the pure ICS file, not with the hacky imported .eml file/msg.
Additionally I moved the "imported .eml file" from my "Local Folders" to an IMAP Inbox. Any action on it (display with folder pane, open the msg, add the invitation to any LG calendar [sqlite-, ics-type]) will NOT show any problems at all.

So as asked before: what was the initial scenario the 'crasher' reported?
OS, TB-, LG-version (my test env see #24 comment).

Comment 29

[Günter]

(In reply to comment #28)
> So as asked before: what was the initial scenario the 'crasher' reported?
> OS, TB-, LG-version (my test env see #24 comment).

Upps, just noticed that info is included with http://crash-stats.mozilla.com/report/index/b0cbe94d-cefe-4981-8565-30d0a2090305 'Details':
Mac OS X /10.5.6 9G55

So I just tested it on our MAC 10.5.6 (Mozilla/5.0 (Macintosh; U; Intel Mac OS X; de; rv:1.8.1.21) Gecko/20090302 Thunderbird/2.0.0.21) -- but without Lightning!
==>>  also NO problem, no crash!

Comment 30

[Peter Lairo]

Attachment: Test Invite from Lotus Notes 6.5 (ICS file)

Test Invite from Lotus Notes 6.5 (ICS file) receive as attachment via webmail.

No special settings were made to the meeting invite (no attachment, no "location", no "category").

Let me know if you want another ICS invite with other settings.

Comment 31

[Peter Lairo]

Attachment: Test Invite from Lotus Notes 6.5 (ICS file) (Binary)

As a biany file this time.

Comment 32

[Andrew Sutherland [:asuth] (he/him)]

Thank you for the ICS file, but we need the entire message that contains the ICS file if we are to have a hope of reproducing the crash.  Also, (in general, to anyone providing attachments on this bug) please note whether you have actually produced a crash with the attachment.  If you haven't, unless inspection of the MIME structure or ICS file shows obvious abnormalities from which we can extrapolate the likely problem, the attachment is unlikely to be of any use.

Comment 33

[Günter]

comment # 30 ..32

Peter send an invitation to me (to my 'testaccount' based on WIXxp, TB2.0.0.21/LG0.9). I had NO problem to receive it, opened, added the invitiation to the calendar, 'accepted' it with sending the reply msg. NO problem at all.

Later today I will be able to test it on MAC and will report back.

Will post the invitation msg as attachment to this bug.

Günter

Comment 34

[Günter]

Attachment: LN invitation msg

Invitation msg based on:
 Lotus Notes Release 6.5.5 November 30, 2005

Some names, id etc changed to respect privacy ;-)

Comment 35

[Mark Banner (:standard8)]

Comment on attachment 376713 [details] [diff] [review]
proposed fix - protect against null options and options->state.

Okay, given lack of reproducible test case and its occasional appearance in crash-stats, I think its worth taking this and seeing what effects it has.

sr=Standard8 subject to filing a bug as asuth suggests in comment 18.

Comment 36

[David :Bienvenu]

Fix checked in. Bug 496886 filed for underlying issue.

Comment 37

[David :Bienvenu]

re-opening...

Comment 38

[David :Bienvenu]

Attachment: more bullet-proofing

I hate playing whack-a-mole with this crash, but it's the number one crash now, and we're not getting anywhere finding a message that exhibits this problem.

Comment 39

[Mark Banner (:standard8)]

Comment on attachment 391734 [details] [diff] [review]
more bullet-proofing

Although it is top crash, I think it is just one user or one set of users - the windows version is exactly the same, and the additional dlls referenced under modules (e.g. RocketDock.dll) all appear to be the same (on a random straw poll of about 20 out of the 120 odd crashes listed).

So I think someone has a testcase ;-)

As we don't know who they are, we should definitely add this check for now.

Comment 40

[David :Bienvenu]

yes, and judging from the short uptimes, they're usually crashing fairly close to startup - one possible cause is gloda trying to index the same message.

Comment 41

[Wayne Mery (:wsmwk)]

v.fixed - last crash is from build 20090731