Closed Bug 355237 Opened 20 years ago Closed 5 years ago

mbox parser gets confused by "From " at beginning of line

Categories

(MailNews Core :: Movemail, defect)

defect
Not set
major

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: g.s, Assigned: aceman)

References

Details

Attachments

(3 files)

User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.8.0.7) Gecko/20060910 SeaMonkey/1.0.5 Build Identifier: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.8.0.7) Gecko/20060910 SeaMonkey/1.0.5 There's a bug in the mbox file parser. If Thunderbird/Seamonkey opens a file containing plain text mails with "From " at the beginning of a line, this is interpreted as the beginning of a new email. When Seamonkey/Thunderbird adds mails to an mbox file, it modifies "From " to ">From" in the file. But if you use movemail, the "From " is stored without being quoted and then Seamonkey/Thunderbird gets confused. Reproducible: Always Steps to Reproduce: 1. Configure Seamonkey/Thunderbird to use movemail 2. Send yourself an email with "From " at the beginning af a line (doesn't work if sending from Seamonkey/Thunderbird, they insert a ">" before sending) 3. Receive this mail Actual Results: This one email is interpreted as two seperate emails. Expected Results: Display the email as one email with the Line containing the "From ". Happens in Seamonkey 1.0.x on Linux and Windows.
that's a bug in the movemail code - the mbox parser is doing what it's supposed to do.
-> movemail
Assignee: mscott → pkwarren
Component: General → MailNews: Movemail
Product: Thunderbird → Core
QA Contact: general → movemail
Version: unspecified → Trunk
Attached file Export from kmail —
Here's an mbox file exported from kmail. It prepends > in front of From. It displays fine in kmail but the > is visible in Thunderbird.
I have modified the export to remove the leading ">". If I copy this into the Mail directory for Thunderbird then it believes there are three messages in the file when there are only two.
Based on my comments on the attachments above I don't believe that this is a movemail bug. I see this problem on Thunderbird on Windows and movemail shouldn't apply to that.
Following page says that there are two version of movemai; ( http://www.gnu.org/software/emacs/manual/html_node/emacs/Movemail.html ) - "emacs version" (is able to retrieve mail from usual UNIX mailbox formats) - "mailutils version" (is able to handle a wide set of mailbox formats) mailbox URLs in "mailutils version" is; proto://[user[:password]@]host-or-file-name And "proto" for local mailbox can be mbox, mh, maildir, file. To g.s@arcor.de(bug opener): Which version of movemail do you use? If "mailutils version", "proto" is "mbox"? Or other? Is there any way to know which "proto" movemail is using?
How can I find out which version the system is using? I've never started an emac on that machine and it's a RedHat Enterprise 4 with sendmail-8.13.1-3.RHEL4.5.
Regardless of what is morally correct and pure and blah, blah, blah, ... the days of having these hardcoded assumptions ought to be behind us. It would be very easy for TB to grok "foreign" mbox files without requiring an explicit conversion step. I haven't run this down in the code, but it looks like TB is currently looking for just From_ at the beginning of a line. Of course, an actual mbox timestamp is more than that. There are lots of variations in the wild, and if TB looked a bit further along the line, perhaps requiring a regex match (controlled by a config variable), that would be quite handy and save a lot of people a lot of frustration. (I myself sometimes share with TB mbox files created by the emacs mailer VM. VM has a list of regexes for various popular mbox types. If anyone is interested, look for the function vm-find-leading-message-separator in vm-folder.el.)
Product: Core → MailNews Core
Confirmed with current tip of comm-central. TB sees text ">From " as ">From " and "From " as a new delimiter. I'm not sure what most linux distributions use as the format, but mboxrd (s/^\(>*\)From />\1From /) is probably the most common and safest. Other formats: mboxo (s/^From/>From/) mboxcl (mboxo + Content-Length header) mboxcl2 (Content-Length header only)
Status: UNCONFIRMED → NEW
Ever confirmed: true
Here are two discussions from the support forums about this bug. The first investigates the bug seriously and goes into quite a bit of detail. http://forums.mozillazine.org/viewtopic.php?f=39&t=1001425 http://forums.mozillazine.org/viewtopic.php?f=39&t=375641 I have experienced this bug with older mail folders in mbox format that were created in Pine. Any messages with plain text lines beginning in "From " are split if the folders are viewed in Thunderbird. I would appreciate this bug being prioritized.
I think we can make the "From " parsing more robust and check if there is the timestamp after it.
Assignee: pkwarren → acelists
When I tested this now, when composing a message in TB and there is a "From " line in the message body, it is prepended by " ", not by ">". Can anybody confirm it?
Attachment #273252 - Attachment mime type: application/octet-stream → text/plain
Attachment #273253 - Attachment mime type: application/octet-stream → text/plain
(In reply to :aceman from comment #13) > When I tested this now, when composing a message in TB and there is a "From" line in the message body, > it is prepended by " ", not by ">". > Can anybody confirm it? When mail composition, Mozilla MailNews and old Tb versions escaped "From" line by ">From" in msgStore file(Save as draft), and removed ">" upon mail display(Edit draft, Edit as new, ...). IIRC, this behaviour was changed by a Tb release. "From line" is stored as " From" in msgStore file, and " " for escaping is not removed upon next Edit draft. This is perhaps for "format=flowed". If format=flowed, " From" is perhaps better than ">From".
Attached patch WIP patch — — Splinter Review
So what do you think about this: - we parse the spool file in "strict mode" checking proper format of the timestamp (it accepts what I found on my system and also the sample attachments). - if that succeeds we can properly escape false "From " delimiters in the message body. - but if it fails (no proper header is found), we retry the file parsing in "non-strict" mode, ignoring the timestamps. That basically falls back to the current parsing. So on system that do not have the supported timestamp formet there should be no change. Ok, there is a small change - I also check if the line preceding a "From " was empty (as described by RFC4155). If you do not want this change just say so. If the approach passes, I try to make some test for it.
Attachment #8424945 - Flags: feedback?(Pidgeot18)
The exact meaning of this bug, the comments associated with this bug, and the patch attempt to fix this bug are all offering different interpretations. The problem is that we have several "mbox parsers", and people are offering different evidences of failures for different bugs. The original interpretation of this bug is that when the following line is seen in the movemail spool file (i.e., nsMovemailService.cpp): >From[space] we should emit From[space] instead in the message body. Subsequent comments by WJCarpenter or Ian Watson imply that they are having issues with the regular mbox storage backend (i.e., nsParseMailbox.cpp). I think it is best to keep this bug focused on the issue that we should delete the leading > from ">From " lines.
Comment on attachment 8424945 [details] [diff] [review] WIP patch Review of attachment 8424945 [details] [diff] [review]: ----------------------------------------------------------------- I'm wary of doing things here that would break the common mbox formats, at least not without specific and compelling evidence that it won't break us on common distributions. I do know that I've seen code that emits mbox separators as "From -". This also doesn't solve the real problem here: that we're not stripping the > in >From.
Attachment #8424945 - Flags: feedback?(Pidgeot18) → feedback-
While it's on my mind, after reviewing <http://www.jwz.org/doc/content-length.html>, it's very likely that any handling of the distinction between mboxo and mboxrd ought to be controlled by a preference.
(In reply to Joshua Cranmer [:jcranmer] from comment #17) > I'm wary of doing things here that would break the common mbox formats, at > least not without specific and compelling evidence that it won't break us on > common distributions. I do know that I've seen code that emits mbox > separators as "From -". And such separators will be still supported in my patch. > This also doesn't solve the real problem here: that we're not stripping the > > in >From. I do not see anywhere in the original comment that this is the request. I understand the original comment that if the 'From ' IS NOT escaped in any way then TB splits it into a new message. That is fixed by the patch IF the timestamps after the 'From ' match the expected format. We can argue about the precise format. (In reply to Joshua Cranmer [:jcranmer] from comment #16) > Subsequent comments by WJCarpenter or Ian Watson imply that they are having > issues with the regular mbox storage backend (i.e., nsParseMailbox.cpp). No, I assume my interpretation is supported by the comments of g.s@arcor.de, WJCarpenter and Ian Watson. They talk about parsing the "foreign" mbox file, not our in nsParseMailbox.cpp .

Interesting bug. I tried to put some old mbox files under Local Folders that had been saved long ago from a tb ImapMail mbox file. It worked OK but several emails had today's time/date and appeared truncated and had no subject or other header info shown. I didn't know about this bug and traced it to several the leading From's in the message bodies.
Adding a space in front of the leading "From" fixed the problem.
I don't know if this is a Local Folder only problem since I probably would have noticed the problem back when the files were used in Imap.
Also, this does seem to be case sensitive since a leading "from" didn't cause a problem.

There seems to be a fix for the problem by defining "STRICT_ENVELOPE" here and rebuilding:
https://searchfox.org/comm-central/rev/551ff478566e07e5854ade095b6d7ac74cb8a0d4/mailnews/local/src/nsParseMailbox.cpp#659

Now for mbox files placed in Local Folders the parsing of the file ignores the "From " at the beginning of lines of the message body instead of erroneously starting a new message. One of my mbox files had about 20k messages and over 300 messages got fragmented and had to be manually edited. With STRICT_ENVELOPE defined there were zero problems and no manual cleanup.

This may also fix problems that originated from a "movemail" setup, but not sure. Maybe aceman's patch above is also needed for movemail even with STRICT_ENVELOPE defined? Looking some at his patch, it appears it is fixing up the mbox file created by movemail so a non-strict envelope parse of mbox file will work. So his fix will probably fix the problem for movemail but it will not fix a legacy mbox file manually placed into local folders.

Don't know why STRICT_ENVELOPE code is not enabled. Maybe it's too strict as per comment 18 that links to this: http://www.jwz.org/doc/content-length.html. Or maybe with it some 3rd party mbox imports would fail (but it does seem to be strict but somewhat tolerant)?
Edit: Well, I see why. I just ran into a 3rd party mbox file that separates messages with just "From " (no user or timestamp following). With strict envelope defined, only the 1st message in the mbox is visible in tb. Also, tb deletes all the other stuff from the file so that an mbox file that started out as multi-MB in size becomes a few KB in size. (The 3rd party file, is in mboxo format since lines beginning with "From" are ">From". TB just displays them as ">From" which is another issue. TB should strip off the leading ">" before displaying the email.)

FWIW this link provides the best explanation I've found of the various mbox* formats: https://jdebp.eu/FGA/mail-mbox-formats.html.

I see the same bug also when copying Imap mbox files to Local that were created by the same version of TB (78.4.3 in my case).
I recently added and synchronized an imap profile, then deleted the account and manually imported/moved the mboxes in the Local folder. Emails beginning with "From" are parsed incorrectly and added as unread fragments.

Is there a difference between imap mbox parsing and local folder mboxes?

Support for Movemail has been removed as of version 87 per Bug 1625741

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX

I don't remember any details on this even though I researched and wrote about it 8 months ago. But I think my conclusion was, and seems to be supported by comment 23, that a problem exists independent of movemail (now removed). Not sure how big a problem it is or if it really needs to be fixed, so maybe WONTFIX is still appropriate.

(In reply to gene smith from comment #25)

I think my conclusion was [comment 22?], and seems to be supported by comment 23, that a problem exists independent of movemail (now removed). Not sure how big a problem it is or if it really needs to be fixed, so maybe WONTFIX is still appropriate.

I have no qualms about reopening. Let's see what others think.

Flags: needinfo?(mkmelin+mozilla)

The symptoms still apply, so if you have a mbox with such data it will display slightly wrong.
But the point is, you will not have such a file, because Thunderbird will escape the From before adding it to the mbox.

Flags: needinfo?(mkmelin+mozilla)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: