581940 - Drag and Drop of message to OS folder (e.g. Desktop) truncates filename prematurely and omits the .eml file extension if subject has ampersand character (&) or is empty

Status: RESOLVED FIXED

(Thunderbird :: OS Integration, defect)

Description

[Justin]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.7) Gecko/20100713 Thunderbird/3.1.1

Latest version of Thuderbird omits the "eml" extension when dragging and dropping an email to the desktop.

Also just as a note:
*Emails with no subject just give a filename as "null", should do "null.eml"
*Drag and drop to an IShellFolder in Vista does not work (bug 567354).
*Multiple item drag & drop would be nice

Reproducible: Always

Steps to Reproduce:
1.Select a single email
2.Drag it to the desktop or explorer window
3.Drop it there, the name is stored, but no eml extension as before
Actual Results:  
Email file is dropped but with no eml extension

Expected Results:  
Email file is dropped but with an .eml extension

Comment 1

[Justin]

Only seems to be a certain type of text email, header is as follows (edited to hide emails):

Return-Path: <[e-mail]>
Received: from mtain-mj02.r1000.mx.aol.com (mtain-mj02.r1000.mail.aol.com [172.21.164.86]) by air-md03.mail.aol.com (v129.4) with ESMTP id MAILINMD034-8b7f4c4b0d7940; Sat, 24 Jul 2010 11:57:45 -0400
Received: from mxpool19.ebay.com (mxpool19.ebay.com [66.135.197.25])
	by mtain-mj02.r1000.mx.aol.com (Internet Inbound) with ESMTP id 6B7B8380001D2
	for <[e-mail]>; Sat, 24 Jul 2010 11:57:42 -0400 (EDT)
Received: from sj-wconta005 (sj-wconta005.sjc.ebay.com [10.6.117.44])
	by mxpool19.ebay.com (8.13.8/8.13.8) with ESMTP id o6OFvWee006010
	for <[e-mail]>; Sat, 24 Jul 2010 08:57:42 -0700 (GMT)
DomainKey-Signature: a=rsa-sha1; s=dksm28; d=ebay.co.uk; c=nofws; q=dns;
	h=message-id:date:from:reply-to:to:subject:mime-version:
	content-type:content-transfer-encoding:x-ebay-mailtracker:x-ebay-mailversiontracker;
	b=J07efqtcNazv2jaL2M/TiNob/J35HArgtdJUWuHDzqO85nclpGyuhcdxj9K5aCShs
	5/Rcm6RnvBDfBopKcHghw==
Message-ID: <778317412.1279987062006.JavaMail.SYSTEM@sj-wconta005>
Date: Sat, 24 Jul 2010 08:57:42 -0700 (MST)
From: "eBay Member: userID" <[e-mail]>
Reply-To: [e-mail]
To: [e-mail]
Subject: wilko2929 has sent a question about: Payment & Checkout for item #111111111111, that ended on 24-Jul-10 16:53:50 BST - ++ PORTABLE DANCEFLOOR FLOOR for MARQUEE EVENT VENUE ++
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-eBay-MailTracker: 11050.677.3.2.8981225065382702558
X-eBay-MailVersionTracker: 677.11555337
x-aol-global-disposition: G
x-aol-sid: 3039400c89a44c4b0d7624c9
X-AOL-IP: 66.135.197.25
X-AOL-SPF: domain : ebay.co.uk SPF : pass
X-Mailer: Unknown (No Version)

                      =20

-----------------------------------------------------------------
eBay sent this message to Person Name (username).
Your registered name is included to show this message originated from eBay=

Comment 2

[Wayne Mery (:wsmwk)]

(In reply to comment #0)
> User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.8)
> Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729)
> Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.7)
> Gecko/20100713 Thunderbird/3.1.1
> 
> Latest version of Thuderbird omits the "eml" extension when dragging and
> dropping an email to the desktop.
> 
> Also just as a note:
> *Emails with no subject just give a filename as "null", should do "null.eml"
can you file a bug for this if one does not exist?

> *Drag and drop to an IShellFolder in Vista does not work (bug 567354).
> *Multiple item drag & drop would be nice
there's an existing bug report iirc.

> 
> Reproducible: Always
> 
> Steps to Reproduce:
> 1.Select a single email
> 2.Drag it to the desktop or explorer window
> 3.Drop it there, the name is stored, but no eml extension as before
> Actual Results:  
> Email file is dropped but with no eml extension
> 
> Expected Results:  
> Email file is dropped but with an .eml extension

WFM both to desktop and explorer window Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.8pre) Gecko/20100714 Lanikai/3.1.2pre - except for the null case mentioned above

perhaps the issue is specific to a type of message. can you attach such a testcase message to this bug?  or caused by an extension?

(not sev=major see https://bugzilla.mozilla.org/page.cgi?id=fields.html#importance - in fact there is a workaround, so normally we'd use "minor", unless it's a significant workflow issue)

Comment 3

[Justin]

> WFM both to desktop and explorer window Mozilla/5.0 (Windows; U; Windows NT
> 6.0; en-US; rv:1.9.2.8pre) Gecko/20100714 Lanikai/3.1.2pre - except for the
> null case mentioned above
> 
> perhaps the issue is specific to a type of message. can you attach such a
> testcase message to this bug?  or caused by an extension?
> 
> (not sev=major see
> https://bugzilla.mozilla.org/page.cgi?id=fields.html#importance - in fact there
> is a workaround, so normally we'd use "minor", unless it's a significant
> workflow issue)

See my previous post, has email header that causes it.  It is caused by text emails from eBay.

Comment 4

[Wayne Mery (:wsmwk)]

I can't test for now. FYI, testcases are somewhat easier to handle if attached as files.

xref bug 567507, also about extensions

Comment 5

[Justin]

Attachment: Testcase1.eml (long subject with & character)

The file name Thunderbird gave was "buyer has sent a question about Payment" and not "buyer has sent a question about Payment.eml" like it should have been.

Email addresses, usernames and Item ID have been replaced to hide identity.

Comment 6

[Justin]

(In reply to comment #4)
> I can't test for now. FYI, testcases are somewhat easier to handle if attached
> as files.
> 
> xref bug 567507, also about extensions

See attachment, changed emails, usernames and eBay item id to hide identities.  Filename is as Thunderbird made it (notice no .eml).

Comment 7

[Justin]

(In reply to comment #5)
> Created attachment 460869 [details]
> The should be .eml file dragged to desktop
> 
> The file name Thunderbird gave was "buyer has sent a question about Payment"
> and not "buyer has sent a question about Payment.eml" like it should have been.
> 
> Email addresses, usernames and Item ID have been replaced to hide identity.

I have just done a few tests, it is something to do with the following subject:

eBaybuyer has sent a question about: Payment & Checkout for item #111111111111, that ended on 24-Jul-10 16:53:50 BST - ++ PORTABLE DANCEFLOOR FLOOR for MARQUEE EVENT VENUE ++

I did a new email to myself with a couple of words in the body and used a long subject, even with a ':', all ok.  Once I used the above subject line with the same couple of words in the subject (they read "this is a test"), the bug re-appeared.

Comment 8

[Justin]

(In reply to comment #7)
> (In reply to comment #5)
> > Created attachment 460869 [details] [details]
> > The should be .eml file dragged to desktop
> > 
> > The file name Thunderbird gave was "buyer has sent a question about Payment"
> > and not "buyer has sent a question about Payment.eml" like it should have been.
> > 
> > Email addresses, usernames and Item ID have been replaced to hide identity.
> 
> I have just done a few tests, it is something to do with the following subject:
> 
> eBaybuyer has sent a question about: Payment & Checkout for item #111111111111,
> that ended on 24-Jul-10 16:53:50 BST - ++ PORTABLE DANCEFLOOR FLOOR for MARQUEE
> EVENT VENUE ++
> 
> I did a new email to myself with a couple of words in the body and used a long
> subject, even with a ':', all ok.  Once I used the above subject line with the
> same couple of words in the BODY(they read "this is a test"), the bug
> re-appeared.

Comment 9

[Thomas D. (:thomas8)]

confirming for the following two cases:

1) subject includes ampersand character (&)
subject:foo&bar -> filename after d&d: foo (expected: foo.eml)

2) subject is empty
subject: [empty] -> filename after d&d: null (expected: null.eml or something more intelligent and more scalable in case of dragging multiple msgs with empty subjects)

Comment 10

[Thomas D. (:thomas8)]

Related problems for consideration (needs new bug):

The filename created from long subjects is truncated after 124 characters without indication. E.g., sample subject of comment 7:

eBaybuyer has sent a question about: Payment & Checkout for item #111111111111, that ended on 24-Jul-10 16:53:50 BST - ++ PORTABLE DANCEFLOOR FLOOR for MARQUEE EVENT VENUE ++

After d&d, that's truncated like this:

eBaybuyer has sent a question about: Payment & Checkout for item #111111111111, that ended on 24-Jul-10 16:53:50 BST - ++ PORT.eml

That's less then one message reader's line of subject content on my screen (for the sample subject of comment 7), and 1 1/2 line in Winword, Times New Roman. Imo, that's a bit short, and we should indicate truncation in the filename, e.g. with a concluding ~ before the .eml extension (foo....bar~.eml). I also wonder how this scales for multiple emails that happen to have the same 124 characters in their subject.

Comment 11

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #9)
> confirming for the following two cases:
> 
> 1) subject includes ampersand character (&)
> subject:foo&bar -> filename after d&d: foo (expected: foo.eml)
Sorry, typo; should be:
subject:foo&bar -> filename after d&d: foo (expected: foo&bar.eml)

Comment 12

[Thomas D. (:thomas8)]

Attachment: Testcase2.eml (with ampersand in subject)

(In reply to Thomas D. from comment #9)
> confirming for the following two cases:
> 
> 1) subject includes ampersand character (&)
> subject:foo&bar -> filename after d&d: foo (expected: foo.eml)

Here's a reduced testcase for that.
Note how the file name (derived from subject) is truncated prematurely at the position of ampersand character in subject.

Comment 13

[Thomas D. (:thomas8)]

:thun,mail su:drag,d&d,drop su:eml,extension
https://bugzilla.mozilla.org/buglist.cgi?quicksearch=%3Athun%2Cmail%20su%3Adrag%2Cd%26d%2Cdrop%20su%3Aeml%2Cextension;list_id=4795425

No dupes found -> confirming

Comment 14

[Thomas D. (:thomas8)]

Attachment: Testcase3.eml (with empty subject)

(In reply to Thomas D. from comment #9)
> 2) subject is empty
> subject: [empty] -> filename after d&d: null (expected: null.eml or
> something more intelligent and more scalable in case of dragging multiple
> msgs with empty subjects)

Testcase for empty subject

Comment 15

[Mike Koleszar]

This bug is still present in Thunderbird 17.0.2 under Windows 7.

If an email is dragged to the desktop or another folder, and if the email's subject contains an ampersand ("&" character), the saved file's name will be truncated and not include the ampersand and any of the text after it. However, if the email is saved using the "Save As..." command, the file name will be correct.

For example, when an email with the subject "Ampersand Test & Missing Text" is dragged to the desktop or another folder, the file name will be "Ampersand Test", without the .eml extension. However, if the email is saved using the "Save As..." command, the file name will be "Ampersand Test & Missing Text.eml".

I suspect that only the email's subject affects this bug. I have tested this by sending an email to myself using Thunderbird, as well as Blat. Under both tests, the bug existed in the identical manner.

To test this, please send an email to yourself with an ampersand in the subject. Drag the email to the desktop and you should see a truncated file name with the .eml extension missing.

- Mike

Comment 16

[Thomas D. (:thomas8)]

(In reply to Mike Koleszar from comment #15)
> This bug is still present in Thunderbird 17.0.2 under Windows 7.

Hi Mike, short confirmation is helpful, but try to avoid extensive confirmation of scenarios that have already been confirmed (e.g. in my comment 9). What you can do, and what both description of comment 0 and my comment 9 haven't done yet, is to provide short & concise steps to reproduce in note form when the original steps are missing, unclear, or insufficient (as in this case), like this:

STR (steps to reproduce)

1 do this
2 do that
3 ...

Actual Result

- details of what actually happens

Expected Result

- details of what should have happened

Then, add [STR comment x] to whiteboard so that users reading the bug can easily find the better STR.

Comment 17

[Thomas D. (:thomas8)]

Aceman, this might be easy to fix. Something for you?

Comment 18

[:aceman]

Sorry, I do not use drag&drop.

Comment 19

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #10)
> Related problems for consideration (needs new bug):
> 
> The filename created from long subjects is truncated after 124 characters
> without indication. E.g., sample subject of comment 7:
> 
> eBaybuyer has sent a question about: Payment & Checkout for item
> #111111111111, that ended on 24-Jul-10 16:53:50 BST - ++ PORTABLE DANCEFLOOR
> FLOOR for MARQUEE EVENT VENUE ++
> 
> After d&d, that's truncated like this:
> 
> eBaybuyer has sent a question about: Payment & Checkout for item
> #111111111111, that ended on 24-Jul-10 16:53:50 BST - ++ PORT.eml

Well, to avoid running into the ampersand bug, there shouldn't be any ampersands for testing the truncation of long subjects (so the above won't work yet)...

- drag msg with following subject to OS folder:

eBaybuyer has sent a question about  Payment Checkout for item #111111111111, that ended on 24-Jul-10 16 53 50 BST - ++ PORTABLE DANCEFLOOR FLOOR for MARQUEE EVENT VENUE ++.eml

actual result:
- gets truncated after 123 characters without indication of truncation, like this:

eBaybuyer has sent a question about Payment Checkout for item #111111111111, that ended on 24-Jul-10 16 53 50 BST - ++ PORT.eml

expected result:
- file name should be full subject + .eml extension
- or at least truncation should include more characters, and be indicated somehow (e.g. as blabla~.eml)

Comment 20

[Thomas D. (:thomas8)]

Perhaps this just needs a better encoding or quoting of the file name?

Comment 22

[:aceman]

Attachment: WIP patch

I can fix the empty subject and add the ~ if subject is too long. But dragging to my KDE desktop didn't use the generated file name (but asked for new one). So I would be grateful if anybody could try the patch on Windows.

Comment 23

[Suyash Agarwal (:sshagarwal)]

Comment on attachment 8391899 [details] [diff] [review]
WIP patch

All the three testcase files are saved correctly when dragged and dropped.
Message with empty subject is saved as "message.eml", long subjects are truncated with a tilde sign. & is saved in the filename.

Thanks.

Comment 24

[Thomas D. (:thomas8)]

Comment on attachment 8391899 [details] [diff] [review]
WIP patch

Review of attachment 8391899 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks aceman for fixing this.

From my reading (can't test patches), this patch does exactly what we wanted to fix here.

I was hesitating with feedback because I wanted to look into truncation and ux-consistency between various flavors of saving some more (d&d vs. Save As; single vs. multiple msgs), but it turns there's a can of worms and bugs in this corner when it comes to that. I think we'll have deal with such issues in a more systematic approach in other bugs. This patch is definitely an improvement over the status quo, so let's land what we have and take it from there.

::: mail/base/content/msgMail3PaneWindow.js
@@ +1372,5 @@
>    // But first ensure suggestUniqueFileName is efficient enough on 10000+ dragged
>    // messages.
> +  let messengerBundle = document.getElementById("bundle_messenger");
> +  let noSubjectString = messengerBundle.getString("defaultSaveMessageAsFileName")
> +                                       .replace(".eml", "");

This assumes that localizers must never use ".eml" in the base file name part, e.g. the must not define:
defaultSaveMessageAsFileName=.emlfile.eml
Fair enough, they probably won't.

@@ +1373,5 @@
>    // messages.
> +  let messengerBundle = document.getElementById("bundle_messenger");
> +  let noSubjectString = messengerBundle.getString("defaultSaveMessageAsFileName")
> +                                       .replace(".eml", "");
> +  let longSubjectString = messengerBundle.getString("longMsgSubjectTruncator");

For easier reading, perhaps better:

let longSubjectTruncatorString = ...

@@ +1391,5 @@
> +    if (!subject) {
> +      uniqueFileName = noSubjectString;
> +    } else {
> +      uniqueFileName = (subject.length <= 124) ?
> +                       subject : subject.substr(0, 123) + longSubjectString;

dito:

... + longSubjectTruncatorString;

Comment 25

[Thomas D. (:thomas8)]

Mental note of bugs and ux-inconsistencies when saving messages as .eml files:

single msg
- d&d -> truncatedsubject123chars~.eml
- save as -> fullsubject.eml (for shorter subjects)
          -> or no file name suggested in the dialogue (for longer subjects)

multiple msgs
- d&d -> not yet possible (existing bug)
- save as -> fullsubject - 'SenderDisplayName' ([e-mail]) - timestamp.eml (for shorter subjects)
          -> failing silently, without error msg, without saving any files, if subject of one msg is too long

Preliminary thoughts & conclusions:

1) So depending if user save's single or multiple msgs, target file name differs (and I suspect it shouldn't - sequential saving of single msgs should arguably yield same result as saving multiple msgs at once).
2) Even for saving single msg, target file name length differs depending if d&d or "save as" method was used (certainly shouldn't).
3) When total length of target file path exceeds 260 chars OS limit (on windows), we fail silently. Which can happen easily even for short subjects, e.g. if 'SenderDisplayName' and/or sender's email address and/or target folder path are longer. Ebay testcase above shows such things actually happen in real life.

Comment 26

[Thomas D. (:thomas8)]

Windows 260 chars limit for file path:
http://windows.microsoft.com/en-us/windows/file-names-extensions-faq#1TC=windows-7

Comment 27

[:aceman]

(In reply to Thomas D. from comment #25)
> single msg
> - save as -> fullsubject.eml (for shorter subjects)
>           -> or no file name suggested in the dialogue (for longer subjects)
> 
> multiple msgs
> - save as -> fullsubject - 'SenderDisplayName' ([e-mail]) -
> timestamp.eml (for shorter subjects)
>           -> failing silently, without error msg, without saving any files,
> if subject of one msg is too long
>
> 3) When total length of target file path exceeds 260 chars OS limit (on
> windows), we fail silently. Which can happen easily even for short subjects,
> e.g. if 'SenderDisplayName' and/or sender's email address and/or target
> folder path are longer. Ebay testcase above shows such things actually
> happen in real life.
Is there a bug for this?

Comment 28

[:aceman]

Attachment: patch v2

Thanks, fixed the nits ;)

Comment 29

[Magnus Melin [:mkmelin]]

Comment on attachment 8399559 [details] [diff] [review]
patch v2

Review of attachment 8399559 [details] [diff] [review]:
-----------------------------------------------------------------

Looks ok to me. r=mkmelin

::: mail/base/content/msgMail3PaneWindow.js
@@ +1392,5 @@
> +    if (!subject) {
> +      uniqueFileName = noSubjectString;
> +    } else {
> +      uniqueFileName = (subject.length <= 124) ?
> +                       subject : subject.substr(0, 123) + longSubjectTruncator;

where does the 124 come from? (also add a code comment about what this is all about)

Comment 30

[:aceman]

Attachment: patch v3

Thanks, done.

Comment 31

[Josiah Bruner [:jsbruner]]

Hmm, so this isn't good, but I am unable to move any mail to the desktop on OS X. I can save files fine, but dragging them doesn't do anything.

No messages in the error console though.

Comment 32

[:aceman]

Yeah, this is mainly for Windows. If you can't test it there please forward to other ui-reviewer. But you can actually judge the patch by reading the code :) You do not need to review drag and drop but about the format of the file name produced.

Comment 33

[Josiah Bruner [:jsbruner]]

I don't mind testing it on Windows if you could get me a packaged executable. :)

Comment 34

[:aceman]

I can't but you can push it to try server :) Or edit your omni.ja in an existing TB windows build :)

Comment 35

[Josiah Bruner [:jsbruner]]

Comment on attachment 8400859 [details] [diff] [review]
patch v3

Review of attachment 8400859 [details] [diff] [review]:
-----------------------------------------------------------------

This look fine to me, except I think we should use triple periods "..." instead of a '~', since '~' is commonly used for prefixing files that are being accessed E.G.

file.txt
~file.txt

Where ~file.txt is the file being temporarily written to.

Other than that ui-r+.

Comment 36

[Thomas D. (:thomas8)]

(In reply to Josiah Bruner [:JosiahOne] from comment #35)
> Comment on attachment 8400859 [details] [diff] [review]
> patch v3
> 
> Review of attachment 8400859 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> This look fine to me, except I think we should use triple periods "..."
> instead of a '~', since '~' is commonly used for prefixing files that are
> being accessed E.G.
> 
> file.txt
> ~file.txt
> 
> Where ~file.txt is the file being temporarily written to.
> 
> Other than that ui-r+.

After initial hesitation, I actually like that idea.
To advanced users, ~ indeed looks like an indication of temporary or incomplete files.
To normal users, ~ means nothing, but might be irritating, whereas ... is the normal way of indicating shortened file names (e.g. when column width is too small to show all). I tried it and it actually looks natural and self-explanatory. We really want three dots as Truncator so the actual file name will correctly end up with 4 dots:

Message with a very long subject which....eml

That's required because known extensions are hidden by default on windows, so for most users, it looks like this:

Message with a very long subject which...

Comment 37

[:aceman]

Attachment: patch v4

Thanks, done.

Comment 38

[:aceman]

Thomas, please file the bug about all this logic not being applied to Save As action.

Comment 39

[Thomas D. (:thomas8)]

FTR:

Current patch truncates subjects longer than 124 characters to become 121 characters + "..." Truncator.
+  // Clip the subject string to 124 chars to avoid problems on Windows,
+  // see NS_MAX_FILEDESCRIPTOR in m-c/widget/windows/nsDataObj.cpp .
+  const maxUncutNameLength = 124;

Then we call
> suggestUniqueFileName(uniqueFileName, ".eml", fileNames)
which adds ".eml" extension, resulting in 128 characters for the file name (without path).
(Would be interesting to find out if suggestUniqueFileName might return longer filenames if the input was not unique?)

128 characters limit is mentioned here:
http://mxr.mozilla.org/comm-central/source/mozilla/widget/windows/nsDataObj.cpp#347

341 /* 
342  * deliberately not using MAX_PATH. This is because on platforms < XP
343  * a file created with a long filename may be mishandled by the shell
344  * resulting in it not being able to be deleted or moved. 
345  * See bug 250392 for more details.
346  */
347 #define NS_MAX_FILEDESCRIPTOR 128 + 1

I'm a bit surprised that we're handling file name length on the app side - suppose the OS should complain if it isn't happy with the length, otherwise leave it to the user.
Also surprising that we don't look into the total path length, which is the real limit on Windows per comment 26:

> Windows has 260 chars limit for file path:
> http://windows.microsoft.com/en-us/windows/file-names-extensions-faq#1TC=windows-7

The comment in nsDataObj.cpp seems to suggest that we would be safe for more flexible behaviour for OS >= Windows XP. I don't think we're still supporting anything below Windows XP, and even Windows XP is on its way out...

Comment 40

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/comm-central/rev/15c1bea389b2

Comment 42

[Tiger]

TB 45.3.0 on W10 still truncates filenames to 128 chars length when emails dragged and dropped (eg on the Desktop). The OS imposed filename-length limitation has gone for decades now. 

This is a serious limitation if the drop-recieving application (like mine) relies on patterns in the subject (=filename). Information gets lost. To reproduce, send an email having a subject > 128 chars to yourself and d&d on your desktop. 

It works as expected when used in "Save as..." context menu, so this should be a low hanging fruit. Users often refuse "Save as..." when d&d is supported.

Just trying to resurrect this issue, since imho it is not really fixed. Found no better place.