230293 - CSV needs to be sent with Content-Disposition: attachment

Status: RESOLVED FIXED

(Bugzilla :: Query/Bug List, defect)

Description

[Asger Jensen]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6a) Gecko/20031030
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6a) Gecko/20031030

I don't know if its just me, or my locale, that makes this a problem, but my
Windows defaults to ";" as CSV seperator (from Regional settings under control
panel).

Is it possible to change the "," seperator manually in the current codebase
(2.17-6), or would it be possible to implement it as a user-config option?

Right now, Excel refuses to acknowledge the columns when copy/pasting the text
from a buglist (Excel 2000).


Reproducible: Always

Steps to Reproduce:
1. Log-in
2. Press My Bugs
3. Press CSV
4. Press Ctrl-A + Ctrl-C
5. Switch to Excel
6. Press Ctrl-V


Actual Results:  
The whole text was included in column a. No subsequent columns was made.

Expected Results:  
1 column in excel / column in bug list

Comment 1

[Stephen Lee]

For a copy-paste of text, excel uses whatever separator(s) you last asked for 
in the text import wizard, or using the "Data" -> "Text to Columns..." menu 
option. Semicolon isn't selected by default either - Tab is the character it 
generally uses as a separator in copy-pasted text.

Hmmm.... why are you copy-pasting rather than just loading the CSV file direct 
into excel anyway - when I do it, it just pops up straight in excel with no 
copy-paste needed.


tum-te-tum....

HTTP/1.1 200 OK
Date: Thu, 08 Jan 2004 10:55:35 GMT
Server: Apache/1.3.27 (Unix)  (Red-Hat/Linux) mod_ssl/2.8.12 OpenSSL/0.9.6b 
DAV/1.0.3 PHP/4.1.2 mod_perl/1.26
Content-Disposition: inline; filename=bugs-2004-01-08.csv
Connection: close
Content-Type: text/plain

bug_id,"severity","priority","platform","owner","status","resolution","summary",
[...]


text/plain ???  shouldn't that be text/csv or similar?

Using internet explorer as the browser, it interprets the .csv and just loads 
it into excel anyway, but presumably Mozilla respects the returned mime type 
instead, and displays it as plain text?


Tweaking Summary, and putting across into correct component for buglist.cgi 
problems - suggest morphing bug into a request to return as text/csv so that 
browsers stand a better chance of knowing what to do with it...

Comment 2

[Asger Jensen]

Correct, I have done additional testing on more machines, and on one (Using IE 
5.5) Excel loaded up inside the browser frame, but still ignored the "," as 
seperator. Perhaps this can be fixed by doing a dummy import in excel, but I 
would rather it did it out-of-the-box.

I then tried to change the content-type as your suggested, by modifying
Bugzilla/Constants.pm

changing
use constant contenttypes =>
..
   "csv" => "text/plain"


to 


..
    "csv" => "text/csv"


And it worked splendidly. Not only did mozilla open Excel (it didn't before), 
but Excel also caught on to the idea of a "," seperator.

Thanks.

I Checked both IE and Mozilla

Perhaps this will not be an option for others, i.e. they want to do something 
else with the CSV than showing it in Excel, so perhaps _this_  could be user 
configurable. i.e. param(csvcontenttype) style stuff.

Comment 3

[Stephen Lee]

Ok... so as far as I can see the correct fix is to do the text/csv thing.

If someone wants to do something else with CSV files, they can configure that 
in their browser or file associations, or click "save target as..." or 
equivalent on the "CSV" link.

Confirming bug...

Comment 4

[Stephen Lee]

Attachment: Suggested patch implementing changes suggested in comment #2

Comment 5

[Bradley Baetz (:bbaetz)]

I'm sure that htere was a reason for that, but I don't know what it is... gerv?

Comment 6

[Gervase Markham [:gerv]]

This content-type originally went in in bug 12282, and there's no text there
justifying my choice. text/csv (assuming it's a registered MIME type) certainly
seems like a better choice.

Gerv

Comment 7

[Gervase Markham [:gerv]]

Comment on attachment 138691 [details] [diff] [review]
Suggested patch implementing changes suggested in comment #2

r=gerv.

Gerv

Comment 8

[Vlad Dascalu]

Thanks Asger for the suggestion!

Comment 9

[Vlad Dascalu]

Checking in Constants.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Constants.pm,v  <--  Constants.pm
new revision: 1.4; previous revision: 1.3
done

Comment 10

[Stephen Lee]

Ok... there seems to be a small spanner in the works here....

http://www.iana.org/assignments/media-types/text/
does not list any MIME type for csv, although there is one for tsv:
http://www.iana.org/assignments/media-types/text/tab-separated-values

I find it astonishing that such a widely used data type is not on the "official 
list". Having seen other references to "text/csv" as a mime type, it didn't 
even occur to me to check before suggesting its use. It appears (from comment 2)
that at the very least Mozilla understands what text/csv means, and since any 
unrecognised text/* SHOULD be interpreted as text/plain, browsers will render 
it no worse than before the patch.

The rest of the world seems to be of mixed opinion between using:
text/csv
text/comma-separated-values
application/vnd.ms-excel

the last of these is at least on the "official list", but implies that other 
spreadsheet / database apps wouldn't understand it, and doesn't make clear 
the "text-ness" of the format.

Perhaps all it needs is for someone to go to
http://www.iana.org/cgi-bin/mediatypes.pl
and enter something similar to what it says for the TSV format?

Comment 11

[Gervase Markham [:gerv]]

Getting a media type registered is a big pain. The XUL one took ages, and it's
in the vendor tree not the main one.

I'd just stay with what we have.

Gerv

Comment 12

[Dave Miller [:justdave]]

we could always do "Content-Disposition: attachment" which would force the
browser to offer a download dialog instead of displaying the page.  This is what
I recommended when this was first implemented, and several people vetoed that so
it got left for the browser to display in the window.

I'll re-introduce that suggestion though.  95% of the time when you run a CSV,
you're intending to download the file in order to feed it to some other app. 
With the current behavior, you have to right-click on the CSV link and do "Save
As..." to get that, which makes it a pain in the butt to actually export it
anywhere.

Comment 13

[Gervase Markham [:gerv]]

Why'd it get vetoed?

Gerv

Comment 14

[Dave Miller [:justdave]]

You tell me, you're the one that vetoed it. :)  (bug 177430 comment 12)

OK, so it was one veto and not several...

Note that the patch on this bug did get checked in changing the content type to
text/csv (which is apparently not actually legal).  Do we need to back that out
and reopen this then?

Comment 15

[Gervase Markham [:gerv]]

Hmm. Let me change my mind, then :-) After all, I don't use the CSV format, and
if people who are using it are saying that it's for downloading, let's make it
easier to download.

The original logic was that, if it's displayed in-browser, people have a choice...

As for text/csv... let's switch back to text/plain and use Content-Disposition,
and see if that does the trick for Excel.

Gerv

Comment 16

[Dave Miller [:justdave]]

backed out.

Checking in Bugzilla/Constants.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Constants.pm,v  <--  Constants.pm
new revision: 1.5; previous revision: 1.4
done

Comment 17

[Stephen Lee]

Reassigning to default component owner - my involvement was merely to 
mistakenly suggest text/csv and to put the reporter's comment 2 into patch form.

Comment 18

[Asger Jensen]

Too bad about text/csv. 

hmm, what was cool before was that I didn't need to put my temporary data in a 
file somewhere on my desktop just to make a SUM of Orig.Est time. 

I just pressed "CSV", excel started on the data, press column E, press sum. 

Any chance of a param(csvcontenttype) ? 

Otherwise, I can probably manage to fix it on my own install.

Regards

Comment 19

[Vlad Dascalu]

Clearing approval flag on an opened bug with no patch to check in.

Comment 20

[timeless]

i'm with the old gerv. i like reading csv in browsers. i don't like excel :).

note that mozilla's handling of text/* is fairly poor, and text/csv approaches 
the edge case (text/rtf and text/csh and text/totally-incomprehensible+xml are 
among the reasons why mozilla doesn't consider text/* like text/plain - there's 
of course also a thing about not being able to volunteer as a catch all later) 
any fixes for those bugs in mozilla shouldn't be expected for a few releases.

Comment 21

[Dave Miller [:justdave]]

Joel: there's nothing to approve here yet.  See comment 16 and comment 19.

Comment 22

[Gervase Markham [:gerv]]

Attachment: Patch v.1

Change the Content-Disposition to "attachment".

Hmm. Perhaps this should be generalised to an extra URL parameter, much like
"format" and "ctype"... GetFormat would need to take a CGI object, and we'd
need to use $format->{'disposition'}.

Anyone think that's worth it? It's a fairly big API change to GetFormat...

Gerv

Comment 23

[Dave Miller [:justdave]]

I'm not sure that's worth it.  URL editing generally isn't fun for humans, and
most scripts aren't going to care what the disposition is.

Comment 24

[Gervase Markham [:gerv]]

Fixed.

Checking in buglist.cgi;
/cvsroot/mozilla/webtools/bugzilla/buglist.cgi,v  <--  buglist.cgi
new revision: 1.250; previous revision: 1.249
done

Gerv

Comment 25

[Stephen Lee]

I did some CSV-related tests on http://landfill.bugzilla.org/bugzilla-tip/


1) did a search (any search as long as it returns results!) and click on 
the 'CSV' link. Example:
http://landfill.bugzilla.org/bugzilla-tip/buglist.cgi?bug_id=1844&ctype=csv
This causes the CSV file to be saved in a temporary file and opened in an 
external browser.

2) uploaded a CSV attachment, with the type set to "auto-detect" onto that bug:
( http://landfill.bugzilla.org/bugzilla-tip/attachment.cgi?id=238&action=view )
Clicking on the attachment causes it to appear in the browser window. IE6 and 
Bugzilla seem to have decided on a content type of application/octet-stream 
between them (not exactly very descriptive :-( )

3) uploaded the same attachment, forcing the content-type to text/plain.
( http://landfill.bugzilla.org/bugzilla-tip/attachment.cgi?id=239&action=view )
behaviour the same as in 2.

4) When looking at both of the above attachments:
http://landfill.bugzilla.org/bugzilla-tip/attachment.cgi?
bugid=1844&action=viewall
the one uploaded in 3 is displayed as an excel object in the browser window. 
See screenshot:
( http://landfill.bugzilla.org/bugzilla-tip/attachment.cgi?id=240&action=view )

This is inconsistent behaviour, as CSV output is treated one way in one case, 
and a different way in another case. Other OS / Browser / Spreadsheet 
combinations may vary. Mine is W2K / IE6 / Excel 2002.

Worth reopening?

Comment 26

[Gervase Markham [:gerv]]

1), 2) and 3) are all correct behaviour IMO. Attachments are not the same thing
as buglists. I don't think we want to get into allowing people to set
content-disposition for attachments.

Gerv

Comment 27

[Stephen Lee]

(In reply to comment #26)
> 1), 2) and 3) are all correct behaviour IMO. Attachments are not the same
> thing as buglists.

Fair enough...

As long as behaviour on other browsers (esp. Mozilla) is correct in this 
situations, I'm happy with that.

Comment 28

[David Geldreich]

(In reply to comment #10)
> Ok... there seems to be a small spanner in the works here....
> 
> http://www.iana.org/assignments/media-types/text/
> does not list any MIME type for csv, although there is one for tsv:

Hello text/csv is not registered to the iana, so it is perhaps time to put this back in ?

See my bug 356807 for continuation of this topic.

Comment 29

[Gervase Markham [:gerv]]

> Hello text/csv is not registered to the iana,

I assume you meant "now registered"? :-)

Discussion going on in bug 356807.

Gerv