386671 - Add csv export to graph server

Status: RESOLVED FIXED

(Webtools Graveyard :: Graph Server, defect)

Description

[Mike Schroepfer]

Given a particular graph e.g.:

http://graphs.mozilla.org/dgraph.html#spst=range&spstart=0&spend=392&bpst=cursor&bpstart=0&bpend=392&m1tid=37&m1bl=0&m1avg=0

It would be useful to have a "save as csv" option to export the data as is in the graph to csv for further offline analysis

Comment 1

[alice nodelman [:alice] [:anode]]

I think that there is a simple way to do this with sqlite databases - as they already have built in export to csv functions.  I'm going to play around with that and see if I can get something working quickly.

Comment 2

[alice nodelman [:alice] [:anode]]

oops - assigning to me

Comment 3

[alice nodelman [:alice] [:anode]]

Attachment: adds dump to csv option to graph.html and dgraph.html

Add support to both dgraph and graph.html for dump to csv.  Behaves in much the same way as the "Link to this graph" function, ie.  you have to graph something before you can dump it.  It can dump multiple datasets at once, so whatever is currently graphed will be dumped.

Currently dumps all the data in a given set - in future should be able to limit by a time range.

Comment 4

[alice nodelman [:alice] [:anode]]

Attachment: database alterations for dump to csv

Some added indexes to the database schema so that the dump to csv isn't painfully, painfully slow.  This should be run against existing databases, new databases will be created with these indexes.

Comment 5

[Jeremy Orem [:oremj]]

Looks like we are porting graphs.m.o to MySQL.  Adding Bug 387141 as a depend to this bug.

Comment 6

[alice nodelman [:alice] [:anode]]

Attachment: version #2: add dump to csv option to graph.html & dgraph.html

Previous version of the diff broke bulk.cgi.  This is a very minor change to stop that breakage.

Comment 7

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Comment on attachment 272117 [details] [diff] [review]
version #2: add dump to csv option to graph.html & dgraph.html

Looks fine, though in dumpData you'll need to quote each arg.. I'm not sure what the quoting rules are in CSV, but especially if you're dumping raw data, there will almost definitely be commas in the output.

r+ with that change

Comment 8

[alice nodelman [:alice] [:anode]]

Attachment: version #3: add dump to csv option to graph.html & dgraph.html

Fixed up the csv formatted outputted by dumpdata.cgi by adding quotes around the values, also correctly escaping quotes within the data values themselves.

Comment 9

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Comment on attachment 272233 [details] [diff] [review]
version #3: add dump to csv option to graph.html & dgraph.html

i'm not a super big fan of the magic "Tinderbox.raw = 1", but meh.

Comment 10

[Reed Loden [:reed]]

Checking in bulk.cgi;
/cvsroot/mozilla/webtools/new-graph/bulk.cgi,v  <--  bulk.cgi
new revision: 1.2; previous revision: 1.1
done
Checking in collect.cgi;
/cvsroot/mozilla/webtools/new-graph/collect.cgi,v  <--  collect.cgi
new revision: 1.7; previous revision: 1.6
done
Checking in dgraph.html;
/cvsroot/mozilla/webtools/new-graph/dgraph.html,v  <--  dgraph.html
new revision: 1.3; previous revision: 1.2
done
RCS file: /cvsroot/mozilla/webtools/new-graph/dumpdata.cgi,v
done
Checking in dumpdata.cgi;
/cvsroot/mozilla/webtools/new-graph/dumpdata.cgi,v  <--  dumpdata.cgi
initial revision: 1.1
done
Checking in graph.html;
/cvsroot/mozilla/webtools/new-graph/graph.html,v  <--  graph.html
new revision: 1.11; previous revision: 1.10
done
Checking in js/GraphFormModule.js;
/cvsroot/mozilla/webtools/new-graph/js/GraphFormModule.js,v  <--  GraphFormModule.js
new revision: 1.5; previous revision: 1.4
done
Checking in js/TinderboxData.js;
/cvsroot/mozilla/webtools/new-graph/js/TinderboxData.js,v  <--  TinderboxData.js
new revision: 1.5; previous revision: 1.4
done
Checking in js/dGraphFormModule.js;
/cvsroot/mozilla/webtools/new-graph/js/dGraphFormModule.js,v  <--  dGraphFormModule.js
new revision: 1.3; previous revision: 1.2
done
Checking in js/graph.js;
/cvsroot/mozilla/webtools/new-graph/js/graph.js,v  <--  graph.js
new revision: 1.12; previous revision: 1.11
done
Checking in utils/import.py;
/cvsroot/mozilla/webtools/new-graph/utils/import.py,v  <--  import.py
new revision: 1.11; previous revision: 1.10
done