173005 - Add bar charts, pie charts etc. to reporting

Status: RESOLVED FIXED

(Bugzilla :: Reporting/Charting, enhancement, P2)

Description

[Gervase Markham [:gerv]]

We should use the Template Toolkit interface to GD to allow the generation of
bar charts, pie charts etc. instead of tables for the reports.

Gerv

Comment 1

[Gervase Markham [:gerv]]

OK, here it is. The patch that makes Bugzilla's reporting capabilities superior
to any other bug tracker ;-) We have pie charts, bar charts and line graphs of
arbitrary queries, in 1, 2 or 3 dimensions, with cumulated or individual data
sets. You can jump from one sort of view to another, resize your charts, or
tweak your parameters at any time. And it makes coffee too. :-)

Also, I promise that this is the last time (for a while) that I'll rewrite the
reporting system. I have other stuff to do...

Brief guide to the patch:

- search-report-*.html.tmpl are the two interfaces to define the report you
want. We have a different one for HTML/CSV tables because the options are
different. (I looked at uniting them more, but it wouldn't work.) The common
select-generator has been split out into search-report-select.html.tmpl.

- report.html.tmpl is a wrapper; it can either enclose a set (1 or more) of HTML
tables (report-table.html.tmpl), or a set of PNG graphics, generated by calling
back into reports.cgi. So, the "outer part" of the old report-table.html.tmpl
has moved into report.html.tmpl.

- *.png.tmpl generate the individual charts, in response to being called from
<img> tags in report.html.tmpl. These output PNG data directly, using the
Template Toolkit's bindings to GD::Graph.

- The CSV system works like the HTML system, but via report.csv.tmpl and
report-table.csv.tmpl.

The only open issue is that I can't find a way to pass row_names to
graph.set_legend() in the PNG files, because it requires a list, and row_names
is a list reference, and I don't know if TT lets me at the underlying list.
Ideas welcome.

It'll be up on Landfill as soon as I can get GD::Graph installed there.

Gerv

Comment 2

[Gervase Markham [:gerv]]

Attachment: Patch v.1

Let's rock... joel, are you up for reviewing this, given that you know this
code better than anyone else except me? :-)

Gerv

Comment 3

[Joel Peshkin]

GD::Text::Align seems to be mandatory for GD::Graph to work, even though it does
not automatically get it.  Installing GD::Graph does not make checksetup happy
by itself.

If GD::Graph itself if installed, but GD::Text::Align is not, then runtests fails.

I'm hacking through this, but it will be necessary to update checksetup and have
some other people who have not already hacked through this try it out.

Comment 4

[Joel Peshkin]

Comment on attachment 104259 [details] [diff] [review]
Patch v.1

In addition to the checksetup issues,  the graphs don;t seem to come out (tried
on mozilla build 2002053012 and NS 6) even though the same site will happily
generate the traditional graphical charts.

Comment 5

[Gervase Markham [:gerv]]

Dave: please could you add GD::Text::Align to your list for landfill?

Joel: is the GD::Text::Align issue the reason you sent me the earlier mail?

If the charts aren't coming out, then load some up, do "View Image" on the
broken image, then hack the "plot" section of reports.cgi to print a
content-type of text/plain (and then some random text for verification). That
should allow you to read the error message, if there is any.

Debugging this stuff is a right pain, I've found. :-(

Gerv

Comment 6

[Joel Peshkin]

Actually, unless we plan to make GD::Text::Align mandatory for ALL
installations, we should wait until after we have confirmed that no dependencies
on the optional modules break anything before updating landfill.  (yes - that is
the reason for the earlier email)

Comment 7

[Joel Peshkin]

I tried that debug technique and get no output.   This package is going to need
some real debug hooks.  Please put those in and then I'll look again.  Whatever
is going on will happen to users and developers in the future and needs a
moderate-to-low-pain debug technique to be viable.

Comment 8

[Gervase Markham [:gerv]]

> Actually, unless we plan to make GD::Text::Align mandatory for ALL
> installations, we should wait until after we have confirmed that no dependencies
> on the optional modules break anything before updating landfill.

I'm confused - what might "anything" be? I've asked Dave to install GD::Graph
and GD::Text::Align on landfill - why might that be a problem?

As for debug hooks, it's a bit difficult. GD::Graph's way of saying "something
is wrong" is outputting a zero byte file (which isn't a valid PNG.) I've never
come across a way of making it produce better errors.

Once the PNG content-type has been printed, any template error will never
display in the browser. :-| I'll see what I can do, but I think we just have to
make sure it always works ;-)

Gerv

Comment 9

[Gervase Markham [:gerv]]

Attachment: Patch v.2

OK, this has some debugging in it - add "debug=1" to the URL of an image which
isn't displaying. But if all you get is 0 bytes of nothing, it just means that
GD::Graph is moaning about the data it's being fed. Check your HTTPD error log;
but I don't know of a way to get it to report better errors.

Gerv

Comment 10

[Joel Peshkin]

Comment on attachment 104273 [details] [diff] [review]
Patch v.2

1) Fails runtests.sh  
2) Still fails with blank image and no message to either the browser or
error_log.
3) If GD::Text::Align is supposed to be madatory for all sites, then checksetup
should terminate immediately if i t is not found and not pretend that it is
only required for sites using this charting.  If not, it should not cause a
fatal error in any file, even during runtests.sh

My point is that the landfill (because of tindrboxes) should not permit the
code that fails  to compile in the basis of an optional module.

Comment 11

[Gervase Markham [:gerv]]

Attachment: Patch v.3

> 1) Fails runtests.sh	

Small bug (semicolon instead of comma); fixed in this patch.

2) Still fails with blank image and no message to either the browser or
error_log.
> 3) If GD::Text::Align is supposed to be madatory for all sites, then 
> checksetup should terminate immediately if i t is not found and not pretend 
> that it is only required for sites using this charting.  If not, it should
not
> cause a fatal error in any file, even during runtests.sh

Ah, now I see what you are going on about. I didn't, at first, grasp the
significance of:
> If GD::Graph itself if installed, but GD::Text::Align is not, then runtests 
> fails.

How does it fail?

> My point is that the landfill (because of tindrboxes) should not permit the
> code that fails  to compile in the basis of an optional module.

Now that the bug in checksetup.pl (which had nothing to do with
GD::Text::Align) is fixed, then both runtests and checksetup run fine without
errors. So I'm not sure what you mean here; if you don't want graphical
reports, don't install GD::Graph. If you do, you have to install both GD::Graph
and GD::Text::Align.

Gerv

Comment 12

[Dave Miller [:justdave]]

Just for the record, the tinderboxes test all of the code, even the optional
stuff (but not contrib).  The machines running the tinderboxes need to have all
of the optional modules available for the tests to pass.

Comment 13

[Joel Peshkin]

OK... I updated from GD-1.19 to GD-1.38
The GD build and install process was rather nightmarish.  While GD-1.19 would
compile, GD-1.20 and GD-2.02 would not and failed with unhelpful errors. 
GD-1.38 would build, though make test only passed a few of its tests.  It did
respond to the make install and then this works.

I have been watching the level of a lot of the questions on the IRC and
newsgroup and, as Linux gets easier to install, the barrier to non-hackers has
been coming down.  We need to get some other users to make sure they build newer
GDs OK.  If it was just a nightmare for me, that's cool.

Comment 14

[Joel Peshkin]

Oh yeah, and by the way, this is way cool.

Comment 15

[Joel Peshkin]

Comment on attachment 104289 [details] [diff] [review]
Patch v.3

This is really cool, but ...

Items identified from testing so far....
1) Need carefull resolution to the GD version issue.
2) Bar charts horizontal axis is strange.  It shows "1st" through "9th" instead
of the 4 actual items that pie and line get when I put the product on the X
axis. Also, the verical axis on bar charts has the wrong stuff.

Comment 16

[Gervase Markham [:gerv]]

> 1) Need carefull resolution to the GD version issue.

The resolution is: "Require version 1.20 or above" :-)

> Bar charts horizontal axis is strange.  It shows "1st" through "9th" instead
> of the 4 actual items that pie and line get when I put the product on the X
> axis.

I can't reproduce this. Please give exact steps to reproduce :-)

> Also, the verical axis on bar charts has the wrong stuff.

Dude, I'll need a better bug report than that :-)

Gerv

Comment 17

[Joel Peshkin]

Gerv,

   Using the machine you know about....

A bar chart has the problem
http://machine:port/gervzilla/report.cgi?y_axis_field=bug_status&cumulate=0&z_axis_field=&format=bar&x_axis_field=product&short_desc_type=allwordssubstr&short_desc=&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&changedin=&chfieldfrom=&chfieldto=Now&chfieldvalue=&action=wrap&field0-0-0=&type0-0-0=noop&value0-0-0=
Clicking on lines, it becomes fine
http://machine:port/gervzilla/report.cgi?report.cgi?bug_file_loc_type=allwordssubstr&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bugidtype=include&chfieldto=Now&emailassigned_to1=1&emailassigned_to2=1&emailcc2=1&emailreporter2=1&emailtype1=substring&emailtype2=substring&long_desc_type=allwordssubstr&short_desc_type=allwordssubstr&type0-0-0=noop&x_axis_field=product&y_axis_field=bug_status&width=600&height=350&action=wrap&format=line
Editing that query and resubmitting, it stays fine
http://machine:port/gervzilla/report.cgi?y_axis_field=bug_status&cumulate=0&z_axis_field=&format=line&x_axis_field=product&short_desc_type=allwordssubstr&short_desc=&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&changedin=&chfieldfrom=&chfieldto=Now&chfieldvalue=&action=wrap&field0-0-0=&type0-0-0=noop&value0-0-0=

  Also, under the chart, it lets the user switch between table and line, but not
pie.

Comment 18

[Gervase Markham [:gerv]]

> Also, under the chart, it lets the user switch between table and line, but not
> pie.

That's intentional; pie charts don't make any sense if you have a Y axis.
Allowing the user to click on Pie would merely produce confusing data.

Hack the template report.html.tmpl if you want to see what happens.

I'll look into the other issue.

Gerv

Comment 19

[Gervase Markham [:gerv]]

> Bar charts horizontal axis is strange.  It shows "1st" through "9th" instead
> of the 4 actual items that pie and line get when I put the product on the X
> axis.

This was remaining debugging code from when I was trying to debug the GD
problem. I've now removed it, and things appear to be running sweetly :-)

In any case, could you please review the code (as opposed to trying it out)? It
would be a pain if we took time to get it working, then you asked for
architectural changes so we had to take time to get it working again :-)

Thanks,

Gerv

Comment 20

[Joel Peshkin]

The patch looks architecturally sound.  I am still trying to decide if I want to
nit about measuring too many things in pixels as I keep hoping for a 9000DPI
display for my birthday.  The only issue I keep coming back to is the level of
debuggability and I wonder if it would be possible to process the GD output into
a perl scalar and check it before sending anything to the browser, keeping
options open on a sensible error message.

I prefer doing the final review on a final working patch, so I'll wait for that.
 I always read and test and test and read.  That's how I tell what needs to be
tested.

Comment 21

[Joel Peshkin]

Gerv: Is it necesaary to require logins to plot?

-quietly_check_login();
+confirm_login();

Comment 22

[Gervase Markham [:gerv]]

Attachment: Patch v.1

Latest all-singing, all-dancing patch. Hopefully, it works well enough to
review.

> Gerv: Is it necesaary to require logins to plot?

I'd say so. It's not like querying, which anyone needs to do - and the page is
quite resource-intensive, and so open to the possibility of being DOSed. If
anyone is abusing it, we can then track them down and ask them to stop.

Gerv

Comment 23

[Bradley Baetz (:bbaetz)]

Well, its going to cause problems if the <img> request happens on an
un-authenticated connection.

Its no more resource intensive than a sufficeiently complicated buglist.cgi
query, is it?

Comment 24

[Gervase Markham [:gerv]]

> Well, its going to cause problems if the <img> request happens on an
> un-authenticated connection.

Do cookies not get sent on img requests? Or are you talking about the fact that
we theoretically support cookie-less operation?

Also, if people are casually stopped from doing this (I think there will be a
temptation for random people to "play", thereby using up resources), the fact
that it's not completely secure is not a big deal.

> Its no more resource intensive than a sufficeiently complicated buglist.cgi
> query, is it?

Well, putting together a sufficiently-complicated buglist.cgi query takes
effort, and then you get a "pondering" screen. In this case, you could ask for
20 complicated queries at once without really realising it. 

Gerv

Comment 25

[Joel Peshkin]

Comment on attachment 104356 [details] [diff] [review]
Patch v.1

2xr=joel
There are a few things that could be polished about this, but they are all in
the templates and can be easily customized.

Comment 26

[Gervase Markham [:gerv]]

Woohoo! Fixed.

RCS file:
/cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-report-graph.html.tmpl,v
done
Checking in template/en/default/search/search-report-graph.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-report-graph.html.tmpl,v
 <--  search-report-graph.html.tmpl
initial revision: 1.1
done
RCS file:
/cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-report-select.html.tmpl,v
done
Checking in template/en/default/search/search-report-select.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-report-select.html.tmpl,v
 <--  search-report-select.html.tmpl
initial revision: 1.1
done
Checking in template/en/default/search/search-report-table.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-report-table.html.tmpl,v
 <--  search-report-table.html.tmpl
new revision: 1.3; previous revision: 1.2
done
Checking in report.cgi;
/cvsroot/mozilla/webtools/bugzilla/report.cgi,v  <--  report.cgi
new revision: 1.5; previous revision: 1.4
done
Checking in checksetup.pl;
/cvsroot/mozilla/webtools/bugzilla/checksetup.pl,v  <--  checksetup.pl
new revision: 1.199; previous revision: 1.198
done
Checking in query.cgi;
/cvsroot/mozilla/webtools/bugzilla/query.cgi,v  <--  query.cgi
new revision: 1.111; previous revision: 1.110
done
RCS file:
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-pie.png.tmpl,v
done
Checking in template/en/default/reports/report-pie.png.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-pie.png.tmpl,v
 <--  report-pie.png.tmpl
initial revision: 1.1
done
RCS file:
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report.html.tmpl,v
done
Checking in template/en/default/reports/report.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report.html.tmpl,v
 <--  report.html.tmpl
initial revision: 1.1
done
RCS file:
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-bar.png.tmpl,v
done
Checking in template/en/default/reports/report-bar.png.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-bar.png.tmpl,v
 <--  report-bar.png.tmpl
initial revision: 1.1
done
RCS file:
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report.csv.tmpl,v
done
Checking in template/en/default/reports/report.csv.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report.csv.tmpl,v
 <--  report.csv.tmpl
initial revision: 1.1
done
Checking in template/en/default/reports/report-table.csv.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-table.csv.tmpl,v
 <--  report-table.csv.tmpl
new revision: 1.2; previous revision: 1.1
done
Checking in template/en/default/reports/report-table.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-table.html.tmpl,v
 <--  report-table.html.tmpl
new revision: 1.4; previous revision: 1.3
done
RCS file:
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-line.png.tmpl,v
done
Checking in template/en/default/reports/report-line.png.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-line.png.tmpl,v
 <--  report-line.png.tmpl
initial revision: 1.1
done
Checking in template/en/default/reports/menu.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/menu.html.tmpl,v
 <--  menu.html.tmpl
new revision: 1.2; previous revision: 1.1
done
Checking in template/en/default/global/field-descs.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/field-descs.html.tmpl,v
 <--  field-descs.html.tmpl
new revision: 1.4; previous revision: 1.3
done
Checking in template/en/default/global/user-error.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/user-error.html.tmpl,v
 <--  user-error.html.tmpl
new revision: 1.20; previous revision: 1.19
done
Checking in template/en/default/global/code-error.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/global/code-error.html.tmpl,v
 <--  code-error.html.tmpl
new revision: 1.21; previous revision: 1.20
done
Checking in template/en/default/index.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/index.html.tmpl,v  <-- 
index.html.tmpl
new revision: 1.8; previous revision: 1.7
done

Gerv

Comment 27

[Toms Baugis]

Instructions on GD and libgd upgrade
======================================
Maybe you will find these instructions usefull (at least dummies like me, will :)  )
I had same problems upgrading libgd and and GD.
After reading several times README file, I found simple way how to perform this
upgrade:

1. Download libgd and GD with _same_ versions and extract them somewhere
2. Patch libg with patch file included in GD (perl patch_gd.pl)
3. ./configure in libgd ; make install in libgd
4. make install in GD

At that point at last everything was fine :)