Closed Bug 173005 Opened 22 years ago Closed 21 years ago
Add bar charts, pie charts etc
. to reporting
We should use the Template Toolkit interface to GD to allow the generation of bar charts, pie charts etc. instead of tables for the reports. Gerv
OK, here it is. The patch that makes Bugzilla's reporting capabilities superior to any other bug tracker ;-) We have pie charts, bar charts and line graphs of arbitrary queries, in 1, 2 or 3 dimensions, with cumulated or individual data sets. You can jump from one sort of view to another, resize your charts, or tweak your parameters at any time. And it makes coffee too. :-) Also, I promise that this is the last time (for a while) that I'll rewrite the reporting system. I have other stuff to do... Brief guide to the patch: - search-report-*.html.tmpl are the two interfaces to define the report you want. We have a different one for HTML/CSV tables because the options are different. (I looked at uniting them more, but it wouldn't work.) The common select-generator has been split out into search-report-select.html.tmpl. - report.html.tmpl is a wrapper; it can either enclose a set (1 or more) of HTML tables (report-table.html.tmpl), or a set of PNG graphics, generated by calling back into reports.cgi. So, the "outer part" of the old report-table.html.tmpl has moved into report.html.tmpl. - *.png.tmpl generate the individual charts, in response to being called from <img> tags in report.html.tmpl. These output PNG data directly, using the Template Toolkit's bindings to GD::Graph. - The CSV system works like the HTML system, but via report.csv.tmpl and report-table.csv.tmpl. The only open issue is that I can't find a way to pass row_names to graph.set_legend() in the PNG files, because it requires a list, and row_names is a list reference, and I don't know if TT lets me at the underlying list. Ideas welcome. It'll be up on Landfill as soon as I can get GD::Graph installed there. Gerv
Severity: normal → enhancement
Status: NEW → ASSIGNED
Priority: -- → P2
Target Milestone: --- → Bugzilla 2.18
Let's rock... joel, are you up for reviewing this, given that you know this code better than anyone else except me? :-) Gerv
GD::Text::Align seems to be mandatory for GD::Graph to work, even though it does not automatically get it. Installing GD::Graph does not make checksetup happy by itself. If GD::Graph itself if installed, but GD::Text::Align is not, then runtests fails. I'm hacking through this, but it will be necessary to update checksetup and have some other people who have not already hacked through this try it out.
Comment on attachment 104259 [details] [diff] [review] Patch v.1 In addition to the checksetup issues, the graphs don;t seem to come out (tried on mozilla build 2002053012 and NS 6) even though the same site will happily generate the traditional graphical charts.
Attachment #104259 - Flags: review-
Dave: please could you add GD::Text::Align to your list for landfill? Joel: is the GD::Text::Align issue the reason you sent me the earlier mail? If the charts aren't coming out, then load some up, do "View Image" on the broken image, then hack the "plot" section of reports.cgi to print a content-type of text/plain (and then some random text for verification). That should allow you to read the error message, if there is any. Debugging this stuff is a right pain, I've found. :-( Gerv
Actually, unless we plan to make GD::Text::Align mandatory for ALL installations, we should wait until after we have confirmed that no dependencies on the optional modules break anything before updating landfill. (yes - that is the reason for the earlier email)
I tried that debug technique and get no output. This package is going to need some real debug hooks. Please put those in and then I'll look again. Whatever is going on will happen to users and developers in the future and needs a moderate-to-low-pain debug technique to be viable.
> Actually, unless we plan to make GD::Text::Align mandatory for ALL > installations, we should wait until after we have confirmed that no dependencies > on the optional modules break anything before updating landfill. I'm confused - what might "anything" be? I've asked Dave to install GD::Graph and GD::Text::Align on landfill - why might that be a problem? As for debug hooks, it's a bit difficult. GD::Graph's way of saying "something is wrong" is outputting a zero byte file (which isn't a valid PNG.) I've never come across a way of making it produce better errors. Once the PNG content-type has been printed, any template error will never display in the browser. :-| I'll see what I can do, but I think we just have to make sure it always works ;-) Gerv
OK, this has some debugging in it - add "debug=1" to the URL of an image which isn't displaying. But if all you get is 0 bytes of nothing, it just means that GD::Graph is moaning about the data it's being fed. Check your HTTPD error log; but I don't know of a way to get it to report better errors. Gerv
Attachment #104259 - Attachment is obsolete: true
Comment on attachment 104273 [details] [diff] [review] Patch v.2 1) Fails runtests.sh 2) Still fails with blank image and no message to either the browser or error_log. 3) If GD::Text::Align is supposed to be madatory for all sites, then checksetup should terminate immediately if i t is not found and not pretend that it is only required for sites using this charting. If not, it should not cause a fatal error in any file, even during runtests.sh My point is that the landfill (because of tindrboxes) should not permit the code that fails to compile in the basis of an optional module.
Attachment #104273 - Flags: review-
> 1) Fails runtests.sh Small bug (semicolon instead of comma); fixed in this patch. 2) Still fails with blank image and no message to either the browser or error_log. > 3) If GD::Text::Align is supposed to be madatory for all sites, then > checksetup should terminate immediately if i t is not found and not pretend > that it is only required for sites using this charting. If not, it should not > cause a fatal error in any file, even during runtests.sh Ah, now I see what you are going on about. I didn't, at first, grasp the significance of: > If GD::Graph itself if installed, but GD::Text::Align is not, then runtests > fails. How does it fail? > My point is that the landfill (because of tindrboxes) should not permit the > code that fails to compile in the basis of an optional module. Now that the bug in checksetup.pl (which had nothing to do with GD::Text::Align) is fixed, then both runtests and checksetup run fine without errors. So I'm not sure what you mean here; if you don't want graphical reports, don't install GD::Graph. If you do, you have to install both GD::Graph and GD::Text::Align. Gerv
Attachment #104273 - Attachment is obsolete: true
Just for the record, the tinderboxes test all of the code, even the optional stuff (but not contrib). The machines running the tinderboxes need to have all of the optional modules available for the tests to pass.
OK... I updated from GD-1.19 to GD-1.38 The GD build and install process was rather nightmarish. While GD-1.19 would compile, GD-1.20 and GD-2.02 would not and failed with unhelpful errors. GD-1.38 would build, though make test only passed a few of its tests. It did respond to the make install and then this works. I have been watching the level of a lot of the questions on the IRC and newsgroup and, as Linux gets easier to install, the barrier to non-hackers has been coming down. We need to get some other users to make sure they build newer GDs OK. If it was just a nightmare for me, that's cool.
Oh yeah, and by the way, this is way cool.
Comment on attachment 104289 [details] [diff] [review] Patch v.3 This is really cool, but ... Items identified from testing so far.... 1) Need carefull resolution to the GD version issue. 2) Bar charts horizontal axis is strange. It shows "1st" through "9th" instead of the 4 actual items that pie and line get when I put the product on the X axis. Also, the verical axis on bar charts has the wrong stuff.
Attachment #104289 - Flags: review-
> 1) Need carefull resolution to the GD version issue. The resolution is: "Require version 1.20 or above" :-) > Bar charts horizontal axis is strange. It shows "1st" through "9th" instead > of the 4 actual items that pie and line get when I put the product on the X > axis. I can't reproduce this. Please give exact steps to reproduce :-) > Also, the verical axis on bar charts has the wrong stuff. Dude, I'll need a better bug report than that :-) Gerv
Gerv, Using the machine you know about.... A bar chart has the problem http://machine:port/gervzilla/report.cgi?y_axis_field=bug_status&cumulate=0&z_axis_field=&format=bar&x_axis_field=product&short_desc_type=allwordssubstr&short_desc=&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&changedin=&chfieldfrom=&chfieldto=Now&chfieldvalue=&action=wrap&field0-0-0=&type0-0-0=noop&value0-0-0= Clicking on lines, it becomes fine http://machine:port/gervzilla/report.cgi?report.cgi?bug_file_loc_type=allwordssubstr&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bugidtype=include&chfieldto=Now&emailassigned_to1=1&emailassigned_to2=1&emailcc2=1&emailreporter2=1&emailtype1=substring&emailtype2=substring&long_desc_type=allwordssubstr&short_desc_type=allwordssubstr&type0-0-0=noop&x_axis_field=product&y_axis_field=bug_status&width=600&height=350&action=wrap&format=line Editing that query and resubmitting, it stays fine http://machine:port/gervzilla/report.cgi?y_axis_field=bug_status&cumulate=0&z_axis_field=&format=line&x_axis_field=product&short_desc_type=allwordssubstr&short_desc=&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&changedin=&chfieldfrom=&chfieldto=Now&chfieldvalue=&action=wrap&field0-0-0=&type0-0-0=noop&value0-0-0= Also, under the chart, it lets the user switch between table and line, but not pie.
> Also, under the chart, it lets the user switch between table and line, but not > pie. That's intentional; pie charts don't make any sense if you have a Y axis. Allowing the user to click on Pie would merely produce confusing data. Hack the template report.html.tmpl if you want to see what happens. I'll look into the other issue. Gerv
> Bar charts horizontal axis is strange. It shows "1st" through "9th" instead > of the 4 actual items that pie and line get when I put the product on the X > axis. This was remaining debugging code from when I was trying to debug the GD problem. I've now removed it, and things appear to be running sweetly :-) In any case, could you please review the code (as opposed to trying it out)? It would be a pain if we took time to get it working, then you asked for architectural changes so we had to take time to get it working again :-) Thanks, Gerv
The patch looks architecturally sound. I am still trying to decide if I want to nit about measuring too many things in pixels as I keep hoping for a 9000DPI display for my birthday. The only issue I keep coming back to is the level of debuggability and I wonder if it would be possible to process the GD output into a perl scalar and check it before sending anything to the browser, keeping options open on a sensible error message. I prefer doing the final review on a final working patch, so I'll wait for that. I always read and test and test and read. That's how I tell what needs to be tested.
Gerv: Is it necesaary to require logins to plot? -quietly_check_login(); +confirm_login();
Latest all-singing, all-dancing patch. Hopefully, it works well enough to review. > Gerv: Is it necesaary to require logins to plot? I'd say so. It's not like querying, which anyone needs to do - and the page is quite resource-intensive, and so open to the possibility of being DOSed. If anyone is abusing it, we can then track them down and ask them to stop. Gerv
Attachment #104289 - Attachment is obsolete: true
Well, its going to cause problems if the <img> request happens on an un-authenticated connection. Its no more resource intensive than a sufficeiently complicated buglist.cgi query, is it?
> Well, its going to cause problems if the <img> request happens on an > un-authenticated connection. Do cookies not get sent on img requests? Or are you talking about the fact that we theoretically support cookie-less operation? Also, if people are casually stopped from doing this (I think there will be a temptation for random people to "play", thereby using up resources), the fact that it's not completely secure is not a big deal. > Its no more resource intensive than a sufficeiently complicated buglist.cgi > query, is it? Well, putting together a sufficiently-complicated buglist.cgi query takes effort, and then you get a "pondering" screen. In this case, you could ask for 20 complicated queries at once without really realising it. Gerv
Comment on attachment 104356 [details] [diff] [review] Patch v.1 2xr=joel There are a few things that could be polished about this, but they are all in the templates and can be easily customized.
Status: ASSIGNED → RESOLVED
Closed: 21 years ago
Resolution: --- → FIXED
Woohoo! Fixed. RCS file: /cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-report-graph.html.tmpl,v done Checking in template/en/default/search/search-report-graph.html.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-report-graph.html.tmpl,v <-- search-report-graph.html.tmpl initial revision: 1.1 done RCS file: /cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-report-select.html.tmpl,v done Checking in template/en/default/search/search-report-select.html.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-report-select.html.tmpl,v <-- search-report-select.html.tmpl initial revision: 1.1 done Checking in template/en/default/search/search-report-table.html.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-report-table.html.tmpl,v <-- search-report-table.html.tmpl new revision: 1.3; previous revision: 1.2 done Checking in report.cgi; /cvsroot/mozilla/webtools/bugzilla/report.cgi,v <-- report.cgi new revision: 1.5; previous revision: 1.4 done Checking in checksetup.pl; /cvsroot/mozilla/webtools/bugzilla/checksetup.pl,v <-- checksetup.pl new revision: 1.199; previous revision: 1.198 done Checking in query.cgi; /cvsroot/mozilla/webtools/bugzilla/query.cgi,v <-- query.cgi new revision: 1.111; previous revision: 1.110 done RCS file: /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-pie.png.tmpl,v done Checking in template/en/default/reports/report-pie.png.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-pie.png.tmpl,v <-- report-pie.png.tmpl initial revision: 1.1 done RCS file: /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report.html.tmpl,v done Checking in template/en/default/reports/report.html.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report.html.tmpl,v <-- report.html.tmpl initial revision: 1.1 done RCS file: /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-bar.png.tmpl,v done Checking in template/en/default/reports/report-bar.png.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-bar.png.tmpl,v <-- report-bar.png.tmpl initial revision: 1.1 done RCS file: /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report.csv.tmpl,v done Checking in template/en/default/reports/report.csv.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report.csv.tmpl,v <-- report.csv.tmpl initial revision: 1.1 done Checking in template/en/default/reports/report-table.csv.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-table.csv.tmpl,v <-- report-table.csv.tmpl new revision: 1.2; previous revision: 1.1 done Checking in template/en/default/reports/report-table.html.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-table.html.tmpl,v <-- report-table.html.tmpl new revision: 1.4; previous revision: 1.3 done RCS file: /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-line.png.tmpl,v done Checking in template/en/default/reports/report-line.png.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/report-line.png.tmpl,v <-- report-line.png.tmpl initial revision: 1.1 done Checking in template/en/default/reports/menu.html.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/reports/menu.html.tmpl,v <-- menu.html.tmpl new revision: 1.2; previous revision: 1.1 done Checking in template/en/default/global/field-descs.html.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/global/field-descs.html.tmpl,v <-- field-descs.html.tmpl new revision: 1.4; previous revision: 1.3 done Checking in template/en/default/global/user-error.html.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/global/user-error.html.tmpl,v <-- user-error.html.tmpl new revision: 1.20; previous revision: 1.19 done Checking in template/en/default/global/code-error.html.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/global/code-error.html.tmpl,v <-- code-error.html.tmpl new revision: 1.21; previous revision: 1.20 done Checking in template/en/default/index.html.tmpl; /cvsroot/mozilla/webtools/bugzilla/template/en/default/index.html.tmpl,v <-- index.html.tmpl new revision: 1.8; previous revision: 1.7 done Gerv
Instructions on GD and libgd upgrade ====================================== Maybe you will find these instructions usefull (at least dummies like me, will :) ) I had same problems upgrading libgd and and GD. After reading several times README file, I found simple way how to perform this upgrade: 1. Download libgd and GD with _same_ versions and extract them somewhere 2. Patch libg with patch file included in GD (perl patch_gd.pl) 3. ./configure in libgd ; make install in libgd 4. make install in GD At that point at last everything was fine :)
QA Contact: matty_is_a_geek → default-qa
You need to log in before you can comment on or make changes to this bug.