599582 - revive production of blocker stats graph

Status: RESOLVED FIXED

(Websites :: Other, defect)

Description

[chris hofmann]

murali produced these kind of graphs to help track progress towards shipping and keep people focused on minimizing nominations and killing blockers.

http://people.mozilla.com/~chofmann/project-metrics/blocker-dashboard-20091208.png

we should revive these graphs to help track progress towards shipping 4.0

I think these are the two queries to plug in:

https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocking2.0%3Afinal,beta

https://bugzilla.mozilla.org/buglist.cgi?quicksearch=OPEN%20blocking2.0%3A%22%3F%22

Comment 1

[chris hofmann]

Attachment: 4.0 graph to date

it would also be cool to add a breakdown of the number of bugs that "block any beta" and "block next beta" since we have been tracking those numbers too over the summer.

here is the .csv alues to pre-populate, based on data from the weekly platform meetings

date    ,tl blockers,open noms,blocking any beta,blocking next beta
20100720,331,258,100,32
20100727,385,500,123,35
20100803,462,462,204,34
20100810,559,275,218,56
20100817,571,290,237,94
20100824,660,105,388,105
20100831,692,146,399,120
20100907,709,217,406,114
20100914,739,246,329,73,
20100921,758,281,410,51,
20100927,823,184,460,29,

getting more eyes on these trends should help get the numbers moving in the right direction.

Comment 2

[chris hofmann]

blocking any beta query is

https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocking2.0%3Abeta

blocking next beta will have to be updated every few weeks but currently is

https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocking2.0%3Abeta7

Comment 3

[chris hofmann]

one more ask for the graph, or graphs (depending on how you might want to do it) would be to add tracking blockers that are also regressions.  query for that is

https://bugzilla.mozilla.org/buglist.cgi?quicksearch=blocking2.0%3Afinal,beta&keywords=regression

Comment 4

[chris hofmann]

I'll try and take a snapshot of the numbers once a day and keep this csv file up-to-date until we can get it integrated into a graph thats more automated and pretty.

http://people.mozilla.com/~chofmann/project-metrics/40-blocker-and-nomination-counts.csv

Comment 5

[Mark Côté [:mcote]]

Here's an explanation of Murali's system, for posterity.

The system is divided into a number of scripts, which can generally be divided into two categories: "dumper" scripts that get bug data from Bugzilla according to a given query and insert this data into a local MySQL database, and "metrics" scripts which generate static HTML pages using the data in the local db.  These scripts are driven by a shell script, which also uploads the generated pages via scp to a web server (e.g. a people.mozilla.org account).

There are quite a few versions and varieties of these scripts.  It appears that the old ones used a SQLite database instead of a MySQL database (as seen in the OLD directory).  There are varieties of the dumper/metrics scripts to obtain information about AMO (add-ons) bugs, SUMO (support) bugs, bugs flagged for review, blockers, and bugs nominated for blockers.  There also appears to be an effort to make at least some of these (fairly redundant) files generic to some extent: generic-dumper.pl and genericbugmetrics.pl, confusingly stored in the SUMO directory.  These generic scripts obtain the output filename, page title, and search query from a file stored on the "fs" server retrieved via HTTP.  The output appears to be similar to that generated by the AMO and SUMO scripts (but *not* the same as the blocker scripts).

The entire process is driven by a shell script called "batchjob.sh".  Presumably it was run once or twice a day.  Obtaining all the bug information is a time-consuming process, since the XML for each bug is retrieved one-by-one.  Generating the HTML pages is extremely fast, however.

The database-storage step seems very strange at first; it appears to be completely unnecessary and only a result of the complete separation of the dumper and metrics scripts.  However, there is at least one script (timelinemetrics.pl) that gives results over time, thus necessitating the storage of the results of previous queries.

The generation and uploading of static HTML pages is also somewhat strange.  The usual solution would be an inversion of the process: placing a slightly modified version of the metrics scripts on the web server as CGI scripts to generate pages on the fly, as requested.  However, given that the data is only refreshed periodically, perhaps there is nothing to be gained in this situation by inverting the process.

It's likely that direct (read-only) access to the Bugzilla database would speed up the dumper scripts immensely.  It is unclear whether this would be fast enough to allow completely dynamic, up-to-date reports; however, certainly the frequency of page generation could be increased.

There are also a couple obvious things that can be done to improve the scripts:

- Move DB data (database name, credentials) to a config file.
- Move queries from nongeneric scripts to a config file.
- Eliminate the redundancies in the scripts.  Even if multiple scripts are needed for special purposes (e.g. blocker reports vs. AMO/SUMO reports), much of the code could be put into modules for reuse.

Comment 6

[chris hofmann]

If we plan to rework this rather than use the old stuff here are some thoughts.

I think the "dumper" stuff may have used to track all kinds of metrics so it might have needed extra complexity to support a variety of data gathering tasks.

For this particular thing we want to track we can get much more simple.

http://people.mozilla.com/~chofmann/project-metrics/40-blocker-and-nomination-counts.csv is an example of the data we need dumped off a daily cron job.  plotting tools could work off a .csv rather than database.


Its a pretty simple process to get at that set of data.  I'm  doing a wget and scrapping content from bugzilla web queries, but I'm sure there better ways using the other bugzilla api's.   the queries to plug-in are listed above.

Comment 7

[Mark Côté [:mcote]]

Ah okay.  Yeah, the dumper scripts gather a bunch of info for every bug--priority, severity, etc.--to generate several graphs. If you think that the important part is just the number of bugs matching every query, then, yes, the scripts could be quite a bit simpler (and quicker).

I can create new scripts based on Murali's metrics' scripts to display this data in the same kind of graph, if that is what you are looking for.

Comment 8

[chris hofmann]

yeah, that would solve the immediate need of trying to give more attention to the blocker counts and a feeedback loop to developers on how we are making progress to shipping 4.0

Comment 9

[Mark Côté [:mcote]]

Attachment: Original-style graph with new data

I wrote a small script to generate graphs in the style of the original system.  Attached is a graph with the latest CSV data.  Note that we don't have quite as much information available--specifically, we can't tell how many blockers/nominations were added and removed, only the net difference.  We also can't link to the full bug list as displayed (although we could link to the general query, but that would be current info, not as displayed).

Anyway if this looks good to you, I will update the script to incorporate the other queries.  It might be too confusing to have them all on one graph, but we can split them up or something.  Or if you like, we can use a line graph instead of a bar graph, as in your screenshot.

Comment 10

[chris hofmann]

(In reply to comment #9)
> Created attachment 482950 [details]
> Original-style graph with new data
> 
> I wrote a small script to generate graphs in the style of the original system. 

looks good.

> Attached is a graph with the latest CSV data.  Note that we don't have quite as
> much information available--specifically, we can't tell how many
> blockers/nominations were added and removed, only the net difference. 

I think we can get by with just showing the net numbers.

I'm generating updates to the csv by running a script by hand once or twice a day to get the snapshot since people doesn't have cron.

I wonder if we can put this somewhere where it can run off cron, or do you have a plan for generating the .csv?

> We also
> can't link to the full bug list as displayed (although we could link to the
> general query, but that would be current info, not as displayed).

Maybe we could just have a static list of "interesting blocker queries" on the page below the chart with the set of queries shown in this bug.

> 
> Anyway if this looks good to you, I will update the script to incorporate the
> other queries.  It might be too confusing to have them all on one graph, but we
> can split them up or something.

I'm ok either way,  the most important numbers for the project are the overall numbers for blockers and nominations.   The others are just more details for a smaller set of people that want to digg into details about the breakdown of those two numbers.

>  Or if you like, we can use a line graph
> instead of a bar graph, as in your screenshot.

Your chart looks pretty good to me, and a lot like muralli produced which worked well in the past.

Comment 11

[Mark Côté [:mcote]]

(In reply to comment #10)
> I'm generating updates to the csv by running a script by hand once or twice a
> day to get the snapshot since people doesn't have cron.
> 
> I wonder if we can put this somewhere where it can run off cron, or do you have
> a plan for generating the .csv?

We're getting space on a server set up right now.  If you feel like attaching your CSV-generating script, I can set that up as a cron job on this machine.  No point in writing a new one if yours does the job! :)

> Maybe we could just have a static list of "interesting blocker queries" on the
> page below the chart with the set of queries shown in this bug.

Sounds good.  I can see about making a tighter link between the CSV-generator and the HTML view with a central configuration point so that it will be easy to add/update queries in the future.

Comment 12

[chris hofmann]

Attachment: hack to get interesting blocker bug metrics out of bugzilla

don't know that this should be considered "production quality" but its worked pretty reliably to get good numbers for the last few weeks.

its pretty dependent on the html output produced by bugzilla so any changes there might break it.   probably wouldn't take to much work to convert this over to the XML output or bugzilla rest API but I haven't had a chance.

feel free to use all or any parts of this

Comment 13

[chris hofmann]

Attachment: script that updates a .csv of blocker counts

Comment 14

[Mark Côté [:mcote]]

Okay I have a first draft of the script ready.  It's not quite done, but it should be indicative of the final look, at least (I have more under-the-hood changes to make; see below).

I have set it up on brasstacks; however, I am waiting for IT to set up a hostname for me so I can serve it as a virtual host.  It should eventually be accessible as blocker-reports.brasstacks.mozilla.com.  You can actually view it right now if you can fake it, e.g. by adding that hostname to your local hosts file with brasstacks's IP.

I split the data into two graphs, one with blockers, nominations, and blocker regressions, and the other with any beta blockers (heh) and next beta blockers.  The net changes are combined into one table at the bottom.

Question: some of this data is presumably a subset of others (next beta vs any beta, blockers vs. blocker regressions).  Do you want me to reflect this somehow (e.g. by subtracting the number of blocker regressions from the total number of blockers), or is it good as is?

There is a cron job running at 9 AM PST that runs your shell scripts to grab the data and my script to generate the HTML page.  I will probably rewrite your scripts shortly, just for two reasons: as previously mentioned, I'd like to just have one config file which specifies the queries (and ideally the entire layout--which queries are in which tables) to make it easier to change this stuff in the future, and to use the REST API, which should be more stable than scraping the HTML (though I am very impressed with your awk skills :)

Comment 15

[chris hofmann]

(In reply to comment #14)
> Okay I have a first draft of the script ready.  It's not quite done, but it
> should be indicative of the final look, at least (I have more under-the-hood
> changes to make; see below).
> 
> I have set it up on brasstacks; however, I am waiting for IT to set up a
> hostname for me so I can serve it as a virtual host.  It should eventually be
> accessible as blocker-reports.brasstacks.mozilla.com.  You can actually view it
> right now if you can fake it, e.g. by adding that hostname to your local hosts
> file with brasstacks's IP.
> 
> I split the data into two graphs, one with blockers, nominations, and blocker
> regressions, and the other with any beta blockers (heh) and next beta blockers.
>  The net changes are combined into one table at the bottom.
> 
> Question: some of this data is presumably a subset of others (next beta vs any
> beta, blockers vs. blocker regressions).  Do you want me to reflect this
> somehow (e.g. by subtracting the number of blocker regressions from the total
> number of blockers), or is it good as is?

nominations are bugs awaiting the module owner to have a look and determine blocker status.  some nonminations will become blockers, other will be dupped or otherwise 'minused' and not be taken on for a beta or final.  historically about 30-60% of nominations actually become blockers depending on the quality of the bugs filed and the size of the list.  the larger the list the 'noisier' it gets with more cruft that needs to get weeded out, but it still represents a pile of work to be done to get a more clear understanding of where we are in the schedule.

blocking next beta+, blocking betaN+, and blocking final+ are a subset of total blockers.

blocking regressions are just a different view of a class of bug within the set of blockers.   the blocking list may also contain "feature bugs" or bugs on new features that haven't appeared in previous releases.   regression bugs are most interesting of these since they help us to not go backwards in quality from previous releases.   its easier to throw non-regressions overboard if the schedule starts to lag.  regressions might be "targeted" to be fixed across next beta, betaN, and final.

Comment 16

[Mark Côté [:mcote]]

Thanks for the clarifications.  However I'm still wondering if you want all these numbers displayed separately.  For instance, the CSV file might indicate 832 total open blockers and 229 regressions.  So I'm wondering if the graph that displays open blockers and regressions should display 832 and 229 or 603 (832-229) and 229.

However, the more I think about it, if we're also mixing in open nominations, then we may as well display the total numbers (e.g. 832 and 229 in the above example), since nominations aren't a subset of blockers.  So I'll leave it as is for now.

I am basically finished the little app now.  Everything is defined in one config file.  I will attached an example.  So in the future, updating and creating new queries, and even creating new graphs, should be much easier and involve no code changes.

Comment 17

[Mark Côté [:mcote]]

Attachment: Sample configuration file for blocker-report graph generation

Comment 18

[chris hofmann]

showing nominations stacked on top of total blockers is the main and most important graph.   it might also be good to link to the .csv file with all the numbers. that would allow people to play with interesting variations of the data.

Comment 19

[chris hofmann]

the other issue that murali dealt with is around security closed bugs.  I think he was able to apply permissions to the quries to get security closed bugs in the query results.  that opens up a possible security hole.  I think it probably good enough to just put a disclaimer on the chart that says"

   "graphs exclude security bugs"

Comment 20

[Mark Côté [:mcote]]

Attachment: initial patch--Python files, default configuration, and batch-job shell script

Here's the first version of the app.  It's a couple of python files.  Usage is pretty simple:

./bugtrends.py -u > index.html

The "-u" option grabs the latest bugzilla data according to the queries in the configuration file, bugtrends.conf, which should be in the same directory.  Without -u, it generates the page based on the data currently in bugtrends.csv.  With the "-u" option and no CSV file, one will be created (note that bugtrends.py won't work without the -u option if there is no CSV file, since there's no data to plot!).

The batchjob.sh is just the couple commands that are run via cron once daily on brasstacks.  I just wanted to make sure it is stored somewhere other than on brasstacks.

Comment 21

[Mark Côté [:mcote]]

Attachment: Python files, default configuration, and batch-job shell script

A couple small fixes over the last patch: fixed the trend query links; fixed adding/changing/removing trends from config file.

Comment 22

[Mark Côté [:mcote]]

Attachment: Python files, default configuration, and batch-job shell script

Just a couple more tiny mods... removed unused class variable and added a couple comments.

Comment 23

[Mark Côté [:mcote]]

Attachment: Python files, default configuration, and batch-job shell script

Okay last edit for real: slight correction to a path in batchjob.sh.

Comment 24

[Jeff Hammel]

(In reply to comment #5)
> Here's an explanation of Murali's system, for posterity.
> 
> The system is divided into a number of scripts, which can generally be divided
> into two categories: "dumper" scripts that get bug data from Bugzilla according
> to a given query and insert this data into a local MySQL database, and
> "metrics" scripts which generate static HTML pages using the data in the local
> db.  These scripts are driven by a shell script, which also uploads the
> generated pages via scp to a web server (e.g. a people.mozilla.org account).
> 
> There are quite a few versions and varieties of these scripts.  It appears that
> the old ones used a SQLite database instead of a MySQL database (as seen in the
> OLD directory).  There are varieties of the dumper/metrics scripts to obtain
> information about AMO (add-ons) bugs, SUMO (support) bugs, bugs flagged for
> review, blockers, and bugs nominated for blockers.  There also appears to be an
> effort to make at least some of these (fairly redundant) files generic to some
> extent: generic-dumper.pl and genericbugmetrics.pl, confusingly stored in the
> SUMO directory.  These generic scripts obtain the output filename, page title,
> and search query from a file stored on the "fs" server retrieved via HTTP.  The
> output appears to be similar to that generated by the AMO and SUMO scripts (but
> *not* the same as the blocker scripts).
> 
> The entire process is driven by a shell script called "batchjob.sh". 
> Presumably it was run once or twice a day.  Obtaining all the bug information
> is a time-consuming process, since the XML for each bug is retrieved
> one-by-one.  Generating the HTML pages is extremely fast, however.
> 
> The database-storage step seems very strange at first; it appears to be
> completely unnecessary and only a result of the complete separation of the
> dumper and metrics scripts.  However, there is at least one script
> (timelinemetrics.pl) that gives results over time, thus necessitating the
> storage of the results of previous queries.
> 
> The generation and uploading of static HTML pages is also somewhat strange. 
> The usual solution would be an inversion of the process: placing a slightly
> modified version of the metrics scripts on the web server as CGI scripts to
> generate pages on the fly, as requested.  However, given that the data is only
> refreshed periodically, perhaps there is nothing to be gained in this situation
> by inverting the process.
> 
> It's likely that direct (read-only) access to the Bugzilla database would speed
> up the dumper scripts immensely.  It is unclear whether this would be fast
> enough to allow completely dynamic, up-to-date reports; however, certainly the
> frequency of page generation could be increased.
> 
> There are also a couple obvious things that can be done to improve the scripts:
> 
> - Move DB data (database name, credentials) to a config file.
> - Move queries from nongeneric scripts to a config file.
> - Eliminate the redundancies in the scripts.  Even if multiple scripts are
> needed for special purposes (e.g. blocker reports vs. AMO/SUMO reports), much
> of the code could be put into modules for reuse.

Please document this in the code when ready for checkin

Comment 25

[Jeff Hammel]

Comment on attachment 483815 [details] [diff] [review]
Python files, default configuration, and batch-job shell script

So I'm a bit confused on whether this is an application or just a few scripts that are specialized to the brass-tacks server.  I'm going to comment as if this was a general purpose app with the standard requirements that it should be deployment-agnostic (that is, it should run on a wide class of computers).  

>diff --git a/batchjob.sh b/batchjob.sh
>new file mode 100755
>--- /dev/null
>+++ b/batchjob.sh
>@@ -0,0 +1,6 @@
>+#!/bin/sh
>+cd $HOME/blocker-reports
>+./bugtrends.py -u > index.html
>+mv index.html /usr/local/nginx/html/blocker-reports/
>+cp bugtrends.csv /usr/local/nginx/html/blocker-reports/
>+
Maybe /usr/local/nginx/html/blocker-reports should be made a variable?
And maybe its worth adding comments as far as what's going on here?

>diff --git a/blocker_counts.py b/blocker_counts.py
>new file mode 100755
>--- /dev/null
>+++ b/blocker_counts.py
>@@ -0,0 +1,67 @@
>+#!/usr/bin/env python2.6
>+
>+import ConfigParser
>+import csv
>+import datetime
>+import json
>+import os
>+import urllib
>+from collections import defaultdict
>+
>+def get_current_counts():
>+    try:
>+        bugstats_rdr = csv.DictReader(open('bugtrends.csv', 'r'))
>+    except IOError:
>+        return []
Is this return too magical?  How is it okay to return [] blindly?

>+    return [row for row in bugstats_rdr]
>+
>+def update_blocker_counts():
>+    cfg = ConfigParser.RawConfigParser()
>+    cfg.read('bugtrends.conf')
>+
>+    new_query_data = {'date': datetime.date.today().isoformat()}
>+
>+    for name, query in cfg.items('queries'):
>+        json_data = urllib.urlopen('https://api-dev.bugzilla.mozilla.org/0.7/bug?'
>+                                   + query).read()
>+        bugs = json.loads(json_data)['bugs']
>+        new_query_data[name] = len(bugs)
Ideally, this function would just take a dict.  Then you could do the 'bugtrends.conf' specific stuff outside


>+        #print '%s: %d' % (name, len(bugs)) 
Nix this, plezse.  or at least add an `if DEBUG`

>+
>+        ### Number of bugs assigned to people... not needed for now.
>+        #assigned_to = defaultdict(int)
>+        #for name in [x['assigned_to']['name'] for x in bugs]:
>+        #    assigned_to[name] += 1
>+        #bug_assigns = assigned_to.items()
>+        #bug_assigns.sort(lambda x, y: cmp(y[1], x[1]))
>+        #for name, bugs in bug_assigns:
>+        #    print '%5d %s' % (bugs, name)

If we're not going to use this, why keep it around?  Could it at least be put into a function that wasn't called?


>+
>+    bugstats = get_current_counts()
>+    bugstats.append(new_query_data)
>+
>+    # FIXME: Using the headers only from the most recent data, based on the
>+    # conf file, means we will lose data if a trend is removed.  Perhaps we
>+    # need to keep old headers in case of a mistaken conf-file edit...?
>+    headers = new_query_data.keys()
>+
>+    # ensure date field is first, since it looks better when glancing at the
>+    # CSV file.
>+    headers.remove('date')
>+    headers.insert(0, 'date')
>+
>+    bugstats_out_file = open('bugtrends.csv.new', 'w')
Why this specific filename?  Maybe it should go as a default argument?


>+    bugstats_out_file.write(','.join(headers) + '\n')
>+    bugstats_wtr = csv.DictWriter(bugstats_out_file, headers, restval='0',
>+                                  extrasaction='ignore')
>+    for row in bugstats:
>+        bugstats_wtr.writerow(row)
>+    bugstats_out_file.close()
>+
>+    os.rename('bugtrends.csv.new', 'bugtrends.csv')
I would *not* do this.  why?  If there is a reason, it should really be commentted on.  If not it should be removed.  r- 

>+
>+    return bugstats
>+
>+if __name__ == '__main__':
>+    print update_blocker_counts()
>+
>diff --git a/bugtrends.conf b/bugtrends.conf
>new file mode 100644
>--- /dev/null
>+++ b/bugtrends.conf
>@@ -0,0 +1,29 @@
>+# Names of trends to track.  Each trend must also have a query defined in the
>+# [queries] section.
>+[trends]
>+total_blockers = Total blockers
>+nominations = Nominations
>+regressions = Regression blockers
>+blocking_any_beta = Blocking any beta
>+blocking_next_beta = Blocking next beta
>+
>+# Query URL suffix.  This query is appended to the REST API URL to get the
>+# results.
>+[queries]
>+total_blockers = quicksearch=blocking2.0%3Afinal,beta
>+nominations = quicksearch=OPEN%20blocking2.0%3A%22%3F%22
>+regressions = quicksearch=blocking2.0%3Afinal,beta&keywords=regression
>+blocking_any_beta = quicksearch=blocking2.0%3Abeta
>+blocking_next_beta = quicksearch=blocking2.0%3Abeta7
>+
>+# Define one or more graphs to generate.  Each graph can plot one or more
>+# trends.  To ensure that they are displayed in the desired order, you can
>+# start each graph ID with a number; the IDs are only used internally.
>+[graphs]
>+1open_nom_blockers = total_blockers, nominations, regressions
>+2beta_blockers = blocking_any_beta, blocking_next_beta

Not a huge fan of the magic numbers, but okay.

>+# Name of the graph, displayed above it in the resulting HTML page.
>+[graph names]
>+1open_nom_blockers = Daily trends of Open and Nominated Blocker bugs
>+2beta_blockers = Daily trends of Beta Blockers
>diff --git a/bugtrends.py b/bugtrends.py
>new file mode 100755
>--- /dev/null
>+++ b/bugtrends.py
>@@ -0,0 +1,196 @@
>+#!/usr/bin/env python2.6
>+
>+import ConfigParser
>+import csv
>+import errno
>+import sys
>+from optparse import OptionParser
>+
>+import blocker_counts
>+
>+class DataTable(object):
>+
>+    """ Produces the JavaScript and HTML for a google DataTable visualization.
>+    """
>+
>+    def __init__(self, vis_type, title, name):
>+        self.vis_type = vis_type
>+        self.title = title
>+        self.name = name
>+        self.cols = []
>+        self.rows = []
>+        self.html_header = ''
>+        self.html_footer = ''
>+        self.div_height = 300
>+
>+    def js(self):
>+        s = '            var %s = new google.visualization.DataTable();\n' % \
>+            self.name
>+        for c in self.cols:
>+            s += '            %s.addColumn("%s", \'%s\');\n' % (self.name, c[0], c[1])
>+        s += '            %s.addRows(%d);\n' % (self.name, len(self.rows))
>+        for i, r in enumerate(self.rows):
>+            for j, c in enumerate(r):
>+                # ignore all extra, undefined columns
>+                if j >= len(self.cols) or c == '' or c == None:
>+                    break
>+                s += '            %s.setCell(%d, %d, ' % (self.name, i, j)
>+                if self.cols[j][0] == 'string':
>+                    s += "'%s'" % c
>+                else:
>+                    s += '%s' % c
>+                s += ');\n'
>+        if self.vis_type == 'columnchart':
>+            s += '            new google.visualization.ColumnChart(document.getElementById("%s")).draw(%s, {isStacked:true,is3D: true, min:0});\n' % (self.name, self.name)
>+        elif self.vis_type == 'table':
>+            s += '            new google.visualization.Table(document.getElementById("%s")).draw(%s, options);\n' % (self.name, self.name)
>+        return s
>+
>+    def html(self):
>+        s = self.html_header + self.html_title() + \
>+            '<div id="%s" style="' % self.name
>+        if self.div_height:
>+            s += 'height: %dpx; ' % self.div_height
>+        s += 'background-color: #FFFFFF"></div>\n' + self.html_footer
>+        return s
>+
>+    def html_title(self):
>+        return '<h3>%s</h3>\n' % self.title
>+
>+
>+class DataTableCollection(object):
>+
>+    """ Produces JavaScript and HTML for a number of Google DataTable
>+    visualizations.
>+    """
>+
>+    def __init__(self):
>+        self.next_table_num = 0
>+        self.tables = []
>+
>+    def new_table(self, vis_type, title):
>+        """ Factory function for new tables.  Use this rather than creating
>+        the tables directly to ensure that they all have new names and that
>+        the Collection knows about them all and hence can display them in one
>+        shot. """
>+        d = DataTable(vis_type, title, 'data%d' % self.next_table_num)
>+        self.next_table_num += 1
>+        self.tables.append(d)
>+        return d
>+
>+    def js(self):
>+        s = self.js_header()
>+        for t in self.tables:
>+            s += t.js()
>+        s += self.js_footer()
>+        return s
>+
>+    def html(self):
>+        return '<hr><br>\n'.join(map(lambda x: x.html(), self.tables))
>+
>+    def js_header(self):
>+        return '''
>+        <script type="text/javascript" src="http://www.google.com/jsapi"></script>
>+        <script type="text/javascript">
>+            google.load('visualization',  '1', {packages: ['columnchart']});
>+            google.load('visualization',  '1', {packages: ['table']});
>+        </script>
>+        <script type="text/javascript">
>+            function drawVisualization() {
>+            // Create and populate the data table.
>+            var options = {};
>+            options['allowHtml'] = true;
>+'''
>+
>+    def js_footer(self):
>+        return '''
>+            }
>+            google.setOnLoadCallback(drawVisualization);
>+        </script>
>+        '''
>+
>+parser = OptionParser()
>+parser.add_option('-u', '--update', dest='update', action='store_true',
>+                  default=False, help='update counts from Bugzilla first')
>+(options, args) = parser.parse_args()
>+
>+if options.update:
>+    bugstats = blocker_counts.update_blocker_counts()
>+else:
>+    bugstats = blocker_counts.get_current_counts()
>+
>+last_date = bugstats[-1]['date']
>+
>+tables = DataTableCollection()
>+
>+cfg = ConfigParser.RawConfigParser()
>+cfg.read('bugtrends.conf')
>+
>+for graph_id, trend_list in cfg.items('graphs'):
>+    try:
>+        graph_name = cfg.get('graph names', graph_id)
>+    except ConfigParser.NoOptionError:
>+        graph_name = graph_id
>+    graph = tables.new_table('columnchart',
>+                             '%s :: %s' % (graph_name, last_date))
>+    graph.cols = [('string', 'Date')]
>+    trends = map(lambda x: x.strip(), trend_list.split(','))
>+    for t in trends:
>+        graph.cols.append(('number',  cfg.get('trends', t)))
>+    for b in bugstats:
>+        row = [b['date']]
>+        for t in trends:
>+            row.append(b[t])
>+        graph.rows.append(row)
>+
>+today_stats = bugstats[-1]
>+if len(bugstats) > 1:
>+    yesterday_stats = bugstats[-2]
>+else:
>+    yesterday_stats = dict.fromkeys(today_stats, 0)
>+    yesterday_stats['date'] = ''
>+
>+changes_table = tables.new_table('table',
>+                                 'Changes after 9:00 AM %s' % last_date)
>+
>+changes_table.cols = (('string', 'Type of Bugs'),
>+                      ('number', 'Current Number'),
>+                      ('number', 'Net Change'))
>+
>+for trend_id, trend_name in cfg.items('trends'):
>+    changes_table.rows.append((
>+        '<a href="https://bugzilla.mozilla.org/buglist.cgi?%s">%s</a>' %
>+        (cfg.get('queries', trend_id), trend_name),
>+        int(today_stats[trend_id]),
>+        int(today_stats[trend_id]) - int(yesterday_stats[trend_id])))
>+
>+changes_table.div_height = 150
>+changes_table.html_header = '<center>'
>+changes_table.html_footer = '</center>'
>+
>+
>+print '''
>+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
>+<html xmlns="http://www.w3.org/1999/xhtml">
>+  <head>
>+    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
>+    <title>
>+      Open Blocking Bugs Information
>+    </title>'''
>+
>+print tables.js()
>+
>+print '''  </head>
>+  <body style="font-family: Arial;border: 0 none;">
>+  <div style="float: right; text-align: right">
>+  <i>get the <a href="bugtrends.csv">raw stats</a> (CSV)<br/>
>+  note: graphs and statistics do <b>not</b> include security bugs</i>
>+  </div>
>+'''
>+
>+print tables.html()
>+
>+print '''
>+  </body>
>+</html>
>+'''

PLEASE use a template for this.  It could be as simple as having the string in a file and interpolating that.  (using foo = string.Template(file('foo.html')) etc
I would use e.g. tempita (or genshi or...).  But yeah, I can't possibly get behind this.  And please don't use inline styles.  You should have a stylsheet in the <head> block

Comment 26

[Jeff Hammel]

> So I'm a bit confused on whether this is an application or just a few scripts
> that are specialized to the brass-tacks server.  I'm going to comment as if
> this was a general purpose app with the standard requirements that it should be
> deployment-agnostic (that is, it should run on a wide class of computers).

Sorry, this wasn't supposed to make it to the final comment.  hardware errors + PEBCAK

Comment 27

[Mark Côté [:mcote]]

Attachment: Revised patch: Python files, HTML & CSS templates, default configuration, and batch-job shell script

Okay, made the following changes:

- refactored blocker_counts.py into a class and renamed file to (more accurate) bugstats.py
- split reading, writing, and updating code in bugstats into three separate functions
- added comment regarding the setting of bugstats to [] when there is no file (it is desired behaviour)
- removed commented-out code
- moved 99.9% of all HTML into a template, using tempita (only exception is the row names for the net-change table, which use links generated from the (configurable) bugzilla query API and the particular trend query values)
- moved CSS into a separate file
- made template name, csv name, bugzilla REST and regular query URLs configurable with sane defaults
- configuration filename configurable via command-line option
- batchjob.sh uses variables for increased clarity.

Comment 28

[Jeff Hammel]

Comment on attachment 484150 [details] [diff] [review]
Revised patch: Python files, HTML & CSS templates, default configuration, and batch-job shell script

I didn't mean that the CSS had to be separate from the HTML (e.g. inline stylesheets, fine, inline styles...less so).  But WFM

Comment 29

[chris hofmann]

Looks like http://blocker-reports.brasstacks.mozilla.com/ is public now.  Its looking great.  If we could swap the regression blockers and the nomination blockers that would be great.  that would put all the "hard blockers" on the bottom, and the the more fuzzy category of nominations on top.

but that's just an enhancement request.

smooney also found an interesting bugzilla report that breaks some of these categories down by bug owner to help drilling down on who is reposonsible for delivering these fixes.   that would be a good link to add on the page.

Thanks for working on this!

Comment 30

[chris hofmann]

er, I guess if we are going to show that orange stack for the regression blockers on the same graph it might go best on the bottom of the stack.  those are really the key bugs to be fixed so the next release is as good or better than the last.  so the stack would look like

                     |
         nominations |
                     -
                     |
                     |
                     |
rest of the blockers |
                     -
                     |
blocker regressions  |
--------------------------------

but if this gets too complicated or cluttered we can just put the blocker regression on there own chart like muralli's old chart


                     |
         nominations |
                     -
                     |
                     |
                     |
            blockers |
--------------------------------


this would allow us to do some interesting things with regressions showing both the count of nominations that are regressions, and the number of confirmed blockers that are regressions.


nominations that are |
regressions          |
                     -
                     |
                     |
                     |
blocker regresssions |
--------------------------------

Comment 31

[chris hofmann]

the other thing we could do is to go in and clean out some of those multiple samples that I took on some days when I was gathering the data by hand.  lets keep the high point for each of the days where we have multiple samples.

20100928,859,169,481,32,228,
20100929,878,123,488,26,228,  --remove--
20100929,884,124,491,21,229,
20100930,887,134,495,21,233,  --remove--
20100930,900,126,507,23,234,
20101001,903,135,508,23,234,  --remove--
20101001,907,129,509,19,236,
20101002,905,129,507,17,236,
20101002,903,145,505,15,235,  --remove--
20101004,914,167,514,18,240,  --remove--
20101004,956,127,552,17,261,
20101005,955,143,550,17,263,  --remove--
20101005,965,123,559,15,265,
20101006,966,128,559,16,267,
20101006,961,117,561,16,264,  --remove--

Comment 32

[Mark Côté [:mcote]]

Okay all that shouldn't be hard.  I'll see what I can do later today.

Comment 33

[Sheila Mooney]

I like to track assignments since it tells us who is overloaded and in our case, we leave a lot of stuff assigned to nobody or general.js. http://bit.ly/9lZFIj. This URL just gives a snapshot which is useful. Not sure if it's easy to track this on a weekly basis in some kind of graph.

Comment 34

[Mark Côté [:mcote]]

(In reply to comment #31)
> the other thing we could do is to go in and clean out some of those multiple
> samples that I took on some days when I was gathering the data by hand.  lets
> keep the high point for each of the days where we have multiple samples.

Multiple days removed as you noted, plus the lower value for 2010-09-28.

Comment 35

[Jeff Hammel]

pushed:http://hg.mozilla.org/automation/blocker-reports/

Comment 36

[Mark Côté [:mcote]]

(In reply to comment #30)
> [...]
> but if this gets too complicated or cluttered we can just put the blocker
> regression on there own chart like muralli's old chart
>
> [...]
>
> this would allow us to do some interesting things with regressions showing both
> the count of nominations that are regressions, and the number of confirmed
> blockers that are regressions.

So just to clear: for now, would you just like your first suggestion (I don't think it's too complicated, but I'll let you be the judge), or should we just go ahead an implement your second suggestion?

Comment 37

[chris hofmann]

lets do the second suggestion.

Comment 38

[Mark Côté [:mcote]]

Okay, I have a new, test version up at http://people.mozilla.com/~mcote/blocker-reports/.  Let me know if that's what you're looking for.  The second graph isn't too interesting yet because we don't have previous numbers for regression nominations, but it should give you an idea.  If it looks good, I'll put the new script up on brasstacks.

Comment 39

[Mark Côté [:mcote]]

Oh and please let me know if the query for regression nominations is okay, just to be sure.  You can see it if you look at the link in the "Changes after ..." table at the bottom.

Comment 40

[chris hofmann]

> test version up at http://people.mozilla.com/~mcote/blocker-reports/.  
> Let me know if that's what you're looking for.

This looks great!  Thanks for the fast turn around.

> Oh and please let me know if the query for regression nominations is okay, 
> just to be sure.

Yeah, those queries look correct.

> If it looks good, I'll put the new script up on brasstacks.

Ship it! ;-)

Comment 41

[Mark Côté [:mcote]]

Attachment: Patch on top of previous patch to graph more trends and allow simple trend arithmetic

This is a patch to generate the new graphs (updated conf file), but also to allow trend math--meaning in addition to plotting the numbers from a given query, we can plot numbers calculated from several queries, e.g. total blockers - blocker regressions.

This turned out to not be necessary right now, but I think it's still useful and could well be used in the future.

Comment 42

[Jeff Hammel]

Comment on attachment 485057 [details] [diff] [review]
Patch on top of previous patch to graph more trends and allow simple trend arithmetic

This is fine.  I am a bit hesitant on approving this, as CalcParse is a generic module and it feels weird putting it here.  It is too bad that SimpleCalc is not a module.

My approach would be:
 - Make SimpleCalc/CalcParse into a python package and upload it to pypi
 - my blocker-reports a python package and have it depend on e.g. SimpleCalc
 - do whatever cleanup is necessary to make `python setup.py develop` most of a deployment story

This is fine for a quick solution, but if this code ends up being touched much more, I would highly recommend taking a step back and seeing if a more modular approach is worthwhile.

Comment 43

[Sheila Mooney]

Does anybody think it would be useful to show blockers and non-blockers fixed over time? I don't want to add work unless people think it would be useful.

Comment 44

[Mike Beltzner [:beltzner, not reading bugmail]]

Yeah, I think that'd be helpful. A breakdown of state changes showing:

 - blockers/non-blockers marked FIXED
 - blockers/non-blockers REOPENED
 - blockers/non-blockers resolved as DUPLICATES

Comment 45

[Sheila Mooney]

I spoke to mcote. So for the blockers/regressions/nominations we have a running total and net change. Do we want to see blockers/non-blockers fixed over time? Per day? Per week? This might be harder to do.

Similarly, we could think about rate bugs are adding into the system - new bugs, new blockers, new nominations on a daily, weekly basis.

Comment 46

[Mark Côté [:mcote]]

We are now tracking fixed, reopened, and duplicate blockers.

Do you really want all fixed, reopened, and duplicate nonblockers?  That's going to be some huge numbers, given that we have over 600 000 bugs.  But if you think it will be useful, I can add it easily.  Or let me know if I should refine the query somehow.

Comment 47

[Mark Côté [:mcote]]

Should I update the next-beta query to look for beta8 instead of beta7 now?

Comment 48

[chris hofmann]

yeah, good idea.

Comment 49

[Mark Côté [:mcote]]

Done, the next-beta results tomorrow will reflect beta8.

Comment 50

[Mark Côté [:mcote]]

Updated today for beta9.  At some point I might add a UI to make this easier...

Comment 51

[Mark Côté [:mcote]]

So the blocker tag isn't really relevant anymore.  Is there someway that the blocker reports could be made useful again?

Comment 52

[chris hofmann]

there are a set of 'triage' bug queries on these pages that now become the interesting stats to watch current levels and trends on.

https://wiki.mozilla.org/Firefox/Aurora
https://wiki.mozilla.org/Firefox/Beta

figuring out how to make the page show channel, and specific release info in a systematic probably needs some thinking.

Comment 53

[Mark Côté [:mcote]]

This bug is old and was committed and deployed, plus we've changed how we do release tracking and other efforts have since come along, so closing this out.