Closed Bug 606240 Opened 14 years ago Closed 11 years ago

Simple web site to display bug stats by team and by person

Categories

(Websites :: Other, defect)

Product:
Websites
Component:
Other
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mcote, Unassigned)

Details

damons and others have requested a simple web site to display bug stats by team and by person.

The stats that should be tracked are number of
- blockers
- nominations
- reviews
- security bugs
- crashes
- blockers closed in last 30 days
- nonblockers closed in last 30 days

The top page should list all the teams with aggregate numbers.  Choosing a team should display the name and picture for everyone in that team with the same stats per person

http://toolness.github.com/reviewer-dashboard/ can be used as a basis; however, graphs aren't needed, only the numbers.

The pages should require little in the way of web-server support--in order words, it should be simple to drop them into any web-server directory, such as a people.mozilla.com account.  Thus all the dynamic content will have to be collected through JavaScript, on the client side, using the Bugzilla REST APIs.
Assignee: nobody → mcote
Status: NEW → ASSIGNED
View #1: Stale Bugs

So I would like to have some kind of view of all 2.0 blockers(beta, final) showing a time span of the last change date. I don't know the easiest way to do this ie: by day, by week etc. What we are trying to show is that there are current blockers that haven't been updated since June, May etc. It would be nice if for each day (or whatever time we are using), that there are bands for # of regressions, crashes, security bugs and other.

On the blocker/nomination graphs there are popups for the #. Could we link to a bugzilla query returning ie: list of all crashes that haven't been modified since Sept 1 2010...that kind of thing.
Oh, this is a lot more specific than what I described above.  Is this view in addition to the main stats view(s)?
Yes, this is in addition. To summarize, I think there are like 3-4 views we need here in addition to the blocker ones. 

+ Managers, PM types like to see the blocker count and how that's decreasing. They would also like to see stuff like "activity". This means comments in bugs (blocker and non-blocker), new bugs, resolved bugs, nomination queue. Maybe some lookups like bugs changed in last 4 days. I put the stale bugs view in the "manager/pm" type category...people like me, beltzner etc might want to see this.

+ Group leads - they would want to see data for their group. Number of outstanding reviews, noms, unowned bugs, resolved, maybe bugs that haven't been commented on in the last 7 or 14 days. They probably want to be able to see this info on a group basis and an individual basis as well.

+ Developer view - I can see my stuff. This covers much of the stuff in comment #1. The number of blockers I have assigned to me, nominations for my group, reviews in my queue, security bugs assigned to me, crashers assigned to me, blockers and non-blockers I have closed in the past 30 days. It would be also great to see a list of stuff I could choose from next. Maybe all bugs assigned to my group and unassigned bugs for my group.

So one of the subtleties about this is that groups are made of of multiple components in bugzilla. Damon is supposed to provide me with that breakdown. We then have a list of what developers are in each group.

I don't know the right iterations for all this. Perhaps starting with the developer view is a good one and we can go from there. I am open to suggestions.
I should have added...I can put some of these in a separate bug if that's easier. Just let me know.
Hmm maybe just a separate bug for the Manager/PM view.  The other two sound like they have more in common, so I'd probably tackle them first and then get to the Manager view after, if that makes sense.
I've been looking at Atul's other dashboard for inspiration:

http://toolness.github.com/bugzilla-dashboard/

This does more than we need (lists of all kinds of bugs), but it already has a
nice look and does the authentication stuff.  It also has a nice caching system
and good Bugzilla integration.  It's also a little complicated, but I'm slowly
figuring it out.

So the developer screen will have a similar title and right-hand menu (probably
with a couple things removed), then immediately under the title a "quick stats"
section listing the numbers for all the queries you mentioned. (clarification:
we still only want numbers, right?  I can easily link to the full Bugzilla
queries for those numbers as well)  Under that we could list the assigned and
unassigned bugs for the developer's groups--though that will probably be quite
a long list, so I'm not sure how helpful it'll be.  Maybe just the oldest
unassigned bugs?

As for the group assignments, that's too bad that there's apparently no way to
automatically figure out the group assignments (members and components)--this
stuff will have to be updated frequently by someone.  Perhaps under one of the
views I can list users with bugs assigned to them who are not found in a group,
and perhaps the same with product components?
That makes perfect sense re: pm view. I also spoke to Beltzer and got some additional input. I added bug 606596. We might need to iterate on this one since I don't have my head fully wrapped around what it should look like. Starting with the more developer centric stuff makes sense.
Re: security bugs, is this the total number of bugs with the component "security" assigned to that developer/team?  Or only blockers?

What about crashers?
Another question: are the teams defined by Mozilla products/components, or by people, or both?
Re: security bugs....good question. Let me double check this.

As far as crashers are concerned, this would be anything with keyword=crash, topcrash and blocking 2.0. It would be nice to somehow indicate it in the UI if the person has a topcrash - that might be getting too fancy.

The teams are defined by both products/components and people. Damon is supposed to give me the breakdown. I will ping him again about that. The problem with this is that we keep having to make changes every time someone leaves/joins a team. Also, can someone belong to more than one team?
I checked some of the security queries. You can look for the component security and a whiteboard status as well.
You want to include the whiteboard status [sg: in your search as well.
Sorry, to clarify, for the crashers, do you want the number of bugs that have both "topcrash" and "crash" keywords, or the number of bugs that have either keyword?  I suspected you meant the former, but then your comment about indicating if the user has a topcrash doesn't make sense to me (if they have a nonzero number of bugs that have "topcrash" and "crash" keywords, doesn't that indicate that they have a topcrash?).
Sorry, I meant that we should search for either crash OR topcrash. In some cases a crash bug might have either or both. It's not consistent. When I look for topcrashes I just search for anything with the topcrash keyword (it might have crash as well). Some kind of indicator that there is at least one bug in the list that has the "topcrash" keyword would be cool, but we can put that in iteration #2.
Can this please be done as a Bugzilla extension? There are many benefits:

1. Permissions are all handled by bugzilla
2. Can use built in bugzilla reporting system / graphs (?)
3. 1 less external system to maintain
4. Don't need to pull data (keywords, products, people etc) out of bugzilla and keep that in sync
5. If it is general enough, other companies/projects can use it
6. There are similar PM features being discussed for Bugzilla, why duplicate work?

Please see http://www.bugzilla.org/docs/tip/en/html/api/Bugzilla/Extension.html for some docs. I'm not a perl or bugzilla hacker and I created an extension in a day or so. I'd also be happy to help.
(In reply to comment #15)
> Can this please be done as a Bugzilla extension? There are many benefits:
 
Yes and no.

One of the major reasons we want to do this on our own and build off the work by Atul is that we want to experiment with our own workflows and visualizations without worrying about what is possible and impossible in the bugzilla extension environment.  

It was a requirement that this not be server based, so that it could simply be a local page carried around on your computer that syncs itself with bugzilla via the REST API.  Whether that requirement has merit or not is another discussion.

It's also not clear how this would integrate with bugzilla in bugzilla's current UI or whether such integration would be possible in the extension interface.  Solvable, but it's not something we wanted to spend energy on right now.

Ideally, yes, I'd like to see all this behavior and visualization ability wired into bugzilla itself.  I'd like to see all this upstreamed as much as possible.  However, I don't know that we know exactly what we want yet.  This project is still in the prototyping stage.  We can prototype a lot faster in JS+REST API than we can in perl + extension API.

As the REST API is further integrated into bugzilla itself (this is happening in upstream bugzilla development) then we should be able to reuse large pieces of this code moving forward if we chose to, or we can at least reuse the behaviors and visualizations after we write the bugzilla integrations pieces in the bugzilla extension API.
(In reply to comment #16)
> (In reply to comment #15)
> > Can this please be done as a Bugzilla extension? There are many benefits:
> 
> Yes and no.
> 
> One of the major reasons we want to do this on our own and build off the work
> by Atul is that we want to experiment with our own workflows and visualizations
> without worrying about what is possible and impossible in the bugzilla
> extension environment.

Makes sense. I've just seen a bunch of tools that are incomplete but useful with overlapping functionality, differing UIs, hard to remember URLs, etc.

> 
> It was a requirement that this not be server based, so that it could simply be
> a local page carried around on your computer that syncs itself with bugzilla
> via the REST API.  Whether that requirement has merit or not is another
> discussion.

I think this requirement was so users wouldn't need to set up a ton of stuff on their machines to get the benefit. If it's in Bugzilla, it's hosted. Then again, I wasn't around when this discussion went on.

> 
> It's also not clear how this would integrate with bugzilla in bugzilla's
> current UI or whether such integration would be possible in the extension
> interface.  Solvable, but it's not something we wanted to spend energy on right
> now.

Yeah, my initial thought is this would be trivial but I admittedly don't know a ton about what is possible with the extension system. There are bugs to get this sort of thing integrated in bugzilla proper though (bug 130835, bug 397517).

> 
> Ideally, yes, I'd like to see all this behavior and visualization ability wired
> into bugzilla itself.  I'd like to see all this upstreamed as much as possible.
>  However, I don't know that we know exactly what we want yet.  This project is
> still in the prototyping stage.  We can prototype a lot faster in JS+REST API
> than we can in perl + extension API.

Ah, I wasn't aware this was in the prototype stage...I was talking about where I want the final implementation to go.

> 
> As the REST API is further integrated into bugzilla itself (this is happening
> in upstream bugzilla development) then we should be able to reuse large pieces
> of this code moving forward if we chose to, or we can at least reuse the
> behaviors and visualizations after we write the bugzilla integrations pieces in
> the bugzilla extension API.

Totally understand. Though, if the view/data was integrated in bugzilla we could expose it via the WebServices/REST API and others could mash it up as well without providing yet another API.

I'm not pushing too hard to have it developed as an extension, I just wanted to make sure that possibility was at least investigated...sounds like it was.
Okay, I have a first draft up:

http://people.mozilla.com/~mcote/bugzilla-dashboard/

A few notes:

- The team data is currently hardcoded in a file. A better solution will have to be devised--perhaps using Bugzilla groups.

- The UI could be improved--suggestions welcome. :)

- The site works without logging in, but the stats won't include any bugs that require special permissions.  You will have to log in to see those (with your Bugzilla login), which will trigger a complete refresh.

- As requested, it's a collection of HTML & JS files that can be plopped as-is into a public_html directory (as indeed I did).  It *can't* be run locally with a file:/// URL though, since it relies on session storage for caching, which is not allowed for file URLs.  If this is a hard requirement, we can always have a little python script start a local webserver and launch the browser to the right local page.

- It doesn't load pictures--I could do this, but that would require a separate login (with LDAP credentials) to access the phonebook, and I don't really think that's it's worth it.
At smooney's suggestion, I just added open regression blockers to the stats.
Mark, this is really great.  On your points above:

- Bugzilla groups would be great.  I use them a lot, but they also need to be updated (not a problem, I can do that, and that makes things even easier).  Is there even a way to get who's who out of bugzilla groups?

- I'd like to go over some UI ideas with you and Shiela directly I think.  I have a lot of thoughts here, and it would be great to actually spend some real brainstorming time with you.  Looking at this first draft, I think most of what we need data-wise is there, and I think it would be worth getting it right, even if it takes a little more time.

- It's fine that bugs that require special perms are not displayed unless you are logged in.  

- It's fine that we can't run it using the file:/// URL.  

- Not concerned about pictures at all.  In fact, all text is just fine.
1. Right, yeah, unfortunately I don't think there is a way to get group info out at the moment.  That might be an interesting side project for me, but I suppose even if I could implement that, it would take a while to see the light of day on bugzilla.mozilla.org.  Hmm.  It still might be the direction we want to move in, though, in the long run.

2. Yup I totally agree that some UI discussion is warranted.  How do you want to handle that?  Phone conference of some kind?  I won't be in MV for another two weeks, unfortunately.

Btw I also changed the security stats to be security blockers only, since the context here is, as Sheila pointed out, almost entirely blockers.  I may also split the review stats into blockers/nonblockers.
I think setting up a discussion before you come to town would be great if we can find a time. When you're here, we could spend time iterating on some of the ideas. I will take the next action on setting this up.
Summary of next iteration based on today's brainstorm....

1. New top level view grouping the current groups...
+ Platform - Layout, JS, Graphics, Content, General Platform
+ Firefox - User Experience FF, Localization, Firefox Front End
+ Mobile - Mobile
+ Labs
+ General Engineering - Engineering, Developer Tools, Tools & Automation, Security

2. For the top level we will list the current release we are working on ie: FF4 beta7. All the data for now is in the context of 2.0 but we might expand this so you can pick from a release list and drill down.

3. Box display - we are going to try and use a box display for each top level group, group, developer etc.

3. To keep the UI really tightly spaced we are going to use code for all the metrics to shorten them. Full time TBD. Tooltips can be used to get the full name.

4. We are going to try and use colors to indicate problem areas. We can start with a few simple ones. Nominations: >=50 read, >=25 yellow. Review Q: >= 8 yellow, >=15 red.

5. Additional team stats: unowned blockers, unowned topcrashes

4. Additional top level project stats:: Final blockers, beta blockers
Okay, I'm about half done this now, so I figure it's time for a preview:

http://people.mozilla.com/~mcote/bugzilla-dashboard/

I did the team organization and rearranged the display into columns.  I added the indicator lights, but they currently only function as drill-down links.  I basically just made up the abbreviations for the teams, and the developers are calculated dynamically following a simple little algorithm.  I also have yet to add the extra stats.

I want to make the lights a bit prettier--I might get my (artistic) wife to come up with a little transparent image file to add an extra dimension to the otherwise plain coloured div.  However this is generally, I think, what we are aiming for--so now's the time to decide if we like this layout :)

One thing that is a little bit weird is there are only multiple columns at the top level, since there are no further levels of teams.  Was this intended, or did I misinterpret anything?  Should the team/developer screen be a little different somehow?

Next thing I'll be working on is actually providing information with the status indicators, which I should be able to do tomorrow.  Then I'll add the extra stats, and we should have something to discuss by Friday. :)
Mark, this is a great next iteration. Damon is going to comment on the layout. I think we should have an intermediate view that shows the stats for all the devs on one screen like we did before. For example, once you see the stats for Content and all the green boxes, the next level down is all the dev stats. Not sure the best way to navigate there.

I see the bugzilla queries work...awesome. Let me play around with it a bit more.
Ah okay, you don't want status lights for developers, only for subgroups?
Not sure what people mean by status lights for developers vs subgroups.

Once you are in a specific group (i.e., layout, content, graphics), I think having a green/yellow/red box per developer is important. For example, if you click on the graphics team, you get a bunch of boxes, one per developer, and inside each box you'd have:

+------------------------+
|        ircnick         |
| Blockers: 12 Rev Q: 23 |
| Patches In Review: 8   |
| Assigned Bugs:         |
|  Crashes: 2            |
|  Regressions: 14       |
|  Security: 4           |
| Recent Fixes (-30d):   |
|  Blockers:  14         |
|  Non-blockers: 12      |
+------------------------+

or, if abbreviated:

+-----------------+
|     ircnick     |
| +'s: 12  RQ: 23 |
| P: 8  Crash: 8  |
| Reg: 14  sg: 4  |
|    +'s -30d: 14 |
|       Non-+: 12 |
+-----------------+


Then, each entry in each box would be a link to the query results.
Do you want to see a colored box at the top level ie: Graphics turns read if one of the developers within that group has something that puts them in the "red" zone.
Ohhhh sorry I didn't realize you wanted stats *in* the boxes.  Okay that makes more sense now.  So from the top level (divisions or whatever you want to call them), do you want similar, total stats for each group then inside each coloured box?
(In reply to comment #28)
> Do you want to see a colored box at the top level ie: Graphics turns read if
> one of the developers within that group has something that puts them in the
> "red" zone.

I don't know about if just one dev turns red, but maybe that combined with some other set of heuristics maybe, like we see more non-blockers than blockers being fixed by a group or something?
Also, since one of our top objectives is to help devs know what they should
work on next, we should include a section in the single-developer and group
views to include specific queues of items that should be picked next:

Looking for your next bug?  Here are lists of bugs for you organized by
priority to the project:
  1st:  Blocking Security bugs: 14
  2nd:  Blocking Crashes: 3
  3rd:  Blocking Regressions: 2
  4th:  Unowned Blockers: 35
  5th:  Still need triage: 22
  6th:  Check these next beta blockers to make sure they really block:  14
  7th:  Check these final blockers to make sure they really are blockers:  45

Just as an example.
Okay, another quick iteration.  Still lots to do, but I have two important features done:

- the boxes now contain stats in an abbreviated form for the team/dev
- the boxes now start off grey, and once all the stats are loaded for a given box, it turns green, yellow, or red.

Under what circumstances, if any, should the full query names be used in the boxes?

Also, at the moment the group boxes just use the average stats for the group to determine colour--eg total RQ divided by number of people.  It would be more accurate to load each dev separately (along with other heuristics as Damon suggested).  This would be quite a bit slower, though--for each box, it currently only does one query, but to know if a single member of a group is red, it would have to do individual queries for each member before displaying the colour of the group's box.  That would be quite a bit slower, and it's already somewhat slow.

However, I'm starting to rethink the way I'm doing the queries.  Right now, it does a whole bunch of "count" calls, which are a bit faster than full "bug" queries because only the total number is returned.  So for a team, it does one query for each statistic with all the bugzilla IDs of all members in the the group together.  Then it will do the same for all subgroups (if a top-level group) or individual developers (if group view).  But this results in quite a lot of queries--for each statistic, it's doing number of groups X number of subgroups for the top level, and for the group view, it's doing number of devs + 1 (for the group itself) .

Perhaps instead, it should do one big query for each group or subgroup, depending on the current view, but get more bug information, not just a total count.  Then on the client side, it can parse the information and look for the assignees of each bug and tally up the results.  This will mean more processing on the client side, but I think the big bottleneck is the number of queries right now, so some additional processing might be worth it.  This might take me an extra day or two to do.

So here's what's left to do as far as I can tell:

- release data (i.e. FF4 beta 7--simple if I just hardcode it for now)
- additional team and top-level stats
- bug priority list
- faster querying
- better group box indicators (not just average stats)

I'll probably tackle that in that order, if that makes sense to you.
Actually, upon thinking a bit more, I might start looking into the querying business first--the loading time is really slow, and speeding it up would greatly increase the usability of the site, and hence it's showoffability. :)
mcote, a few comments...

+ So I don't think I the data in the boxes right now. I just see blank boxes. Did we want to change the columns to boxes also? I am not entirely happy with the column/box UI look but let's see what they look like with the data. I still think we need to work the visual layout a bit more.

+ We need to abbreviate all the stats ie: BKRS instead of Blockers. We talked about making all that text shorter. Bolding it might make it stand out...perhaps that's getting too fancy. We want this thing, easy to read.

+ Let's do the heuristics for coloring boxes last. I think we need to get the right layout and data first, we can perfect that later.
Here's a hopefully relatively simple question:

Should the patch review queue and patches awaiting review be the total number of *patches* or *bugs*?  Some devs have two patches awaiting review in the same bug.  I'm thinking it should be the total number patches irrespective of how many are in the same bugs, which, coincidentally, was not possible with my old querying system but is possible in the new one I'm working on. :)  The total number of bugs is also possible, though, so let me know which you prefer.
Okay I redid the querying system today, bundling them together by division/group.  Mostly it's faster, since all the boxes immediately appear after the division/group-level stats are loaded.

However some queries are painfully slow--the review queue query for all of Platform (with 79 members) takes 2 minutes!  I'm going to see if splitting it into smaller queries with fewer people makes any difference; perhaps there's an exponential increase in load time as you add more people to the queries.  Perhaps.

Still, I think this is progress; I have yet to verify, but I'm pretty sure the total page load time is a fair bit faster now, even if it does get stuck for a while for on certain queries.
Okay new top-level design up as per smooney's recommendations (stats in a grey box, title links to division page (currently only the same info, more to be added once we're happy with the title page), no indicator lights).  Also added the release name in the title.  I also think I sped up the total stats-loading time by parallelizing the requests.
Mcote, this is starting to look much better. A would like to try something else to see if it helps with the visual display. 

+ On the top level, after the product release, can we have a heading Blockers, then indent All: #, Security: # Crashes:# Regressions:#. Similarly, with the Fixed in last 30 days", then indent below Blockers:#, Non-Blockers: #. Leave the patches, review Q the same.

+ I think it's probably overkill to have the release listed everywhere. I would just put it on the top level and remove it from the second/third level.
So bzAPI, the Bugzilla REST API that the dashboard uses, has been down for the last 7 or 8 hours.  I'm curious to revisit the early decision to make the dashboard app entirely client based--if it were at least partially a server-based app, it could cache data and at least be accessible in this kind of situation, even if the data were a little out of date (and this fact could be communicated to the user).

I am a little hesitant to raise such questions at this stage, but I feel like I have to.  It could well be that there are very good reasons for making this a client-side app, but I don't think I've actually heard the rationale, and maybe there is some in-between solution to the current kind of problem.
A summary of the next iteration....

Group Admin
- Look at using bugzilla groups for queries. This will make it easier since people can add themselves to a group, watch components etc.
- If we can't use bugzilla groups, think about a UI to add people to groups, almost like an admin UI.

Group View
- In the top section, we need a headline for "Stuff to do next". If I think of something better, we can change it. 
- This should include...
+ sg:crit
+ topcrash
+ nominations for the group
+ bugs assigned to nobody (general.js)
+ top 5 most active bugs
+ top 10 stale bugs
- We also want to list all the developers and the number of blockers they have (not sure how this will look visually - we should probably do this last). Maybe the top 5 people who have the most blockers or we can try and change the individual developer boxes to better order them...not sure
- We would like to show the next major release in the group view - not sure how to automate this.
Forgot to add one more. We want the patches and review q to show breakdowns of blockers vs non-blockers.
I've got a server implementation working now that caches results from bugzilla.  The timeout is currently 5 minutes, meaning that is the same query is repeated less than five minutes later, the previous results are used and the bugzilla server isn't queried again.  I might add a method to force a refresh, though that has the potential for abuse...

The site will be put up on brasstacks.mozilla.com (a server which my team uses for various web apps and services), but at the moment it needs some reconfiguration to allow https access, so it will be a day or two before it's up.  Meanwhile, since it looks like the Bugzilla groups functionality isn't good for us at the moment (though I might file some bugs to the Bugzilla people), I'm working on an admin screen to add groups along with people and product/components.
Okay, looks like the server is finally up!  The site is now at 

https://brasstacks.mozilla.com/bugzilla-dashboard/

The administration site is at

https://brasstacks.mozilla.com/bugzilla-dashboard/admin.html

The dashboard now only reflects the information that has been submitted via the admin page, so the numbers will be smaller at the moment, since it doesn't have everything the old site had.  I set up the divisions, the teams, and added one or two members to each team.  I can write a script to populate the team with the members from the old site (those compiled by Clint), if desired, or we can let people sign up themselves.

The admin page is finished except for some sort of permission mechanism, though I think that can wait until we're ready to publicize.

I'm nearly done the most-active bug list. Stale bugs might take a bit longer, so I will upload the new "Stuff to do" view when the most-active list is finished, probably tomorrow, so you can all take a look.
First version of this was deployed quite some time ago; no need to keep this open. (Service was taken down after release-train process largely did away with blockers.)
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.