483833 - Add Editor Reports to Editor tools page

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Admin/Editor Tools, defect, P1)

Description

[Rey Bango]

Create reports that allow editors to receive stats on their individual contributions. This could have the benefit of allowing editors to determine where they’re contributing the most and if they can expand their help into other areas.

* Weekly reports to individual editors showing which add-ons they've reviewed and what the average number of reviews for the editorial group

* Breakdown by:
- Type of add-on reviewed (Firefox, Thunderbird, etc.)
- Category of add-on
- YTD totals of add-ons reviewed

* Display the date of review, add-on name, & review status

* Graphs showing when they've been most active over a period of time

Comment 1

[Wil Clouser [:clouserw]]

What do you think of adding this to the user details page?  and making it public?

Comment 2

[Rey Bango]

I'll generate a mockup and then pass the torch.

Comment 3

[Rey Bango]

Attachment: PRD for the reports

Attached is the spec for the reports.

Comment 4

[Rey Bango]

Attachment: PRD for the reports (PDF format)

Comment 5

[Nick Nguyen [:osunick]]

Would be great if we could get a contractor helping us with this for 5.0.6

Comment 6

[Rey Bango]

Morgamic, I know you've interviewed two potential hires that I've chatted with you about. Possible candidates?

Comment 7

[Nick Nguyen [:osunick]]

This is potentially something our new contractor can work on.

Comment 8

[Scott McCammon [:mccammos]]

Attachment: Editor report 1 mockup

Attached is my interpretation of report 1 as described in the spec. Is this what you had in mind, Rey?

In the weekly summaries, should a partial week be displayed differently?

Comment 9

[Rey Bango]

This looks really good Scott. It looks like you've covered everything for Editor Report 1. Perfect. One thing that I wanted to clarify. Sorting will be allowed by add-on name, correct? I ask because I didn't see the arrows next to the Add-on label in the mockup.

One slight update. Could you please make the date fields default to the 1st day of the current month and the current day the report is being run. It's less info that has to be typed in. So if I ran the report today, the start date would be 5/1/09 & the end date would be 5/11/09.

In terms of weekly totals, no need to change the display for partial weeks.

Comment 10

[Scott McCammon [:mccammos]]

(In reply to comment #9)
Rey, we'll certainly do sorting by Add-on. Sorry for overlooking that in my mockup. Default dates have been noted as well.

Comment 11

[Scott McCammon [:mccammos]]

The best way to calculate the editor team size over any time range?

Calculating team averages over any given time requires knowledge of the team size which itself can fluctuate over time via group membership addition and removal. My thinking is to approximate team size by counting the number of unique users that log a review over a certain time range.

1. Should team size be calculated for each date range that an average is computed (weekly, MTD, YTD, and custom range)?

Some problems with this approach would include:
 - Someone taking the week off would skew a weekly team average up.
 - Several newly active editors would skew a YTD team average down.

2. Should a single team size be calculated for a report based on unique team members over a longer range (3 months for example) that intersects the report's range?

If actual team size sees little fluctuation, this would work well and be easy to calculate.

3. Another approach would be to calculate a moving average of team size which would smooth out the side effects of activity/inactivity spikes.

Now that I've written this out, I'm favoring option 2. However, I don't know how much editor group membership fluctuates. Would anyone care to weigh in? :-)

Comment 12

[Rey Bango]

Great catch Scott. The team size doesn't fluctuate much and when it does, it's usually due to an addition. We get about 1-2 new editors per month it seems. With that in mind, I'm thinking that option 2 is the best route, as you suggested.

Comment 13

[Scott McCammon [:mccammos]]

Rey,

For the category breakdown pie charts, should a review of an add-on having multiple categories contribute:

1) 1 count for each category

or

2) 1/(number of categories) count for each category

I'm thinking option 2 would keep the "other" category pie slice from always dominating since it is very common, although we could ignore "other" when it is not the only category.

Comment 14

[Rey Bango]

Hey Scott. I'm thinking that going w/ option 1 & not counting the "other" category may be the better choice. Can you pull up a query of add-ons that exclusively use the "other category? If it's a small number, then it may not be worth worrying about.

Comment 15

[Scott McCammon [:mccammos]]

Thanks Rey. I'll go with that and we'll see how the distributions work out.

A quick query of my dev database (snapshot at the beginning of the month) shows about 1400 extension add-ons with "other" as their only category. That includes inactive addons, but is still significant. I'll make sure these get included.

Comment 16

[Scott McCammon [:mccammos]]

Attachment: giant patch v1

Attached patch does everything except extension category breakdown for pie charts. Due to fast approaching code freeze, I don't think I can get that level of detail finished in time. Bump to 5.0.7 ?

Comment 17

[Rey Bango]

I have no problems with this going out in 5.0.7 if it'll be feature complete. I'd rather give you more time if necessary, Scott.

Comment 18

[Scott McCammon [:mccammos]]

Attachment: giant patch v2

Same patch but excluding tmp files.

Comment 19

[Scott McCammon [:mccammos]]

Rey, the pie charts currently breakdown on the 7 addon types, so they are still fairly useful and could be pushed in this release. A follow-up release would see the same information, just with the extension slice divided farther.

Comment 20

[Rey Bango]

Ok. I know you just submitted the patch. Is it viewable now? If so, I'll check it out and give you some feedback.

Comment 21

[Scott McCammon [:mccammos]]

Rey, for a preview visit:

http://mccammos.khan.mozilla.org/remora/site/en-US/editors/performance

As an admin, you should be able to generate reports for anyone. Note that my database doesn't have any approvals after early May.

Comment 22

[Wil Clouser [:clouserw]]

Comment on attachment 381418 [details] [diff] [review]
giant patch v2

Thanks Scott.  This is great.  I'm sure Rey will <3 you.  I think it could use some UX polish but overall gets the info to the people that need it.  I don't want to read through a whole 'nuther giant patch so I'll r+ this one with some comments to fix:

In one function you're checking if the user is a senior editor or admin, but if they fail you still let them through for their own account and the other function doesn't have any checks.  You should check SimpleAcl->actionAllowed('Editor', '*') at the top of those two functions.

+        $endDate = 'YYYY-MM-DD';
This should be localized.

+                if (!assert('array_key_exists($weeklyKey, $weeklyTotals)')) {                                     
+                    die;                                                                                          
+                } 
I'd rather not die in code.  Can you use the notification element to give some feedback?  At least load something on the page so the user can give us a message.  Otherwise we just blame IT. :)

You don't need to fight with the WebHelper but can you write some tests to verify things like _performanceTable() returns an array with a couple values you are expecting?  A couple tests like that could keep us from breaking this down the line.

I think you can land this for 5.0.6 and we can get some feedback.  There won't be any L10n though since I already pushed the last of those strings to localizers.  So use the built in fallback to ___() and we'll pick up the strings in 5.0.7.

Comment 23

[Rey Bango]

(In reply to comment #21)
> Rey, for a preview visit:
> 
> http://mccammos.khan.mozilla.org/remora/site/en-US/editors/performance
> 
> As an admin, you should be able to generate reports for anyone. Note that my
> database doesn't have any approvals after early May.

Hey Scott. Dying to see it. Unfortunately, the page doesn't load for me. I have the VPN running but can't hit that page. Any ideas?

Comment 24

[Wil Clouser [:clouserw]]

probably broken DNS.  Hardcode the path in your hosts file.

Comment 25

[Scott McCammon [:mccammos]]

Thanks for the feedback, Wil!

> You should check SimpleAcl->actionAllowed('Editor', '*') at the top of those
> two functions.

I'll add the check in function performance(), which is the single entry point for everything here. What other function were you referring to?

> +        $endDate = 'YYYY-MM-DD';
> This should be localized.

Agreed. I followed the filtering on the review logs page for consistency, but I'll make it right.

> I'd rather not die in code.

I about died when I saw that I let that slip.

Comment 26

[Scott McCammon [:mccammos]]

Attachment: tests and small fixes to patch v2

Comment 27

[Scott McCammon [:mccammos]]

r27171

Comment 28

[Scott McCammon [:mccammos]]

r27181

Moving extension category breakdown to a new bug for 5.0.7

Comment 29

[Stephen Donner [:stephend] Not actively reading bugmail]

removing "push-needed" from 105 AMO 5.0.6 bugs; filter on "I hate stephend!"