Closed Bug 1591290 Opened 4 months ago Closed 3 months ago

Request for data from Pontoon to evaluate Manager engagement

Categories

(Webtools :: Pontoon, task, P2)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: delphine, Assigned: mathjazz)

Details

(This is directly related to our current OKR work for this quarter: https://docs.google.com/document/d/1ccPhSCuzj_eDTAFu8wNRM293EH0rhAXwrqLQVkilM-c/edit?ts=5db1e2a0)

Matjaz: we'd like to get some data out of Pontoon for the following items discussed in our OKR doc linked above.

We've decided that these 5 data points are the most relevant to get out of this exercise at this point:

  • Time to graduate to new role
  • Time to approve/reject unreviewed strings (aka, how long do unreviewed strings remain pending)
  • Self-approval of managers
  • Self-approval of translators
  • Number of managers/translators reviewing strings

From what we've gathered, we have existing scripts (https://github.com/flodolo/scripts/tree/master/pontoon) for three of these already:

  • Time to approve/reject unreviewed strings (aka, how long do unreviewed strings remain pending)
  • Self-approval of managers
  • Self-approval of translators
    But are we sure we can reuse these existing scripts for this?

Therefore, the pending information we'd like is for:

  • Time to graduate to new role
  • Number of managers/translators reviewing strings

Please let us know if you have any questions or concerns about this. There are a lot of things we've already covered in the doc, but I realize there may be other pending questions now that we're kicking this off for realz.
thanks :)

Thanks! Bumping priority to reflect this is a current quarter bug.

Priority: P3 → P2

(In reply to Delphine Lebédel [:delphine - use need info] from comment #0)

  • Time to graduate to new role
  • Number of managers/translators reviewing strings

The scripts are ready:
https://gist.github.com/mathjazz/269a072fb3b42b4640c94015f017cd39
https://gist.github.com/mathjazz/731e05a8f88f47c9c868e617f3a19bf8

Flod, could you please take it for a spin?

Flags: needinfo?(francesco.lodolo)

Thanks Matjaz, I think they're pretty much there.

For both scripts: would it be possible to get the list of all Pontoon locales in the script, and run it on all of them? We might not run the script like that, but it would be useful to have that capability, and hard-code the locales  as needed.

Time to graduate

  • Would it be possible to limit the output to only include promotions happened after a certain date?
  • It would be good to display the date when the change happened.
  • I see some weird date (negative numbers). The time format is also confusing, we probably need some formatting to make it readable (rounding to the day would be enough).
ace,rachmatwahidi.site@XXX,ace managers,-11 days, 19:16:16.687758

We might be only interested in promotions to translator, but the current script is OK for that, we can filter the CSV afterwards.

List of reviewers

  • Number of Approved/Rejected Suggestions: that's the number of someone else's suggestions they approved/suggested, correct?
Flags: needinfo?(francesco.lodolo)

Thanks for the feedback. I've updated the scripts to fulfill your requests. Let me know if things still don't work as expected.

(In reply to Francesco Lodolo [:flod] from comment #3)

  • Number of Approved/Rejected Suggestions: that's the number of someone else's suggestions they approved/suggested, correct?

Correct.

Thanks, they look great. There's only one thing to clean (if possible) on the graduation report.

It looks like I can limit the results to "full graduation" (i.e. not promotion to project translator) by using group__name__iexact instead of group__name__endswith. Is that the right way to do it?

I noticed that we report users multiple time in the graduation report. Take for example Icelandic:

is,almar@XXX,2018-08-08,is translators,47
is,almar@XXX,2018-08-08,common-voice/is translators,47
is,almar@XXX,2018-08-08,is translators,47
is,k@XXX,2018-08-08,common-voice/is translators,1201
is,kgm@XXX,2018-08-08,common-voice/is translators,164
is,vrioniavas@XXX,2019-01-12,is translators,130
is,vrioniavas@XXX,2019-01-12,common-voice/is translators,130
is,almar@XXX,2019-01-12,firefox-for-android/is translators,204
is,vrioniavas@XXX,2019-01-12,firefox-for-android/is translators,130

I came up with some sort of filter, but wondering if there's a cleaner way to do it.
https://github.com/flodolo/scripts/commit/751d6d6df33052e31f394d16938c0fa52bfd717c

Lgtm. You can also just use ‘group=locale.translators_group’ if you don’t care about promotions to project translators.

Duplicates probably mean user has been promoted, demoted and promoted again. Note that you will only catch first promotion using hashes.

Thanks, I think we can close. I'm probably going to put the duplicate check beyond a condition, hopefully are not getting promoted/demoted constantly.

Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Assignee: nobody → m
You need to log in before you can comment on or make changes to this bug.