1591290 - Request for data from Pontoon to evaluate Manager engagement

Status: RESOLVED FIXED

(Webtools Graveyard :: Pontoon, task, P2)

Description

[Delphine Lebédel [:delphine - Use needinfo]]

(This is directly related to our current OKR work for this quarter: https://docs.google.com/document/d/1ccPhSCuzj_eDTAFu8wNRM293EH0rhAXwrqLQVkilM-c/edit?ts=5db1e2a0)

Matjaz: we'd like to get some data out of Pontoon for the following items discussed in our OKR doc linked above.

We've decided that these 5 data points are the most relevant to get out of this exercise at this point:

• Time to graduate to new role
• Time to approve/reject unreviewed strings (aka, how long do unreviewed strings remain pending)
• Self-approval of managers
• Self-approval of translators
• Number of managers/translators reviewing strings

From what we've gathered, we have existing scripts (https://github.com/flodolo/scripts/tree/master/pontoon) for three of these already:

• Time to approve/reject unreviewed strings (aka, how long do unreviewed strings remain pending)
• Self-approval of managers
• Self-approval of translators
  But are we sure we can reuse these existing scripts for this?

Therefore, the pending information we'd like is for:

• Time to graduate to new role
• Number of managers/translators reviewing strings

Please let us know if you have any questions or concerns about this. There are a lot of things we've already covered in the doc, but I realize there may be other pending questions now that we're kicking this off for realz.
thanks :)

Comment 1

[Matjaz Horvat [:mathjazz]]

Thanks! Bumping priority to reflect this is a current quarter bug.

Comment 2

[Matjaz Horvat [:mathjazz]]

(In reply to Delphine Lebédel [:delphine - use need info] from comment #0)

• Time to graduate to new role
• Number of managers/translators reviewing strings

The scripts are ready:
https://gist.github.com/mathjazz/269a072fb3b42b4640c94015f017cd39
https://gist.github.com/mathjazz/731e05a8f88f47c9c868e617f3a19bf8

Flod, could you please take it for a spin?

Comment 3

[Francesco Lodolo [:flod]]

Thanks Matjaz, I think they're pretty much there.

For both scripts: would it be possible to get the list of all Pontoon locales in the script, and run it on all of them? We might not run the script like that, but it would be useful to have that capability, and hard-code the locales  as needed.

Time to graduate

• Would it be possible to limit the output to only include promotions happened after a certain date?
• It would be good to display the date when the change happened.
• I see some weird date (negative numbers). The time format is also confusing, we probably need some formatting to make it readable (rounding to the day would be enough).

ace,rachmatwahidi.site@XXX,ace managers,-11 days, 19:16:16.687758

We might be only interested in promotions to translator, but the current script is OK for that, we can filter the CSV afterwards.

List of reviewers

• Number of Approved/Rejected Suggestions: that's the number of someone else's suggestions they approved/suggested, correct?

Comment 4

[Matjaz Horvat [:mathjazz]]

Thanks for the feedback. I've updated the scripts to fulfill your requests. Let me know if things still don't work as expected.

(In reply to Francesco Lodolo [:flod] from comment #3)

• Number of Approved/Rejected Suggestions: that's the number of someone else's suggestions they approved/suggested, correct?

Correct.

Comment 5

[Francesco Lodolo [:flod]]

Thanks, they look great. There's only one thing to clean (if possible) on the graduation report.

It looks like I can limit the results to "full graduation" (i.e. not promotion to project translator) by using group__name__iexact instead of group__name__endswith. Is that the right way to do it?

I noticed that we report users multiple time in the graduation report. Take for example Icelandic:

is,almar@XXX,2018-08-08,is translators,47
is,almar@XXX,2018-08-08,common-voice/is translators,47
is,almar@XXX,2018-08-08,is translators,47
is,k@XXX,2018-08-08,common-voice/is translators,1201
is,kgm@XXX,2018-08-08,common-voice/is translators,164
is,vrioniavas@XXX,2019-01-12,is translators,130
is,vrioniavas@XXX,2019-01-12,common-voice/is translators,130
is,almar@XXX,2019-01-12,firefox-for-android/is translators,204
is,vrioniavas@XXX,2019-01-12,firefox-for-android/is translators,130

I came up with some sort of filter, but wondering if there's a cleaner way to do it.
https://github.com/flodolo/scripts/commit/751d6d6df33052e31f394d16938c0fa52bfd717c

Comment 6

[Matjaz Horvat [:mathjazz]]

Lgtm. You can also just use ‘group=locale.translators_group’ if you don’t care about promotions to project translators.

Duplicates probably mean user has been promoted, demoted and promoted again. Note that you will only catch first promotion using hashes.

Comment 7

[Francesco Lodolo [:flod]]

Thanks, I think we can close. I'm probably going to put the duplicate check beyond a condition, hopefully are not getting promoted/demoted constantly.