1357499 - [Pontoon] Add support for peer reviewing model

Status: RESOLVED MOVED

(Webtools Graveyard :: Pontoon, enhancement, P3)

Description

[Tomer Cohen :tomer]

Currently, there are three roles in Pontoon: 
* Translator - can suggest translations
* Moderator - A translator the can also approve translations and submit translations without any review.
* Manager - A moderator that can also update user roles.

A good translation model is to make sure that each string will be submitted after it has a consensus, which can be implemented by forcing everyone to submit suggestions (as a translator) and never submit unreviewed translations (as a moderator/manager).

The current way of doing so is to ask every moderator to change their preferences to suggestions instead of submission, but there is no way to enforce it. I'd like a better way to enforce this, probably by a toggle button that can be switched on the project management page, or as a separated role. 

If a project consist of three reviewers, each could submit suggestions but approve translation made only by the other two reviewers. This way at least two people will pay attention to each string, which will make sure the quality is better than when there is no reviewing step.

There should be kept in mind few use cases that could break this model: 
* Small teams that doesn't have enough people to review - higher roles should be able to submit translations without review when in lack of reviewers. Personally speaking, I am approving my own suggestions few hours or days after I submit the suggestion, so I have to review my own changes and make sure that there are no embarrassing typos etc.
* This model can't work when in urgent - There should be kept an option to submit translations with no review when in hurry. 
* This model can't work for new projects with a lot of strings - Since reviewing is time consuming task, when people decide starting to translate Firefox from scratch, they may prefer to submit translations with no reviewing step to speed up the task.

Comment 1

[Matjaz Horvat [:mathjazz]]

This is a dupe of bug 1350686. But since it's so well speced, I'm closing bug 1350686.

Comment 2

[Francesco Lodolo [:flod]]

I think you didn't do what you originally planned.

Comment 4

[Djfe]

Some thoughts of mine on this:
Crowdin separates this into two views.
Proofreading and translating

Though as a moderator you automatically approve a string if you save it in translation mode.
But the proofreading mode uses the available screen space better for proof reading.

You could add proofreader as a role (above moderator maybe)

The last string by a moderator+ should be the one chosen as the string. (even if it wasn't reviewed, yet.)
Reviewing could be treated as some kind of flag. -> don't touch anymore, this string is good and works fine, rather add translations to other strings than looking at this one again. (prevents translators from adding translations; I remember from projects on Crowdin, people are interested in contributing and start going through the file from start to finish, so they start with established strings like the app's name ^^, which of course had to do with the fact that I was fast with adding translations, but I would've liked those people to look at my last suggestions and correct/improve those instead)
If they still come across one of those strings (search or so and notice a spelling mistake), then we could leave a button "notify reviewer" with a comment field, so the reviewer can take a look at it at some point.

If moderators+ add translations to reviewed strings, then the flag "reviewed" stays on. But a flag "updated" will be set to true, so a reviewer can look at it again.
That updated flag needs to be reset afterwards to false (obviously)

Do we want an optional comment field for string suggestions? (especially useful for already reviewed strings)

Reviewed and maybe some other flags could be removed from the overview for lower roles (less crowded overview)

reviewed strings could make the progress bar go to 200% or something like that. (100% ready to ship, 200% we're confident in those translations)

If a project has no account with a reviewer role, then they just don't use the feature.

If there are several unreviewed translators for one string or updates, then those should be prioritized in the reviewer view(s).

What also should be done: Improve the way Pontoon pushes to GitHub.
Example: Common Voice
The git history is crowded with translation commits.
Maybe add a push button to the project or leave it to the reviewer to push them into the repo.
Maybe even better: allizom could be set up in away to retrieve the strings directly from Pontoon on updates (without committing to the dev tree on github). The devs could pull the strings everytime they push a new version of the website to Mozilla.org (or do a manual push)
maybe the translations could also reside in a submodule or something like that.
(I know this sounds very specific to common voice, but I don't have another perspective. If this is just due to the way Common Voice implemented it, then it's fine. But other projects commit history are big enough already (Firefox))

Add an "edit/update string" button, so you don't have to delete your old string and other can compare previous versions of your string. (maybe an optional comment field for changes); this should still update the update flag for reviewers etc.

another thing that bugs me currently:
I get notifications: new strings in project x.
I get into the project. Someone else translated them already. The strings aren't only at the end of the file but also in between. Sorted by the dev in the string file I suppose (likely by string id, so category wise it should be fitting).
Atm. I have to use the time/date filter to find them. sorting by added last/updated last could be helpful.
Though it would remove it from the surrounding strings that it belonged to. (just like when strings got translated already; you have to look up manually with machinery/search what other strings belong to the one you're currently translating, if the other ones got translated already)
it would be nice if pontoon somehow would display strings with similar ids, that have translations already, in the window as well, so you can compare.
Or a way to link several strings together and show and translate them together (like singular and plural or the counting stuff; zero, one, few, many)
Though Online translation on a webpage (wysiwyg) will always be superior to this.
My way of thinking is probably to complex here, if you don't get what I mean, just ask.
If reviewing will be separated from approval (or no approval at all) in the future, then this might not even be to big of an issue anymore. (at least if you limit the reviewers to one person per project)
Maybe simpler: not sort, but show strings with new/updated translations first in the list (higher priority, but sorted the same way; lower priority than untranslated strings)


In general I feel like we need some way to communicate in Pontoon (asynchronous and locally (at the string, where everyone can see it), if more than one person is translating a project.
Maybe one comment section global and one for the language.
Global would be for general discussions about context of a string and links to code/github whatever, stuff that isn't in the string description, yet. Maybe a link to a page that explains how something should be formatted.


Allow (only) moderators and managers to edit string descriptions (what about reviewers, do they even need it?)
I feel like mods work across several languages, reviewers mostly on their native language (maybe more than one)
people can be manager in one project, mods in several projects or a mod for language (dunno) and reviewers are per language per project. At least that's my feeling of how I could/should be. Do you agree? How is it currently?


I know this comment isn't very clean and it lists lots of stuff that isn't really part of roles. But I feel like it belongs to the discussion. If you agree, that something of my comment should be implemented, then we can still open a bug report for it later.
In-case this is already on your agenda: I don't have an overview where you are going with Pontoon in the future, so my perspective is limited to the user/mod on the live version. sry ^^

Comment 5

[Francesco Lodolo [:flod]]

There's a lot in your comment, and some things are definitely interesting and worth discussing in details (e.g. being able to see strings translated for which you received a notification).

I would suggest to limit the discussion on the topic of the bug though (enabling peer review), noting that we plan to address the feedback cycle as a whole in the coming months, and ideas are more than welcome.

It might make sense to split your thoughts up in smaller topics, otherwise it becomes really hard to follow the conversation.
We have a Discourse section to talk about Pontoon, and it might be the best place to do that before filing actionable bugs.
https://discourse.mozilla.org/c/pontoon

Comment 6

[BMO Automation]

*This bug has been moved to GitHub.*

*Please check it out on https://github.com/mozilla/pontoon/issues.*