Closed Bug 1583523 Opened 5 years ago Closed 5 years ago

[Translate.Next] Enable release sampling

Categories

(Webtools Graveyard :: Pontoon, enhancement, P3)

Product:
Webtools Graveyard
Component:
Pontoon
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: adrian, Assigned: adrian)

Details

(Whiteboard: tn-release)

This bug covers writing code to allow us to transparently release Translate.Next to a given, and variable, percentage of our users.

Some questions need to be answered:

  • do we allow users to opt out of Translate.Next?
  • do we allow all users to opt in?
  • do we show a message to users signaling that they're using Translate.Next?

Good news: we're currently using a flag to enable Translate.Next, and that already supports shipping to a percentage of users. No development needed there!

We still need to answer the 3 questions in the first comment though. Opinions, folks?

Assignee: nobody → adrian

(In reply to Adrian Gaudebert [:adrian] from comment #0)

  • do we allow users to opt out of Translate.Next?

Yes

  • do we allow all users to opt in?

Yes

  • do we show a message to users signaling that they're using Translate.Next?

Yes, with instructions on how to report bugs.

After talking with Matjaz, we decided that we should go the easy road. That means there will be no programmed opt-in or opt-out options in the app.

If the waffle flag is on for a user, they get Translate.Next, otherwise they get Translate.Current.

The flag will be on, at first, for all superusers, for all users who currently have Translate.Next activated via their user settings, as well as for a random 10% of all users. We're planning on increasing that number slowly until we get to 100%, barring we don't find any blocking regression.

We will show a message to users in the form of a link to an external piece of communication (likely a blog post on the l10n blog), so that users who discover Translate.Next will be able to quickly get the information they need about it.

Flod, I now see that you disagree with this, mind expanding on it? Do you think we could do without in-app opt-in / opt-out? Note that it is not possible with waffle to specifically disable the feature for a user.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED

Woops, didn't mean to close just yet.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

We talked about two ways of shipping Translate.Next gradually:

  1. Remove Translate.Next switch, force Translate.Next to a limited % of users and increase the % gradually.
  2. Enable Translate.Next switch for everyone, i.e. users need to manually enable Translate.Next.

We picked 1), because testing population size is more predictable and can be gradually increased.

(In reply to Adrian Gaudebert [:adrian] from comment #3)

Flod, I now see that you disagree with this, mind expanding on it? Do you think we could do without in-app opt-in / opt-out? Note that it is not possible with waffle to specifically disable the feature for a user.

I'm OK with not having an opt-in, but I think we should have an opt-out, and measure how many people do. Our testing population was too small to safely assume we won't find blocker bugs for some users. Without an opt-out, that means they won't be able to use Pontoon at all. What's the alternative for them?

Adrian will correct me if I'm wrong, but I would assume the plan is to rollback Translate.Next if we hit any critical bugs (even if they only affect the subset of the population).

(In reply to Matjaz Horvat [:mathjazz] from comment #7)

I would assume the plan is to rollback Translate.Next if we hit any critical bugs (even if they only affect the subset of the population).

And that, at least to me, doesn't seem optimal. If we hit a couple of those, it would be a confusing back and forth for users.

I'm also not sure how links would work in that scenario? There are at least a couple of issues with the /translate URLs:

  • When I enter TN, I can't go back with browser history.
  • I can't share a link as is. If the other person is not using TN, it will just be a broken link, and I've been there a ton of times already.

The other concern is that not everyone is familiar with Bugzilla, or how to report a bug. If we add an opt-out link:

  1. We should make sure it's clear that they won't get TN again until it's deployed to everyone.
  2. We could ask them to fill a survey, asking why they're disabling TN, explaining why it's important, and an email to follow-up. That has a much lower entry barrier compared to filing a bug.

(In reply to Francesco Lodolo [:flod] from comment #8)

(In reply to Matjaz Horvat [:mathjazz] from comment #7)

I would assume the plan is to rollback Translate.Next if we hit any critical bugs (even if they only affect the subset of the population).

And that, at least to me, doesn't seem optimal. If we hit a couple of those, it would be a confusing back and forth for users.

Right, although it will be confusing anyhow I'm afraid: in case of critical bugs we'd have to force everyone back to Translate.Current, and it might be even more surprising to have a manual switch which suddenly gets triggered automatically.

Bug 1583774 should help reduce the confusion a little and the fact that even we sometimes don't know what version are we on is a feature in this case I believe.

I'm also not sure how links would work in that scenario? There are at least a couple of issues with the /translate URLs:

  • When I enter TN, I can't go back with browser history.
  • I can't share a link as is. If the other person is not using TN, it will just be a broken link, and I've been there a ton of times already.

Could you elaborate on the browser history issue?

Both link variants should work in both versions of the app:
https://pontoon.mozilla.org/sl/firefox/all-resources/?string=74127
https://pontoon.mozilla.org/translate/sl/firefox/all-resources/?string=74127

You need to be logged in for the latter though. Question for Adrian, maybe can get rid of the /translate/ piece?

The other concern is that not everyone is familiar with Bugzilla, or how to report a bug. If we add an opt-out link:

  1. We should make sure it's clear that they won't get TN again until it's deployed to everyone.
  2. We could ask them to fill a survey, asking why they're disabling TN, explaining why it's important, and an email to follow-up. That has a much lower entry barrier compared to filing a bug.

Good point. We can also make the message we'll show to users lead to a feedback form.

Could you elaborate on the browser history issue?

With TN enabled go here
https://pontoon.mozilla.org/sl/firefox/

Then click on any of the resources. Then try to go back with the back button.

IIRC, I didn't file a bug after talking with Adrian, because that's a side-effect of the current scenario.

Both link variants should work in both versions of the app:
https://pontoon.mozilla.org/sl/firefox/all-resources/?string=74127
https://pontoon.mozilla.org/translate/sl/firefox/all-resources/?string=74127

The second works only if you're opted in to TN. If you're not logged in, or logged in with TN, you get the OOPS page.

(In reply to Francesco Lodolo [:flod] from comment #10)

Could you elaborate on the browser history issue?

With TN enabled go here
https://pontoon.mozilla.org/sl/firefox/

Then click on any of the resources. Then try to go back with the back button.

IIRC, I didn't file a bug after talking with Adrian, because that's a side-effect of the current scenario.

Sorry, that is actually a bug, and I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1584223. I'm going to fix that very quickly.

Both link variants should work in both versions of the app:
https://pontoon.mozilla.org/sl/firefox/all-resources/?string=74127
https://pontoon.mozilla.org/translate/sl/firefox/all-resources/?string=74127

The second works only if you're opted in to TN. If you're not logged in, or logged in with TN, you get the OOPS page.

Yes, there's another bug in that if you hit /translate/ it doesn't even try to redirect to current. I've filed https://bugzilla.mozilla.org/show_bug.cgi?id=1584226 about it and it should be another easy fix too.

(In reply to Francesco Lodolo [:flod] from comment #8)

(In reply to Matjaz Horvat [:mathjazz] from comment #7)

I would assume the plan is to rollback Translate.Next if we hit any critical bugs (even if they only affect the subset of the population).

And that, at least to me, doesn't seem optimal. If we hit a couple of those, it would be a confusing back and forth for users.

It will indeed be confusing, to some extent (because not that much changes between the versions), but I believe we have ways to mitigate this. We have various communication channels we can use to inform users (discourse, dev-10n mailing list, blog) and if needed we can also use in-app notifications.

The other concern is that not everyone is familiar with Bugzilla, or how to report a bug. If we add an opt-out link:

  1. We should make sure it's clear that they won't get TN again until it's deployed to everyone.
  2. We could ask them to fill a survey, asking why they're disabling TN, explaining why it's important, and an email to follow-up. That has a much lower entry barrier compared to filing a bug.

I agree that bugzilla is a terrible tool, and that's why we've been using discourse for the previous rounds of testing. Can we maybe use that again?

I think you are very right in what you're saying flod, but all of these things will demand a lot more work from us. I don't think the risk of shipping Translate.Next is high enough the justify doing so much to mitigate potential issues.

Regarding gathering feedback from users, I think we should simply propose that people give it using comments on the blog post we're going to publish.

With this, I think we've answered all our questions. Thus closing, on to releasing!

Status: REOPENED → RESOLVED
Closed: 5 years ago → 5 years ago
Resolution: --- → FIXED
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.