1509079 - Update to Fluent Syntax 0.8

Status: RESOLVED FIXED

(Webtools Graveyard :: Pontoon, enhancement, P2)

Description

[Staś Małolepszy :stas]

The implementations are not ready yet, but I'd like to start the conversation early. Syntax 0.8 is planned to be the last major milestone before 1.0. I hope to release it by the end of November or in early December.

The (rather lengthy) changelog is available at https://github.com/projectfluent/fluent/blob/392a01d802cc820884736fe626f7215b8fc13f75/spec/CHANGELOG.md#unreleased.

Comment 1

[Staś Małolepszy :stas]

One big addition in Syntax 0.8 are macros. They were designed in https://github.com/projectfluent/fluent/issues/176 to supersede variant lists. (Vvariant lists are still supported in 0.8, but they are deprecated and will be removed in 1.0.) With macros, term values are always guaranteed to be patterns again. Furthermore, they are guaranteed to have a selector.

We'll need to manually migrate a few strings from variant lists to macros. I'm aware of at least 4 affected locales: it, fr, ru, uk. Here's an example of a transformation we'll need to perform, taken from https://hg.mozilla.org/l10n-central/fr/file/default/browser/browser/branding/sync-brand.ftl#l11.

# Deprecated syntax using a VariantList
-fxaccount-brand-name =
    {
        [lowercase] compte Firefox
       *[uppercase] Compte Firefox
    }

# New syntax using a regular Pattern
-fxaccount-brand-name =
    { $letter-case ->
        [lower] compte Firefox
       *[upper] Compte Firefox
    }

Matjaž, do you think we'll need any UI changes to the rich editor to accommodate macros? Arguably they are closer to regular Patterns now, so I'd expect to be able to use the UI for regular select expressions. I see that right now it's not possible to edit the selector via the rich edit UI, however. For example in https://pontoon.mozilla.org/fr/firefox/browser/browser/preferences/preferences.ftl/?search=download-choose-folder&string=176219 I can only see the PLATFORM() selector once I toggle the source editor. Can we use the same approach for macros? In other words, can we rely on the source editor for cases when the user wishes to edit (or add) the selector? If so, I think the current UI in https://pontoon.mozilla.org/fr/firefox/browser/browser/branding/sync-brand.ftl/?string=177742 would be sufficient.

Comment 2

[Matjaz Horvat [:mathjazz]]

Thanks for the heads-up, stas. I looked into the changelog and identified the following requred changes:

> Don't store the - sigil in Identifiers of Terms. (#142)

We'll need to slightly change the way we stringify TermReference (to be more like VariableReferences, not MessageReferences):
https://github.com/mozilla/pontoon/blob/bfb3199d835d2eb33f19fef4ae2d21f14e248bfd/pontoon/base/static/js/fluent_interface.js#L408
https://github.com/mozilla/pontoon/blob/bfb3199d835d2eb33f19fef4ae2d21f14e248bfd/pontoon/base/templatetags/helpers.py#L328

> Introduce parameterized Terms. (#176, #212)

I agree with your observation that we don't need rich editor UI changes for this particular proposal.

> Transformation of VariantLists to Macros.

In the past we used migrations for that, but it became a burden over time:
https://github.com/mozilla/pontoon/pull/1119

So we either need to be more careful with the migration code or simply run a one-off command.

Sounds like we'll also need to transform references to these strings (from VariantExpression to CallExpressions(?)).

Comment 3

[Staś Małolepszy :stas]

(In reply to Matjaz Horvat [:mathjazz] from comment #2)

> We'll need to slightly change the way we stringify TermReference (to be more
> like VariableReferences, not MessageReferences):

Good point, thanks for bringing it up. It looks like it should be a fairly simple change in the code.

> I agree with your observation that we don't need rich editor UI changes for
> this particular proposal.

Great to hear, thanks!

> > Transformation of VariantLists to Macros.
> 
> In the past we used migrations for that, but it became a burden over time:
> https://github.com/mozilla/pontoon/pull/1119
> 
> So we either need to be more careful with the migration code or simply run a
> one-off command.

VariantLists cannot be transformed to Macros automatically, so we'll need to ask localizers to update their translations manually (or do it for them). That's because macros require the selector and it's up to the localizer to choose the name for it, depending on what it actually selects (grammatical declensions, letter case, etc.). The good news is that Syntax 0.8 supports both VariantLists and Macros, so timing is not as important as it used to be in earlier updates. I think it should be enough to:

1) update Pontoon to Fluent Syntax 0.8,
2) manually move localizations away from VariantLists,
3) (later) update Pontoon to Fluent Syntax 1.0 which will remove the support for VariantLists.

> Sounds like we'll also need to transform references to these strings (from
> VariantExpression to CallExpressions(?)).

Yes, that's a good point as well. I think it needs to happen manually, too.

My take away from this discussion is that we don't need any UI changes to support 0.8 in Pontoon. Can you confirm? I'll be happy to help with the changes in fluent_interface and helpers, if time allows.

Comment 4

[Matjaz Horvat [:mathjazz]]

Oh, right - good point about not being able to automatically transform VariantLists and VariantExpressions.

The plan you outlined SGTM. Confirming that no UI changes are needed. As part of Step 3) we can also get rid of this:
https://github.com/mozilla/pontoon/blob/master/pontoon/base/static/js/fluent_interface.js#L8

I'm not sure though how does the AST of a macro look like (compared to a select expression), and if that requires any changes.

Comment 5

[Staś Małolepszy :stas]

(In reply to Matjaz Horvat [:mathjazz] from comment #4)

> The plan you outlined SGTM. Confirming that no UI changes are needed. As
> part of Step 3) we can also get rid of this:
> https://github.com/mozilla/pontoon/blob/master/pontoon/base/static/js/
> fluent_interface.js#L8

Yes, totally :)

> I'm not sure though how does the AST of a macro look like (compared to a
> select expression), and if that requires any changes.

No changes are needed. It's just a regular Pattern with SelectExpressions. Macros are a runtime solution. The only difference in the AST is that CallExpressions can now have TermReferences or AttributeExpressions on TermReference as callees, too (in addition to FunctionReferences), but serializing that will be handled by FluentSerializer.serializeExpression.

Comment 6

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/pontoon/pull/1141

Comment 7

[[github robot]]

Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/6885c8bf3d1d484d1d3e342db9187cbd4d381996
Fix bug 1509079: Update fluent libraries to support fluent syntax 0.8 (#1141)

* JS part
- update lib
- use `fluentSerializer.serializeExpression()` for all expression types but `SelectExpression`
- use official Fluent element expression type names in tooltips (no space)
- add space around expressions within {}

* Python part
- update lib
- use `fluentSerializer.serialize_expression()` for all expression types but `SelectExpression`
- use isinstance