1464156 - Fluent shouldn't automatically fall back to en-US when some files are missing for a context

Status: NEW

(Core :: Internationalization: Localization, enhancement, P2)

Description

[Francesco Lodolo [:flod]]

In case of Preferences in 62, we're including over 10 files. If one of them is missing, we're automatically falling back to English (en-US), risking to drop hundreds of localized strings in the process.

We should try to display as much localized content as possible, as long as the performance hit is acceptable.

Comment 1

[Francesco Lodolo [:flod]]

For the record, I just got it by this bug: we added a new file in bug 1497694, and now preferences for Italian are falling back entirely to English (9 missing vs several hundreds translated). I was going to file a new bug, then remembered about this one.

Comment 2

[Axel Hecht [:Pike]]

I'm trying to wrap my head around what we actually want the l10n registry to do. Pathological example:

	foo.ftl	bar.ftl	baz.ftl	qux.ftl
distro	X			X
app	X		?
platform	X	X		X

baz.ftl may or may not exist, for the sake of understanding the expected behaviour, let's start with "it exists".

The first bundle is easy, I think:

(distro, foo), (platform, bar), (app, baz), (distro, qux)

Second and third get a little less clear to me. And to prove my point, let's assume that either foo or qux would be a customized branding file. You want all messages in the second bundle to still refer to the customized branding, right?

OTH, if those were all just regular files without message or term references, there would be no reason to generate another bundle with either foo of qux from distro.

I don't think we can optimize for the latter reliably?

So then the second would be

(app, foo), (platform, bar), (app, baz), (distro, qux)

And then the third could be either of

(app, foo), (platform, bar), (app, baz), (platform, qux)
(platform, foo), (platform, bar), (app, baz), (distro, qux)

I should really mock this as a test case and see what the current code does. But I'd also like to know what we want the code to do.

Comment 3

[Axel Hecht [:Pike]]

Attachment: Bug 1464156, more extensive tests

Comment 4

[Axel Hecht [:Pike]]

Attachment: Bug 1464156, move CachedIterable to L10nRegistry

Depends on D30024

Comment 5

[Axel Hecht [:Pike]]

Attachment: Bug 1464156, refactor bundle generation

Depends on D30026

Comment 6

[Axel Hecht [:Pike]]

Attachment: Bug 1464156, yield bundles even if a resource is missing completely

The code path is a bit trickier for async, as that returns promises that
will resolve to false for missing resources.
The complexity here balances with the intent to have parallel IO.
For sync IO, we can rely on fetchFile return false on missing resources,
so we don't need to retry on just existing resources.

Depends on D30028

Comment 8

[Axel Hecht [:Pike]]

Attachment: Bug 1464156, sync and async bundle generator

Depends on D30024

Comment 9

[Axel Hecht [:Pike]]

I've done a couple of attempts here, but I have to fold this. At the point of trying to integrate changes from the registry into the localization classes, I realized that I have no idea what is important and what's not, and I can't start the complete perf testing from scratch.

Comment 10

[Francesco Lodolo [:flod]]

We're getting this more and more, as we migrate content on startup, and me jumping to run migrations simply doesn't scale, also considering the lack of replacement in case I'm not available.

What are the chances to actually scope and prioritize this?

Comment 11

[Zibi Braniecki [:zbraniecki][:gandalf]]

With bug 1660392 stabilizing soon, I would like to start discussing possible paths forward for this.

I filed an issue against l10nregistry-rs to discuss possible avenues - https://github.com/zbraniecki/l10nregistry-rs/issues/8

Comment 12

[Zibi Braniecki [:zbraniecki][:gandalf]]

The original idea for this bug has been to address it by heuristics (see https://github.com/zbraniecki/l10nregistry-rs/issues/8).

But recently introduced by Erik optional resource (bug 1743659) concept should address the most common source of this problem, which is a recently added resource.
I suggest we use optional resource for new resources added to large contexts (such as Preferences) for a couple of releases, until all locales carry it, and then switch the optional to false.

There's still a potential value in introducing heuristics for deciding when a bundle should be produced, but using memoization and algorithms seems more nuanced and I'd suggest to wait a bit to see how optional resource + process solves the issue.

If the problem prevails, I suggest to reopen this bug.

Comment 13

[Francesco Lodolo [:flod]]

(In reply to Zibi Braniecki [:zbraniecki][:gandalf] from comment #12)

I suggest we use optional resource for new resources added to large contexts (such as Preferences) for a couple of releases, until all locales carry it, and then switch the optional to false.

I don't think this process scales: it relies on developers's discretion to mark resources as optional, and requires work to clean up afterwards. And even cleaning up is not trivial: when do you stop marking a resource as optional? Some locales might not catch up for multiple releases, and their product still be almost fully localized.

BTW, I'm out of office, and I'm quite sure that preferences in Nightly are currently in English because of bug 1744466.

Comment 14

[Zibi Braniecki [:zbraniecki][:gandalf]]

I don't think this process scales:

The one proposed in this bug is based on heuristics. Heuristic approach is likely to be more discrete, less explicit, giving nobody control over the experience and likely resulting in nuanced differences of experience continuously trying to balance between covering one edge scenario and another.

optional allows code maintainers, and l10n-drivers decide which resources are allowed to be missing, which has the benefit of minimizing surprises. I'd appreciate if we at least tried to use it before we dive into attempting a significantly more complex and complicated solutions that result in wider range of user experiences and potential regressions.

If you see another solution to the problem space, please share.

Comment 15

[Francesco Lodolo [:flod]]

(In reply to Zibi Braniecki [:zbraniecki][:gandalf] from comment #14)

I'd appreciate if we at least tried to use it before we dive into attempting a significantly more complex and complicated solutions that result in wider range of user experiences and potential regressions.

I don't mind this approach, but I don't believe that's reflected in closing this bug as duplicate of bug 1743659.

This bug would be fixed by an automated solution, it's now become lower priority, and it will be either fixed or wontfixed.

Comment 16

[Zibi Braniecki [:zbraniecki][:gandalf]]

Ah, makes sense. Ok! Reopening.

The target for optional imho should be first:

• New files added to Preferences
• Any emergency files added to ESR/late-Beta etc.

Comment 20

[Francesco Lodolo [:flod]]

I didn't realize this wasn't actually reopened after comment 16.

This is very much a bug that we need to fix, and severity has increased because of bug 1752111.

I talked with Eemeli, and he should be able to look into it after July. Tentatively assigning it to him.

Comment 22

[Eemeli Aro [:eemeli]]

Attachment: WIP: Bug 1464156 - Make L10n files default to optional rather than required