458444 - [silme] Add API for merging objects/entitylists/packages using multilocale

Status: NEW

(Localization Infrastructure and Tools :: Silme, defect)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

We should be able to merge several packages or objects into one using different locale and diff between them.

In result l10nObject/l10nPackage would become a sort of a grid with entities on vertical axe and locales on horizontal.

In result it should be easy to handle on l10nPackage with several locales inside and diff between two locales in one l10nPackage or one locale in one l10nPackage and another in different l10nPackage.

This may be extremely useful for clean code in apps that will have to work with several locales of the same project/version.

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

These three checkins are related to this bug:

http://hg.mozilla.org/users/zbraniecki_mozilla.com/silme/rev/f396acd206c9
http://hg.mozilla.org/users/zbraniecki_mozilla.com/silme/rev/4f4089ce85c6
http://hg.mozilla.org/users/zbraniecki_mozilla.com/silme/rev/a88a7e399f19

What they give us is an API for multilocale operations:

l10nPackage = silme.io.Manager.get('file').getL10nPackage(path, code='ab-CD')

and:

l10nObject = L10nObject()
l10nObject.id = 'foo.dtd'
l10nObject.setFallback(['pl', 'en-US'])
entity = Entity('test.id')
entity.setValue('value2', 'en-US')
entity.setValue('value1', 'pl')
entity.setValue('value3', 'sw-TZ')

l10nObject.addEntity(entity)

print l10nObject.getValue('test.id') # without setFallback it will return first added lang - en-US
print l10nObject.getValue('test.id', 'pl') # it will return 'pl' or raise KeyError
print l10nObject.getValue('test.id', ['pl', 'en-US']) # it will return pl or en-US if no pl available

l10nObject2 = l10nObject.getLocale(['en-US', 'sw-TZ'])

print l10nObject.getValue('test.id') # it will return en-US cause it uses fallback line from l10nObject and no 'pl' available

Now, I'll add ability to merge two l10nPackages/l10nObjects by locales

Comment 2

[Zibi Braniecki [:zbraniecki][:gandalf]]

next rev. http://hg.mozilla.org/users/zbraniecki_mozilla.com/silme/rev/e769688e911a

This adds *.mergeWith (except of L10nPackage)

It means that it's now possible to merge two l10nObject or two entities or two entityLists.

l10nObject.mergeWith(l10nObject2)

Comment 3

[Zibi Braniecki [:zbraniecki][:gandalf]]

two new revisions:

http://hg.mozilla.org/users/zbraniecki_mozilla.com/silme/rev/8d4cebcffb97

This one is really neat. I'm proud of it.

It makes parser note the fallback and it makes serializer accept fallback.

And it's really powerfull with the next revision:

http://hg.mozilla.org/users/zbraniecki_mozilla.com/silme/rev/c915cd0f6fc7

L10nPackage.mergeWith(l10nPackage)

What it all gives us?

  io = silme.io.Manager.get('file')
  l10nPackage = io.getL10nPackage('./tests/swtz/3.1/en-US/browser/locales/en-US', code='en-US')
  l10nPackage2 = io.getL10nPackage('./tests/swtz/3.1/sw-TZ/browser/locales/sw-TZ', code='sw-TZ')
  l10nPackage.mergeWith(l10nPackage2)
  
  l10nObject = l10nPackage.getElement('chrome/browser/browser.dtd')
  print silme.format.dtd.Serializer.serialize(l10nObject, fallback=['sw-TZ', 'en-US'])

And we're getting l10n-merge! The result l10nObject is sw-TZ where possible, and en-US where falled back to en-US :D

Comment 4

[Zibi Braniecki [:zbraniecki][:gandalf]]

This marks planned feature set for this bug except of silme.core.diff part.

We now have *.mergeWith API that should work nice. There may be some rough corners, so I'm not closing this bug until other people test it.
There will be probably some bugs and things I did think of...

Like, I'm extending entities, but I'm overriding source Objects. Does it make sense? I hope so... :)

Anyway, feel free to test it.

Comment 5

[Zibi Braniecki [:zbraniecki][:gandalf]]

one more minor change: http://hg.mozilla.org/users/zbraniecki_mozilla.com/silme/rev/172d9662676f

changed *.mergeWith to *.merge - simpler, cleaner, PEP08, foxy ;)

Comment 6

[Zibi Braniecki [:zbraniecki][:gandalf]]

this bug currently is only waiting for bug 458445 to be fixed.

Comment 7

[Zibi Braniecki [:zbraniecki][:gandalf]]

added L10nPackage.getLocales() and cleaned code a bit.

For now, I tend to believe it's adding too much complexity to build a full featured locale diffing (thing like l10nPackage.diff(l10nPackage2, code='pl', code='en-US')) because it requires a stable logic behind it.

The API and data structures are prepared, you can do io.getL10nPackage(path, code='pl') and the whole package will have entities in 'pl'. The only problem is that when you want to compare it to a package that has entities in 'en-US' or 'de', you have to do entity.diff(entity2) and understand what does it mean to compare two different locales and what to save?

For example, if I'm diffing two l10nPackages with different locales and the diff should contain a package in state 'added', should it be a package with only one language or all of them?
I think so, because then I can have a diff between pl and de, and a diff between en and ca, and apply BOTH...
But this requires some thinking and I don't think it's blocking us from being useful, so I'm tending to consider this solved, and file a new bug on adding diff() between two locales.

Comment 8

[Zibi Braniecki [:zbraniecki][:gandalf]]

it's partially fixed. And what's in is sufficient for initial release.
I'm taking it off the blockers list, and it'll remain open for the full fix to come.

Comment 9

[Francesco Lodolo [:flod]]

Moving Silme bugs to its component.

[Mass change filter: silme-move]