Closed Bug 394105 Opened 17 years ago Closed 15 years ago

[l10n] localize Gecko chrome strings on the trunk

Categories

(Camino Graveyard :: General, defect)

Product:
Camino Graveyard
Component:
General
Platform:
All
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Camino2.0

People

(Reporter: alqahira, Assigned: alqahira)

References

Details

(Keywords: l12y, Whiteboard: l10n [camino-2.0])

Attachments

(4 files, 1 obsolete file)

step 1 [landed]
1.35 KB, patch
mikepinkerton
: superreview+
Details | Diff | Splinter Review
step 2 (properties, less automated) [landed]
220.85 KB, patch
stuart.morgan+bugzilla
: superreview+
Details | Diff | Splinter Review
partial fix [landed]
2.07 KB, patch
mikepinkerton
: superreview+
Details | Diff | Splinter Review
Removes unused strings.in files, adds new used strings.in file
116.96 KB, patch
stuart.morgan+bugzilla
: superreview+
Details | Diff | Splinter Review
Now that bug 248160 has landed, we have the ability to override Gecko chrome strings on the trunk. We now have several dozen Gecko .strings/.properties/.dtd files we need to somehow incorporate into Cocoa .strings files in Camino so that l10n can then translate them. Ideally, I guess, we'd want to automatically convert each Gecko file into a matching Cocoa file in the build system (so we always get current strings and don't have to mess with maintenance, except for ones we have to override)? I'm really not clear how kreeger's code is going to work, though.
(In reply to comment #0) > don't have to mess with maintenance, except for ones we have to override)? These "ones we have to override" being the ones we already override in embed-replacements to un-break ourselves, etc. We want to use this new mechanism for any Gecko strings that we, or Gecko on our behalf, call, which is a bunch of files Embed.jar (though maybe not all of them; there might be SeaMonkey-specific chrome strings in there).
Summary: [l10n] localize Gecko chrome strings → [l10n] localize Gecko chrome strings on the trunk
Attached patch step 1 [landed] — — Splinter Review
Here's the first step, which is to do the lookup in a file based on the chrome file that is being accessed; we need to do something to guarantee that keys are unique (since they may not be across different chrome files), and this way makes bookkeeping--and potential auto-generation--much easier. As an example, with this patch the localizations for chrome://necko/locale/necko.properties would all be in a file called necko_locale_necko_properties.strings
Attachment #315909 - Flags: superreview?(mikepinkerton)
The next step would probably be to, as Smokey suggested, make a script that passes over our (post-replacements) jar contents, and auto-generates all the .strings files in the build directory, so that we never have to deal with those files. Also, it doesn't seem from my (admittedly minimal) testing that this system covers .dtd files; I'm just seeing .properties lookups--kreeger, do you know if your override work was expected to handle both?
Comment on attachment 315909 [details] [diff] [review] step 1 [landed] + tableName = [tableName substringFromIndex:9]; can you make a small comment about where the 9 comes from? sr=pink
Attachment #315909 - Flags: superreview?(mikepinkerton) → superreview+
Comment on attachment 315909 [details] [diff] [review] step 1 [landed] Landed on trunk. Now we need to add some automatic .properties -> .string build magic.
Attachment #315909 - Attachment description: step 1 → step 1 [landed]
When we do the magic, we’ll have to be watchful for straight double-quotes. In an ideal world we’d run the processing through mento's brain, and he'd be able to intelligently replace any starting " with “ and any ending " with ” and we’d live happily ever after, but in the real world I think we’ll just have to mass \-escape every " we find.
(In reply to comment #6) > When we do the magic, we’ll have to be watchful for straight double-quotes. > In an ideal world we’d run the processing through mento's brain, and he'd be > able to intelligently replace any starting " with “ and any ending " with ” > and we’d live happily ever after, but in the real world I think we’ll just > have to mass \-escape every " we find. Can't that magic just be a grep for the characters you want to avoid (in the shell script), and if they're found, stop with an error code? The tboxes might even go red from that.
Why on earth would we want the build to fail any time any core code had " in any of their localization files (which is always)?
Sorry, I think I misunderstood what this was for. :-) But anyway, sed should able to help you, I think?
[12:06am] ardissone: kreeger: what about dtd files? [12:07am] kreeger: dtd files are fun? [12:07am] ardissone: does your code not handle them? [12:07am] kreeger: the localized string override thing? [12:07am] ardissone: yeah [12:08am] kreeger: no [12:08am] kreeger: oh [12:08am] kreeger: um for lookup? [12:08am] ardissone: i don't know [12:08am] kreeger: because it just get's the key from the string-bundle-service and then asks our component if we have an override for it [12:09am] kreeger: i would imagine it works for .properties and DTD... [12:09am] ardissone: smorgan could only get it working for .properties [12:09am] kreeger: I think we have some DTD's in Songbird.... let me check [12:10am] kreeger: eh no just .properties [12:10am] sauron: https://bugzilla.mozilla.org/show_bug.cgi?id=394105#c3 [12:10am] thebot: sauron: Bug 394105 nor, --, Camino2.0, nobody@mozilla.org, NEW, [l10n] localize Gecko chrome strings on the trunk [12:10am] kreeger: let me look at |nsStringBundle| [12:13am] kreeger: er [12:13am] kreeger: ardissone: I think |nsIStringBundle| only reades .properties [12:13am] kreeger: it'll probably take a core patch to wire up to whoever reads the DTD files... [12:14am] ardissone: awesome :/ [12:14am] encro left the chat room. (Quit: Low slung dancefloor dynamics) [12:15am] ardissone: can someone just take Gecko out back behind the barn and shoot it? [12:15am] kreeger: ardissone: is there a specific DTD file we are having trouble with? [12:16am] ardissone: all of them? [12:16am] kreeger: doh [12:16am] kreeger: i'm guessing the old string stuff like the prompts are DTD? [12:16am] kreeger: oh no [12:16am] kreeger: I've seen those keys come through [12:16am] ardissone: no, those are properties [12:16am] ardissone: but stuff in netError are dtd, for example [12:17am] kreeger: ahh [12:17am] kreeger: for the no-site-exists XUL pages and sucH? [12:17am] ardissone: mhm [12:17am] ardissone: config.dtd [12:17am] ardissone: for about :config [12:17am] ardissone: DTDs are for xhtml-type content, i guess [12:18am] kreeger: yeah [12:18am] kreeger: those suck for us [12:20am] kreeger: if the key gets mapped as a RDF XML source inside gecko - that might be quite a problem for us :( [12:22am] kreeger: <!ENTITY nssFailure2.title "Secure Connection Failed"> [12:22am] kreeger: can be easily mapped as <label value="&nssFailure2.title" ... [12:23am] kreeger: which I assume runs through XUL somewhere to make the lookup - but it's not easily findable ATM [12:25am] philor: I think the lookup is just what it is in any XML using any DTD - Expat parses the XUL, reads the DTD, and replaces entities with their values [12:25am] kreeger: yeah i bet [12:25am] sauron: "yay" :/ [12:26am] philor: so I don't know what you could do to change what's used for &nssFailure2.title; other than intercepting the chrome: URL resolution for the DTD, and sending it off to get a different (but entire, not just one entry) file [12:39am] sauron: someone should comment in 394105, at any rate Phil's last comment almost implies we should be able to do this (although we'd have to use actual .dtd files instead of .strings files, but we could ship them in .lprojs still) if the way the override system works is "intercept chrome:// url calls"; is it?
Whiteboard: l10n
Having thought for a minute after duping that bug... we may want to open a new bug (blocking this one) for finding a solution for loading localized copies of DTD strings, since we are bug 248160 really only did half of what it was intended to cover.
Taking, per the meeting. In the short term, we'll look at manually converting the Gecko .properties files into Cocoa .strings files.
Assignee: nobody → alqahira
Commenting here instead of 457290. I never understood the whole story about Camino and gecko strings. Is there some reference for the Cocoa files, and a rationale for why you need what there? The reason for asking here and not in the other bug is that for the rewriting part, you might find silme helpful, which should be in a rather stable state soon. http://diary.braniecki.net/tag/silme/ has more.
It's not the files/file formats that are the issue; they're more of an implementation detail in this story. Instead, it's a conceptual and architectural difference about how you ship localized applications. On Mac OS X, applications are multilingual; all available localizations ship in a single package and the OS determines what localization to load at runtime. Beyond that significant architectural difference, Cocoa localization is broadly similar to Gecko: localized resources are stored in the $langCode.lproj folders (instead of $langCode.jar) and UI strings live in UTF-16-encoded .strings files (instead of UTF-8-encoded .properties and .dtd files). The .strings and .properties files are so similar that I can use four regexp rules to convert the contents of properties files into Cocoa .strings files, save fixing the straight quotes (I could do it in fewer if the comment and spacing style were more consistent in the .properties files, or if my regexp-fu were stronger). In order for us to ship a fully-localized multilingual Camino, we need to be able to ship a Gecko localization for each corresponding Camino l10n, and we need to be able to make a runtime determination to have Gecko use the Gecko localization that corresponds to the Camino l10n in use. There's no way (at least not with the embedding system) to ship a multilingual Gecko. (Some of our localizers did figure out how to create a bilingual Embed.jar which sort-of worked, but that still fails the multilingual test.) We can, however, use the string bundle override to bridge the strings in .properties files, so that when Gecko wants to display the string for, say the "Connecting to foo" network status, it asks Camino if we've overridden that string, and we in then make a Cocoa call that returns the version of that string from the .strings file in the active localization. But the .dtd files don't work like the .properties files (the parser just pulls in the file), so we have bug 457290. There's not a very good Apple overview to localization (at least that I'm aware of), but this might help: http://developer.apple.com/documentation/Porting/Conceptual/win32porting/Articles/intern.html#//apple_ref/doc/uid/20002357
Here's the patch to take manually-extracted, manually-renamed (technically, AppleScript-renamed) *_properties.strings.in files and turn them into valid .strings files and add them to the build. Stuart, this is just a bit of Makefile and project additions plus cvs adds, so going straight to you for sr, but if you'd prefer an additional review, that's fine.
Attachment #347228 - Flags: superreview?(stuart.morgan+bugzilla)
Just for reference/the record, below is a skeleton for automating the extraction part. There are two issues with it yet: doing the renames correctly, and reworking the Makefile's libs:: target so that the $(GENERATED_GECKO_STRINGS_FILES) target gets run after the embed-replacements rsync (so that we have the latest/correct files). If we get extraction automated, we may want to rework the $(GENERATED_GECKO_STRINGS_FILES) target anyway and just work on an auto-generated list of files or something, rather than doing the substitution stuff. mkdir -p strings-tmp cd strings-tmp unzip -ou ../../dist/Embed/chrome/embed.jar -d . find . -name "*.properties" cp locale/en-US/communicator/*.properties \ communicator_locale_*_properties.strings.in global/*.properties \ global_locale_*_properties.strings.in global/dom/*.properties \ global_locale_dom_*_properties.strings.in global/layout/*.properties \ global_locale_layout_*_properties.strings.in global/security/*.properties \ global_locale_security_*_properties.strings.in global-region/*.properties \ global-region_locale_*_properties.strings.in necko/*.properties \ necko_locale_*_properties.strings.in pipnss/*.properties \ pipnss_locale_*_properties.strings.in (pax might be able to do this, or perl or ruby; I used AS) cp *.strings.in resources/localized/English.lproj/gecko (Add strings-tmp to GARBAGE_DIRS) (Rework Makefile.in libs:: rule to get things called in the correct order)
Comment on attachment 347228 [details] [diff] [review] step 2 (properties files, half automated) >+ -E -e "s/^([^\"# ]*)( )=( )(.*)$$/\"\1\" = \"\4\";/" \ >+ -E -e "s/^([^\"#= ]*)( *)=( *)(.*)$$/\"\1\" = \"\4\";/" $< | \ Why does the one pattern allow for an equals sign in the first section, and the other not? If it weren't for that, this could be a single pattern, using [[:space:]] around the equals. For the short term, should we just leave your sed-/makefile-fu here for reference for the full solution, run your script once, and check in the results as .strings files? If we are going to have our own copy of everything for the time being, we may as well skip the extra build step.
(In reply to comment #19) > Why does the one pattern allow for an equals sign in the first section, and the > other not? If it weren't for that, this could be a single pattern, using > [[:space:]] around the equals. Probably because the equals sign was added to the second one when I was collapsing the "no-space-between-key-and-equals-and-string" and "variable-spaces-between-key-and-equals-and-string" into a single rule and the tab being a separate group in the other rule in practice separated the key from the equals. There's never going to be an equals sign in the key itself—as I said, weak regexp-fu ;) > For the short term, should we just leave your sed-/makefile-fu here for > reference for the full solution, run your script once, and check in the results > as .strings files? If we are going to have our own copy of everything for the > time being, we may as well skip the extra build step. When we talked on Friday about how/where we should land these, whether to just land final .strings files or to use the same processing system as we did with the existing .strings files, you said "May as well handle them the same way then" ;) I don't have a strong preference either way, though I have become fond of the fact we don't have any faux-binary text files in cvs any more and we don't ever have to be worried about broken byte order when touching a strings file, regardless of how long a solution may last—and of course I've already set it up this way ;-) Besides skipping the build step once per clobber, the other major benefit of the other way is that I can go through and manually fix all the dumb quotes first.
So I did ;) I was just thinking about the UTF-8/16 thing. I guess what I should have said was "check them in a UTF-8 .strings.in files and treat them exactly like our other UTF-8 strings files." I guess it doesn't really save a step exactly, but it saves doing all the substitution transforms over and over, and means we don't have things with ".strings" in the name that aren't even remotely .strings files.
Let me propose something radical: Have you tried to put the ab-CD.jar and .manifest into the lproj dirs, and then use the "ChromeML" directory service hook to point the chrome registry to that dir, and to pick the plain jars up over there? At least for DTDs, you'll have to end up doing something more or less similar. Not that I knew how much of the chrome registry actually lives inside camino.
Comment on attachment 347228 [details] [diff] [review] step 2 (properties files, half automated) Clearing, since at least the "-escaping rule is wrong in this version, and we have the other changes to make, too.
Attachment #347228 - Flags: superreview?(stuart.morgan+bugzilla)
Here's a patch that addresses the discussion in comment 19-21 and, uh, actually works. I had to co-mingle these strings with the other ones in the source tree because I could not convince Xcode to copy them into English.lproj (instead of the main Resources folder) otherwise. I've spot-tested a handful of these again and found that css.properties seems never to be hitting the string bundle override (we never even come up with a tableName according to the logging). So I'm back to wondering how much useless translating is going to be happening here, between strings we don't use and strings where the Gecko versions are bypassing the string bundle override….
Attachment #347228 - Attachment is obsolete: true
Attachment #348453 - Flags: superreview?(stuart.morgan+bugzilla)
Comment on attachment 348453 [details] [diff] [review] step 2 (properties, less automated) [landed] sr=smorgan If some files are being ignored, we can remove them as we find them.
Attachment #348453 - Flags: superreview?(stuart.morgan+bugzilla) → superreview+
Comment on attachment 348453 [details] [diff] [review] step 2 (properties, less automated) [landed] Checked in on cvs trunk. I landed all the files; we can handle removing "unused" files in another bug (or here if we want). Leaving open for finishing this with complete automation of the process, but I'm not going to actively work on that part.
Attachment #348453 - Attachment description: step 2 (properties, less automated) → step 2 (properties, less automated) [landed]
What's left here won't block b1.
Assignee: alqahira → nobody
Flags: camino2.0b1? → camino2.0b1-
Flagging for b4 to track removing the useless strings files; see http://wiki.caminobrowser.org/User:Sardisson/Localizing_Gecko_Strings for the current status of determining which ones are which.
Flags: camino2.0b4?
Taking this bug again to handle removal of the unused files. There are also a number of files that are used by Core code used by Camino, but said code/stringbundles for whatever reason ignore the nsIStringBundleOverride service (i.e., we never get a call for them, as represented by a lack of tableName in the Console), so we can't substitute our own versions (most notably the four security-related properties files; see http://wiki.caminobrowser.org/User:Sardisson/Localizing_Gecko_Strings#File_used_in_Camino_but_calling_code_ignores_nsIStringBundleOverride). Stuart, should I remove the files in that group, too (or not remove the files, but not ship them either, or some other permutation), or do you think you might be able to investigate the problem and find something interesting in a timeframe relevant to 2.0?
Assignee: nobody → alqahira
Status: NEW → ASSIGNED
Flags: camino2.0b4? → camino2.0b4+
Hardware: PowerPC → All
I tracked this down to the fact that some components are caching nsStringBundle instances during their own init, and their init beats nsStringBundleService noticing our override. This is, as Smokey pointed out, a subset of bug 384796; that solution doesn't actually completely work; it takes a while after we register our override for the notification to get delivered, and in the meantime several other services are starting up. This patch fakes a notification to get nsStringBundleService to notice what has happened sooner (by triggering the code kreeger added in bug 384796). Unfortunately even that's not perfect; nsScriptSecurityManager, for example, is inited before we even have a chance to register compontent in our startup, but this patch fixes a bunch of the files that Smokey found to be bypassing our override bundle.
Attachment #400440 - Flags: superreview?(mikepinkerton)
I think comment 30 is the best I think we can do for 2.0, which means I think: communicator_locale_typeaheadfind_properties.strings * and global_locale_security_caps_properties.strings aren't going to happen. I guess axing the files and filing a placeholder for someday re-adding them if we figure out a solution is probably the best approach.
Attachment #400440 - Flags: superreview?(mikepinkerton) → superreview+
Comment on attachment 400440 [details] [diff] [review] partial fix [landed] sr=pink
Comment on attachment 400440 [details] [diff] [review] partial fix [landed] Landed on CVS trunk and CAMINO_2_0_BRANCH.
Attachment #400440 - Attachment description: partial fix → partial fix [landed]
This patch removes 19 files (that are either unused by Camino or which we can't get the override in early enough to override) from GECKO_STRINGS in the Makefile, from the project, and from CVS itself, and adds one file, global-platform_locale_intl_properties.strings.in, that we're now able to override thanks to Stuart's fix.
Attachment #400645 - Flags: superreview?(stuart.morgan+bugzilla)
Comment on attachment 400645 [details] [diff] [review] Removes unused strings.in files, adds new used strings.in file sr=smorgan
Attachment #400645 - Flags: superreview?(stuart.morgan+bugzilla) → superreview+
Comment on attachment 400645 [details] [diff] [review] Removes unused strings.in files, adds new used strings.in file Landed on CVS trunk and CAMINO_2_0_BRANCH. This bug has gotten too long to usefully contain any additional parts of this process, and, in the main, the goal of this bug has been achieved, so resolving FIXED. I spun out the following bugs on the issues left unaddressed: Bug 516640 - Full automation of the .strings generation in the build system (roughly comment 18) Bug 516636 - Getting our override request in in time to be able to override all necessary stringbundles and re-adding the removed files that we use (comment 30-31) Bug 457290 - Figuring out some solution for the annoying .dtd files (comment 10, comment 22) Thanks to everyone for all the work/help on this!
Status: ASSIGNED → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Whiteboard: l10n → l10n [camino-2.0]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: