1146356 - Automate export/import of source and target strings between SVN repos and github iOS project

Status: RESOLVED FIXED

(Firefox for iOS :: General, defect)

Description

[Jeff Beatty [:gueroJeff]]

This is a follow up bug to bugs 113615 and 1136220 to automate the manual pieces of this process.

The current process is as follows:
Export source strings into en.xliff file:
1) Clone the project repo from https://github.com/mozilla/firefox-ios into a local repo.
2) Open up Client.xcodeproj in xcode.
3) With the Client project active (left-hand pane), click Editor->Export for localization.
4) Create a new folder outside of your local repo clone into which the en.xliff file will be exported.
5) Be sure that the "Developer language only" option is selected from the "Include:" dropdown menu.
6) Once exported, change the name of the file to firefox-ios.xliff.

Export new source strings into target language xliff files:
1) Clone the project repo from https://github.com/mozilla/firefox-ios into a local repo.
2) Open up Client.xcodeproj in xcode.
3) With the Client project active (left-hand pane), click Editor->Export for localization.
4) Create a new folder outside of your local repo clone into which the en.xliff file will be exported.
5) Be sure that the "Existing translations" option is selected from the "Include:" dropdown menu and that all locales are selected from the checkbox menu below it. 
6) Once exported, change each file name to firefox-ios.xliff and place it in it's corresponding directory (maybe this should be done away with and xliff files should retain their originally exported names).

Import target language xliff files:
1) Make sure you have updated your local github project clone, carthage, and your SVN locale repos.
2) Open up Client.xcodeproj in xcode.
3) With the Client project active (left-hand pane), click Editor->Import localizations.
4) Navigate to your locale's local SVN directory, select the .xliff file, and click Open.
5) A sheet will appear which displays what will be imported. Click Import.
If this is a new localization, xcode will read in the locale code from the .xliff file's first <file target-language=""> and add it to the list of the project's localizations. If this is an existing localization, it will import the new strings.

In order for localizations to be included in the Fx for iOS builds, the string resources need to be imported into a local xcode project and the project must then be pushed into github. From there, when xcode server kicks off a build, it will do so according to the xcode project's build specifications in the github xcode project, which should have the localizations added to it. 

Export can be automated easily with xcode commandline tools using this command and then commiting the changes to each SVN directory: 

xcodebuild -exportLocalizations -localizationPath <dirpath> -project <projectname> [[-exportLanguage <targetlanguage>]]

Import is cannot be done with xcode commandline tools. A human must manually import each xliff file into their local xcode project and then push those changes into the master branch of the github project.

The cadence for this is a bit tricky when it comes to simultaneous dev & l10n. Dev can only export new strings for translation *after* having first imported new translations into the project and pushed them to github. 

This cadence is simplified once a string freeze is in place. Export happens only once (at string freeze) and throughout the remaining duration of localization, import is the only action needed until the l10n team has tested and signed-off on their work.

Comment 1

[Francesco Lodolo [:flod]]

The more I try and the less I understand what XCode does. 

1. I opened the project in XCode (I also run git reset --hard origin/master;git clean -fd to make sure there were no extra/changed files), and exported localization (only English is available at this point).

Then I compared en.xliff with my file, and removed all extra strings/files. This is the resulting change
http://viewvc.svn.mozilla.org/vc/projects/l10n-misc/trunk/firefox-ios/it/firefox-ios.xliff?r1=139511&r2=139578

2. I then imported the localization in XCode. I didn't get any window to warn me about missing strings.

3. At this point I go to export, and find a bunch of untranslated strings that were not in the English file when I exported it in #1. 

Right now I restored again my previous version
http://viewvc.svn.mozilla.org/vc/projects/l10n-misc/trunk/firefox-ios/it/firefox-ios.xliff?r1=139578&r2=139579

Is someone able to reproduce this and, more important, explain what's going on?

Comment 2

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

I will spend time on this tomorrow to figure out what is going on.

Comment 3

[Jeff Beatty [:gueroJeff]]

I think it makes sense to cancel tomorrow's strings export to localizers so that this can be sorted out. No sense in generating a possibly broken export and exposing it to 31 locales.

Comment 4

[Francesco Lodolo [:flod]]

I think we need to find a way to update localized files soon, especially if we ask people to test l10n in dog-fooding builds.

https://l10n.mozilla-community.org/~flod/webstatus/?product=firefox-ios

Since XLIFF files are "self-contained", this dashboard works on the assumption that all needed strings are inside the file, it doesn't compare each localized file to a reference (en-US). In other words, it currently says that there are 27 complete locales, but there are actually 3, 2 if we don't count en-US.

If needed I can rewrite the system to compare localization to en-US, but it would be better to actually add the missing strings to each localization. Thoughts?

Comment 5

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

I would like to close this bug because I think we got most of this covered with the scripts and workflow we created in the past month. If anyone objects or thinks there is something that needs to be done on top of that work, please re-open.

Comment 6

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Reopened this bug so that we can implement the final step of a more automated way to export strings and update all locales.

Notes from https://l10n.etherpad.mozilla.org/ios-update-whistler

Add new strings to en-US on SVN (current process)
* (st3fan) exports en-US and commit to SVN - https://github.com/mozilla/firefox-ios/blob/master/scripts/export-base.sh
* (flod) copy en-US to /templates and run "clean-xliff.py templates" to remove English translations (file will be used by Pootle)
* (flod) run update-xliff.py to update all locales from en-US (except 'pl')

Comment 7

[Francesco Lodolo [:flod]]

Just for reference this is the current bash script I use

# Move in the repository
cd ~/mozilla/svn/l10n-misc/trunk/firefox-ios

# Copy en-US file in templates
cp en-US/firefox-ios.xliff templates

# Clean up the file in template
/home/flodolo/mozilla/clean-xliff.py templates

# Update other locales (script excludes pl and en-US)
/home/flodolo/mozilla/update-xliff.py .

Comment 8

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Attachment: PR: https://github.com/mozilla/firefox-ios/pull/767/files

This patch automates all the steps needed to export strings.

Comment 9

[Francesco Lodolo [:flod]]

Comment on attachment 8635398 [details]
PR: https://github.com/mozilla/firefox-ios/pull/767/files

It looks good but I can't make it work locally, it fails (timeout error) on both my Macs on

> STATIC_DEPS=true pip install lxml || exit 1 

It might just be my system, but it would be good to double check with someone else.

The diff is always pretty big for Pootle locales (we clean up the messy export), but no need to worry. The best way to double check a diff is to compare 'it' (there shouldn't be changes, unless there are last minute commits with strings) or other locales at 0 strings missing.

Comment 10

[Francesco Lodolo [:flod]]

Reopening, hopefully one last time. I missed that the order of called script was different from the one I suggested. The way it currently is, it adds target-languages="templates" to the template file.

https://github.com/mozilla/firefox-ios/pull/774/commits

Comment 11

[Francesco Lodolo [:flod]]

merged
https://github.com/mozilla/firefox-ios/commit/d82d74a80bba4b49253425a09a8359a92ca4af44