Closed Bug 623634 Opened 15 years ago Closed 14 years ago

Verbatim exports .po files which can't be compiled

Categories

(Webtools Graveyard :: Verbatim, defect)

Product:
Webtools Graveyard
Component:
Verbatim
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: clouserw, Unassigned)

Details

(Whiteboard: [better verbatim])

When Francesco saves his changes in Verbatim for the Italian AMO .po file, a bunch of deprecated strings are stuck at the end. Normally we wouldn't care, but these are duplicates of other strings in the file and msgfmt won't compile them. Root cause aside, perhaps verbatim should run it's exports through msguniq or something else to ensure it always exports something that can compile.
I have seen this bug before as well, so it doesn't seem to be related to Francesco's files only. I wonder if this is a bug that only appeared with the latest version of Pootle.
I just saw it for the pt_PT locale too, so definitely not just 'it'.
(Alaa's comment from an email, hoping to move the conversation to Bugzilla) ----------- [This] was not reported before. but we did change how obsolete units are handled recently. any insight into sequence of events? was an update from templates (most common way of generating obsolete units) run recently. can you provide links to broken files? cheers, Alaa
> any insight into sequence of events? was an update from templates (most > common way of generating obsolete units) run recently. You can see the history in the SVN log http://viewvc.svn.mozilla.org/vc/addons/trunk/site/app/locale/it/LC_MESSAGES/messages.po?view=log 1) Will merged new strings in r80230 2) I used the "Update all from version control" button to update my locale in Verbatim, translated missing strings and committed in r80247 The problem is at the end of the file. After point 1) this is one of the obsolete strings #~ msgid "Disabled by Admins" #~ msgstr "Disattivato dagli amministratori" After point 2) the same obsolete string has become #: apps/amo/__init__.py:95 msgid "Disabled by Admins" msgstr "Disattivato dagli amministratori"
We've seen this in SUMO a lot, too. I think that the problem is that Verbatim doesn't cope well with us merging the files directly in SVN. Here's what I suspect is happening: 1. A developer updates their code, deleting some strings or changing them significantly (enough so that gettext thinks there was a new string added and the old one was removed; e.g. add a plural form to a previously singular-only string). 2. The developer runs an extraction script, which creates a new POT and then merges existing PO files with it. Seeing deleted strings, msgmerge marks them as obsolete (using #~). This is done as means of a very simple translational memory, as far as I can tell. The developer commits the merged PO files to SVN. 3. A localizer (or sometimes me/Kadir) clicks the "Update from version control" button in Verbatim. The new PO file is checked out from SVN and merged with the old PO file that lived in Verbatim. Here's where things go south: The old file in Verbatim still has the strings that were removed in the new file, and Verbatim's merging mechanics don't remove them (probably to reduce a probability of a translation loss in case of an unwanted removal in the POT). So we end up with a file merged first by msgmerge by the dev (with removed strings marked as obsolete) and then merged again by Verbatim with its previous version kept there (with removed strings being brought back so that data is not lost). Verbatim doesn't remove the obsolete strings though, so the end-product is a file with duplicates. To put this differently, msgmerge does a LEFT OUTER JOIN (where the POT is 'left') and Verbatim does a FULL OUTER JOIN (where the previous version of the file kept in Verbatim is 'right').
I think that one solution would be to run `msgattrib --no-obsolete` in step 2 above, i.e. as a last step of the extraction script. This would remove all obsolete strings from the newly merged PO files before they're committed to the SVN by the developer. The downside is that we'd lose some translation memory for locales who don't use Verbatim. Also, the final PO files committed from Verbatim by the localizers will still have the removed strings. I don't think this has any other consequences than the size of the compiled MO files. This doesn't really solve the problem of Verbatim's merging mechanics, either.
hey stas, thanks for the analysis. Alaa - do you guys have any experience with issues like that hitting other projects that are using pootle/gettext?
Pootle's database should reflect the structure of the last revision of the file after an update from version control operation. on a commit to version control it should not modify the structure of the file but commit it as is. there was a bug in the commit code that allowed strings present in the database but not present in the working copy of the file to be added before commit. This is a mistake because there is no scenario where this is required. however that alone would not explain why all the obsoleted strings where reintroduced in the file in r80247 for that to happen the update step had to have failed somehow. I don't know if we can narrow down the time of this update and check if apache error logs have any tracebacks or warnings. anyway I've committed a fix to the commit half of the problem (will be released in 2.1.5) and continue to test to try and recreate the update half of the problem.
Turns out msgattrib --no-obsolete won't even run if there are duplicate obsolete strings. :) Anyway, I've added deleting all obsolete strings to my merge script and have cleaned them out of all AMO .po files. Can the people having trouble with this verify if that fixes the problem? Edit: I midaired with Alaa, but I think this is still a good thing.
I don't think you should delete all obsolete strings on the migrate script, sure we have to delete them for the currently broken files, but this is a bug and the problem should not repeat once fixed. in fact the I committed already is enough to avoid breaking the files. more importantly the incorrect strings are not the obsolete ones, but rather the duplicate non obsolete ones where incorrectly reintroduced. so by deleting obsolete we don't fix the file we just carry around uneeded strings and present them as translatable.
should I apply it or wait till the release?
(In reply to comment #9) > Turns out msgattrib --no-obsolete won't even run if there are duplicate > obsolete strings. :) Yes, sorry for not mentioning this. I meant this as a possible solution going forward, for the next time we'd merge. It was not, however, a fix to the current compilation problems; that's why you had to clean the dupes. I'm not sure how you did it, but one way is to use `msquniq`. It has its shortcoming, too. It requires localizers to review their files looking for new fuzzy strings (if found duplicates had different translations), or, if used with the `--use-first` option, doesn't produce new fuzzy strings but instead just takes the first occurrence of the duplicated string and removes the other(s). As you can imagine, this can possibly leave an older translation in the file depending on the position of the strings. (FWIW. I think that msguniq is the only tool in gettext's toolchain that can accept files with duplicated strings as arguments.)
since identifying which units should be deleted is not easy it is best to revert to the revision immediatly after merging new strings (r80230) and then use Translate Toolkit's pretranslate tool to migrate translation work done after. this way file structure is returned to the correct state and no translation work is lost.
re release, only think blocking releasing 2.1.5 is identifying why the update failed to delete strings from the database. anything of interest in the apache error log?
unfortunately, the oldest logs we have are from Dec 12, while the issues emerged on Nov 30. From what I see in the logs that we have everything is about: - lack of indexer (ubuntu does not package one) - some random svn update fails maybe 2-3 times throughout the whole interval for one or two locales (rather not related)
(In reply to comment #16) > - lack of indexer (ubuntu does not package one) It provides python-xapian, you should install it for improved performance (install xapian-tools too). http://packages.ubuntu.com/search?keywords=python-xapian&searchon=names&suite=all&section=all
correction - not ubuntu, RHEL is powering the server.
Not sure if it can be useful, but a couple of minutes ago I committed changes from Verbatim to Firefox Input and I see duplicated obsolete strings at the end. http://viewvc.svn.mozilla.org/vc/projects/reporter/locale/it/LC_MESSAGES/messages.po?r1=80918&r2=80927
great what was the sequence of your actions? did you update before commiting? do the error logs mention anything related?
(In reply to comment #20) > great what was the sequence of your actions? did you update before commiting? I found that strings were already updated on Verbatim this time (there were untranslated strings, while the product was completely localized yesterday), so I suppose that someone (fred?) did an "Update from version control" after updating all locales. This morning I completed my translation and committed to VCS.
(In reply to comment #21) > I suppose that someone (fred?) did an "Update from version control" after > updating all locales. Yes, I did!
The current set of errors on the Input project looks like this: ./bg/LC_MESSAGES/messages.po:1182: duplicate message definition... ./bg/LC_MESSAGES/messages.po:1145: ...this is the location of the first definition ./bg/LC_MESSAGES/messages.po:1186: duplicate message definition... ./bg/LC_MESSAGES/messages.po:1157: ...this is the location of the first definition msgfmt: found 2 fatal errors ./it/LC_MESSAGES/messages.po:1232: duplicate message definition... ./it/LC_MESSAGES/messages.po:1175: ...this is the location of the first definition ./it/LC_MESSAGES/messages.po:1235: duplicate message definition... ./it/LC_MESSAGES/messages.po:1187: ...this is the location of the first definition ./it/LC_MESSAGES/messages.po:1238: duplicate message definition... ./it/LC_MESSAGES/messages.po:1175: ...this is the location of the first definition ./it/LC_MESSAGES/messages.po:1241: duplicate message definition... ./it/LC_MESSAGES/messages.po:1187: ...this is the location of the first definition ./it/LC_MESSAGES/messages.po:1245: duplicate message definition... ./it/LC_MESSAGES/messages.po:1175: ...this is the location of the first definition ./it/LC_MESSAGES/messages.po:1249: duplicate message definition... ./it/LC_MESSAGES/messages.po:1187: ...this is the location of the first definition msgfmt: found 6 fatal errors ./zh_TW/LC_MESSAGES/messages.po:1188: duplicate message definition... ./zh_TW/LC_MESSAGES/messages.po:1130: ...this is the location of the first definition ./zh_TW/LC_MESSAGES/messages.po:1192: duplicate message definition... ./zh_TW/LC_MESSAGES/messages.po:1142: ...this is the location of the first definition msgfmt: found 2 fatal errors I'll fix it again, hoping not to trample anyone's translations in the process :(
ok I finally identified why the update broke. committed fixes to Pootle's svn after running a few more tests I will build the 2.1.5 version tomorrow.
adding my 2c: in case of SL addons disconnect, even the file names do not match: SVN shows r-messages.po and verbatim has (still) z-messages.po. status as of 21st jan 2011. Check also my posting in http://groups.google.com/group/mozilla.dev.l10n.web/browse_thread/thread/8b541e7d00bfa235
(In reply to comment #25) > adding my 2c: > in case of SL addons disconnect, even the file names do not match: SVN shows > r-messages.po and verbatim has (still) z-messages.po. > > status as of 21st jan 2011. Check also my posting in > http://groups.google.com/group/mozilla.dev.l10n.web/browse_thread/thread/8b541e7d00bfa235 Getting off topic, but the only files that need localizing are messages.po and z-javascript.po. r-messages and z-messages are built automatically and shouldn't be changed. If there is a way to hide them in verbatim we should do that (in a different bug).
(In reply to comment #26) > r-messages and z-messages are built automatically and > shouldn't be changed. If there is a way to hide them in verbatim we should do > that (in a different bug). Added them to ignore list. should be fixed :)
NOW you tell me :< "r-messages and z-messages are built automatically and shouldn't be changed.If there is a way to hide them in verbatim we should do that (in a different bug)." This is at least careless, if not something worse. Opening a new bug - https://bugzilla.mozilla.org/show_bug.cgi?id=627969
Can someone check my last commit from Verbatim? http://viewvc.svn.mozilla.org/vc/addons/trunk/site/app/locale/it/LC_MESSAGES/messages.po?r1=81049&r2=81391 I updated Verbatim and found quite a few strings to translate. Most of them are placed at the end of the file (as if they weren't in the original file), some of them look like duplicates.
The fix commented in comment 24 has already been released as part of Pootle 2.1.5. See the release announcement: http://markmail.org/thread/emiwtyqctvgybwdg But Verbatim is still running 2.1.4, which should be upgraded.
Verbatim is running Pootle 2.1.5 now. Please, verify if the bug still exists.
Flod, can you please help me reproduce this?
(In reply to Milos Dinic [:Milos] from comment #32) > can you please help me reproduce this? Not really: as far as I'm concerned this bug was solved by that update since I never had that problem in the following months. This summer I switched to off-line localization (SVN+Virtaal) because of the number of strings to translate, so I'm not using Verbatim anymore for AMO.
Whiteboard: [better verbatim]
From my reading of all the comments, it seems that this was fixed with Pootle 2.1.5 which was deployed at the end of January 2011.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.