Closed Bug 665379 Opened 15 years ago Closed 13 years ago

Spanish strings in some Verbatim projects suddenly disappear.

Categories

(Webtools Graveyard :: Verbatim, defect)

Product:
Webtools Graveyard
Component:
Verbatim
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: inma.barrios, Unassigned)

References

(
URL
)

Details

(Whiteboard: [better verbatim])

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 Build Identifier: Spanish translated/suggested strings in the Webify project have suddenly disappeared. You can have a look here https://localize.mozilla.org/projects/webifyme/ Unfortunately, this is not the first time it has happened to us. As far as we know, nobody tried to update from vcs or commit. Reproducible: Sometimes
Just point out that we can't continue translating if this is happening from time to time and we lose all suggestions since we are trying to attract a lot of new contributors in localization doing these suggestions.
Status: UNCONFIRMED → NEW
Ever confirmed: true
I wonder if there could be an easy way to determine if translation loss happens with translations suggested but not sent (nor committed), with translations sent but not committed, or both. This would help us to minimize data loss. This is a question not so much for localizers as for the Mozilla team for Verbatim.
(In reply to comment #1) > Just point out that we can't continue translating if this is happening from > time to time and we lose all suggestions since we are trying to attract a > lot of new contributors in localization doing these suggestions. From time to time means? 1) Its happened before, 2) This is the first time but you don't want it to happen again? (In reply to comment #2) > I wonder if there could be an easy way to determine if translation loss > happens with translations suggested but not sent (nor committed), with > translations sent but not committed, or both. This would help us to minimize > data loss. This is a question not so much for localizers as for the Mozilla > team for Verbatim. Looks to me like the database was recreated. I see Verbatim has been upgraded to Pootle 2.1.6. There are migration scripts as part of Pootle upgrades between versions. So either is a migration bug or the needed libraries weren't installed. Quite a few other people have migrated to 2.1.6 so my first guess is missing tools during the migration.
(In reply to comment #3) > From time to time means? 1) Its happened before, 2) This is the first time > but you don't want it to happen again? It's happened before, at least two times (as far as we know) in other projects.
This might seem like a silly question. But are the people making these translations/suggestion even logged in? If you are logged in your should have submit and suggest buttons. If not logged in all they are doing is moving from one string to another. You can set the permissions for the nobody (anonymous) user to allow them to make suggestions to circumvent this. I'm also concerned that there is confusion about what is a translation and what is a suggestion. Without a little bit more information on how to reproduce this is hard to debug. And just to be clear. Suggestions go into a database. Translations will be in a file and if these are ever pushed to VC they will be in the target file and in that case not lost.
AFAIK, they are logged in and they are doing suggestions, then we have a group of people with submit permissions who review these suggestions. Inma can confirm this, since she was the person who detected the missing strings this time.
(In reply to comment #5) > This might seem like a silly question. But are the people making these > translations/suggestion even logged in? If you are logged in your should > have submit and suggest buttons. If not logged in all they are doing is > moving from one string to another. You can set the permissions for the > nobody (anonymous) user to allow them to make suggestions to circumvent this. Obviously the people who translate o suggest translations do log in. How else can strings be translated? > > I'm also concerned that there is confusion about what is a translation and > what is a suggestion. > We're just trying to simplify things. For instance, in Webify we had a number of suggested strings and all of them have disappeared. However, in SUMO, there were both, suggested and translated strings. As far as I know, translation loss has never happened if strings have been pushed to VC.
(In reply to comment #7) > (In reply to comment #5) > > This might seem like a silly question. But are the people making these > > translations/suggestion even logged in? If you are logged in your should > > have submit and suggest buttons. If not logged in all they are doing is > > moving from one string to another. You can set the permissions for the > > nobody (anonymous) user to allow them to make suggestions to circumvent this. > > Obviously the people who translate o suggest translations do log in. How > else can strings be translated? If you don't log in the forward and back buttons are still enabled and the text area is editable. So you could get the impression that you are submitting suggestions. > > > > I'm also concerned that there is confusion about what is a translation and > > what is a suggestion. > > > We're just trying to simplify things. For instance, in Webify we had a > number of suggested strings and all of them have disappeared. However, in > SUMO, there were both, suggested and translated strings. > > As far as I know, translation loss has never happened if strings have been > pushed to VC. Once you push to VC the strings are taken from the Pootle database and put into the on disk PO files. So in that case you wouldn't lose translations. But your suggestions are still in the database. So the act of pushing to VC makes a physical translation file which in this case would have acted as a backup. Simply downloading the PO file for offline translation would do the same thing.
The problem is that we show "XXX strings to review" and they suddenly disappeared, so the strings are being suggested correctly and for some reason disappear from Verbatim before we could review them (submit) and commit to VCS.
(In reply to comment #8) > (In reply to comment #7) > > (In reply to comment #5) > > > This might seem like a silly question. But are the people making these > > > translations/suggestion even logged in? (...) > > > > Obviously the people who translate o suggest translations do log in. How > > else can strings be translated? > > If you don't log in the forward and back buttons are still enabled and the > text area is editable. So you could get the impression that you are > submitting suggestions. Dwayne, as far as we know, every people contributing to Spanish has an account and is logged in. Anyway, I think our complaint comes from having a Verbatim Project (like Webify) with a lot of suggestions waiting to be accepted, and some already accepted but not committed, and then suddenly all of them disappearing. > Once you push to VC the strings are taken from the Pootle database and put > into the on disk PO files. So in that case you wouldn't lose translations. > But your suggestions are still in the database. So the act of pushing to VC > makes a physical translation file which in this case would have acted as a > backup. Simply downloading the PO file for offline translation would do the > same thing. Yeah, but that breaks the multiuser advantage of Verbatim. :-) It seems clear that the problem lies in database records disappearing or getting disconnected. Since in PO files the "key" is the English phrase, could it be that small corrections in English phrases make the suggestions get lost? For instance, let's assume we have this phrase: - English: "We're committed to preserve the open Web" - Suggestion for Spanish: "Estamos comprometidos a preservar la Web abierta" and someone in Mozilla realizes apostrophes should be escaped and that there is a missing dot. So it changes the English phrase to: - English: "We're committed to preserve the open Web." I wonder if Verbatim then loses the suggestion for Spanish, since the old key no longer exists. Actually, this can't possibly be the reason for massive data loss, but I thought it is an scenario that should be covered. For the massive data loss, something wrong is happening with either translation project updates or with Verbatim updates itself.
(In reply to comment #9) > The problem is that we show "XXX strings to review" and they suddenly > disappeared, so the strings are being suggested correctly and for some > reason disappear from Verbatim before we could review them (submit) and > commit to VCS. Where do you see that "XXX strings to review"? Do you mean "XXX strings need attention"? The later only cover fuzzy and untranslated the former allows review of suggestions. Where you should be in the "Review" tab and then step through the hassuggestion check. Or is that where you where?
(In reply to comment #10) > (In reply to comment #8) > > (In reply to comment #7) > > > (In reply to comment #5) > > > > This might seem like a silly question. But are the people making these > > > > translations/suggestion even logged in? (...) > > > > > > Obviously the people who translate o suggest translations do log in. How > > > else can strings be translated? > > > > If you don't log in the forward and back buttons are still enabled and the > > text area is editable. So you could get the impression that you are > > submitting suggestions. > > > Dwayne, as far as we know, every people contributing to Spanish has an > account and is logged in. Anyway, I think our complaint comes from having a > Verbatim Project (like Webify) with a lot of suggestions waiting to be > accepted, and some already accepted but not committed, and then suddenly all > of them disappearing. And my comment and questions come from trying to get more clarity on exactly what happened and how. Since I write Pootle software but don't maintain Mozilla's instance I'm also in the dark on one question which is is this a result of a Verbatim upgrade. So, where are the Verbatim people? > > Once you push to VC the strings are taken from the Pootle database and put > > into the on disk PO files. So in that case you wouldn't lose translations. > > But your suggestions are still in the database. So the act of pushing to VC > > makes a physical translation file which in this case would have acted as a > > backup. Simply downloading the PO file for offline translation would do the > > same thing. > > > Yeah, but that breaks the multiuser advantage of Verbatim. :-) At some stage you do have to decide on translations to push back to the project! > It seems clear that the problem lies in database records disappearing or > getting disconnected. Since in PO files the "key" is the English phrase, > could it be that small corrections in English phrases make the suggestions > get lost? For instance, let's assume we have this phrase: > > - English: "We're committed to preserve the open Web" > - Suggestion for Spanish: "Estamos comprometidos a preservar la Web abierta" > > and someone in Mozilla realizes apostrophes should be escaped and that there > is a missing dot. So it changes the English phrase to: > > - English: "We're committed to preserve the open Web." > > I wonder if Verbatim then loses the suggestion for Spanish, since the old > key no longer exists. Actually, this can't possibly be the reason for > massive data loss, but I thought it is an scenario that should be covered. That is one area where we work very hard to preserve things. Existing translations and fuzzy translations are preserved in any update of the source text, I'm not sure what happens with suggestions, will need to ask the other Pootle devs. > For the massive data loss, something wrong is happening with either > translation project updates or with Verbatim updates itself. I looked at the news logs and can't see any upgrade info for the month of June for the site. https://localize.mozilla.org/notices/rss.xml But there is a notice that the Webify Spanish locale was created on 16 June. https://localize.mozilla.org/projects/webifyme/notices So I'm not sure if this project is older then that and was recreated by the project manager. I'm about at the end of my diagnosis ability without any input form a Verbatim admin.
Dwayne, Ricardo may be right about the small corrections in the English strings: they could be the cause for data loss, or at least, ONE of the causes. I mean, yesterday I translated again a few strings of the Facebook tab Webhero. The project had been completed and the app launched a few weeks ago. However, suddenly there were strings that needed translating again. I logged in (of course :-) ) and I realised that some of them included URLs that had been changed for a variable, like this <a href='%s' target='_top'>
(In reply to comment #12) > And my comment and questions come from trying to get more clarity on exactly > what happened and how. Since I write Pootle software but don't maintain > Mozilla's instance I'm also in the dark on one question which is is this a > result of a Verbatim upgrade. If feel inclined to think it is, but we all agree this problem is hard to isolate/reproduce and thus to debug. > At some stage you do have to decide on translations to push back to the > project! Let me explain our workflow: - We have users without approve nor commit rights, that just make suggestions. Currently, they account for more than 50% of the translation work, I'd say. - We have some users with approve rights, but not commit, that send translations and approve the suggestions from others. - We have a few users (Nukeador and me among them) with approve and commit rights, that send the approved translations to the VCS. We're trying now to send the approved translations to VCS frequently to minimize the risk of data loss. We don't always succeed, though, as you have noted. :-) > I'm about at the end of my diagnosis ability without any input form a > Verbatim admin. We appreciate your efforts, thanks a lot. In the end, we're all trying to diagnose the problem not to recover our past work, but to avoid other teams (and ourselves) suffering the same problem in the future.
Another example of this, Dwayne: I was just revising some strings in Webhero to document my previous answer and I found some of them have been modified, like this one https://localize.mozilla.org/es/webhero/LC_MESSAGES/messages.po/translate/?unit=986306 Overnight! I am quite sure that the Spanish string was localized (the URLs pointing to a couple of videos in YouTube channel and two SUMO articles covering the Firefox installation in both PC and Mac. In addition to this, in the Spanish version of the tab, the text appears unstranlated http://www.facebook.com/mozillahispano?sk=app_176093342441461 I also noticed some other changes, all of them related to URLs, like this one https://localize.mozilla.org/es/webhero/LC_MESSAGES/messages.po/translate/?unit=986310 I hope it helps to diagnose the problem!
Adding Stas and Gandalf to check it out.
(In reply to comment #13) > Dwayne, Ricardo may be right about the small corrections in the English > strings: they could be the cause for data loss, or at least, ONE of the > causes. > > I mean, yesterday I translated again a few strings of the Facebook tab > Webhero. The project had been completed and the app launched a few weeks > ago. However, suddenly there were strings that needed translating again. I > logged in (of course :-) ) and I realised that some of them included URLs > that had been changed for a variable, like this <a href='%s' target='_top'> Yes that's normal. Changed source text should result in a translation going fuzzy and needed to be corrected. I'm waiting for some Pootle devs in another timezone to come online. But my guess is that in that case you will lose suggestions since the source text (key) has changed. But approved translations, even those not in VC should not have been lost. If the PO update happens outside of Pootle we are dependent on the fuzzy settings that the admin used.
(In reply to comment #15) > Another example of this, Dwayne: I was just revising some strings in Webhero > to document my previous answer and I found some of them have been modified, > like this one > https://localize.mozilla.org/es/webhero/LC_MESSAGES/messages.po/translate/ > ?unit=986306 Overnight! I'm not worried about source text being modified. That's Pootle's power, it allows the source to change while keeping your translation. It then flags it as fuzzy and you go and review and make the needed changes. My concern is what did that look like when you visited it? Was it fuzzy? At the moment it seems to be an approved translation but I don't think its correct since the number e.g. 1MB are missing. > I am quite sure that the Spanish string was localized (the URLs pointing to > a couple of videos in YouTube channel and two SUMO articles covering the > Firefox installation in both PC and Mac. In addition to this, in the Spanish > version of the tab, the text appears unstranlated > http://www.facebook.com/mozillahispano?sk=app_176093342441461 If your translation is not approved in Pootle then it most likely won't be used. So anything that goes fuzzy needs to be approved again to get into the final product. That seems to indicate to me that that string did go fuzzy. > I also noticed some other changes, all of them related to URLs, like this > one > https://localize.mozilla.org/es/webhero/LC_MESSAGES/messages.po/translate/ > ?unit=986310 > > I hope it helps to diagnose the problem! Not yet, those seem to be examples of valid source text changes. My concern, which you can validate for me, is did they go fuzzy.
Some went fuzzy and some others went untranslated. I can't draw a conclusion or set a rule about in what cases or conditions these strings went one state or the other. Sorry I can't be more specific about this! It occurs to me that we could start translating Webify, watch the translation process closely and try to find out a pattern (if any) that would explain this data loss.
(In reply to comment #15) > Another example of this, Dwayne: I was just revising some strings in Webhero > to document my previous answer and I found some of them have been modified, > like this one > https://localize.mozilla.org/es/webhero/LC_MESSAGES/messages.po/translate/ > ?unit=986306 Overnight! > > I am quite sure that the Spanish string was localized (the URLs pointing to > a couple of videos in YouTube channel and two SUMO articles covering the > Firefox installation in both PC and Mac. In addition to this, in the Spanish > version of the tab, the text appears unstranlated > http://www.facebook.com/mozillahispano?sk=app_176093342441461 > > I also noticed some other changes, all of them related to URLs, like this > one > https://localize.mozilla.org/es/webhero/LC_MESSAGES/messages.po/translate/ > ?unit=986310 This is unrelated, I made these changes because otherwise the website was broken. See r90935 and r91089. If there are %s placeables in the original string, there need to be the same number of %s placeables in the transaltions, otherwise PHP breaks :(
(In reply to comment #5) > ... If not logged in all they are doing is moving from one string to another... .... but unfortunately the text field for translation is not off-limits for entries, so one can translate until cows come home and keep losing it all the time without knowing. Of course in the current bug this does not seem to be the case but I want to bring it up again, As nothing has changed since afaik, I entered this bug https://bugzilla.mozilla.org/show_bug.cgi?id=667599 - maybe it is a duplicate. regards smno
Whiteboard: [better verbatim]
I've reread the bug comments. I don't think this is at all related bug 667599 as suggested in comment 21. We'll be deploying a new version of Pootle shortly. I'm going to close this bug, you can reopen or file a new bug if the problem reappears in the new Pootle.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.