Closed Bug 1219212 Opened 10 years ago Closed 10 years ago

Add possibility to download and upload files for projects

Categories

(Webtools Graveyard :: Pontoon, defect, P2)

Product:
Webtools Graveyard
Component:
Pontoon
Type:
defect

Tracking

(firefox44 affected)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox44 --- affected

People

(Reporter: flod, Unassigned)

References

Details

The possibility to download and upload projects in .po format is a key feature for several localizers. If you can download and upload a .po file with your translation, you can: * Translate offline, or perform QA checks with scripts. * Use Pootle as a translation memory and to commit files to repositories without the need of pull requests. Both features were available on Verbatim. Possibly Pontoon should be a little smarter when uploading a localization: instead of replacing the file, it should parse it and only update the strings that actually change, giving attribution to the uploader. As far as I know that wasn't the case for Verbatim.
On IRC you mentioned a potential conflict with the sync cycle for upload. I guess a simple warning message (e.g. "Upload is not currently available. Try again later") would work, as long as you're able to determine which locale is currently being updated and you don't have to block all locales for the entire sync cycle. I also have no clue how much time this cycle takes.
For the record, Pontoon allows offline localization: http://horv.at/blog/?p=798&preview=true It also performs quality checks and uses amagama (Pootle) as one of the sources for translation memory, but you have to be online for that to work. DOWNLOAD We removed support for downloading source files once we lost persistent storage. We need to figure out if we can generate files on the fly quickly enough or we need persistent storage to implement this feature. UPLOAD While I see the benefit of having an upload feature, in the use case described above Pontoon acts only as a tool for pushing translations to repositories, so making a pull request directly until this is implemented should be as easy as that from the localizers point of view. We're trying to reduce the sync time and it currently takes <30 minutes on average to sync all projects for all locales in both directions. That's without persistent storage, which means we sometimes have to clone instead of pull. Since sync is atomic on a project level, that message would have to be displayed for the entire period the project is syncing and for all locales. Which means often and not neccessarily briefly. What could also happen is that we allow upload, and while uploading and merging, the sync starts. So I think we need to find an alternative solution to this.
Priority: -- → P3
To clarify: offline translation with other tools ;-) I didn't localize much on Verbatim but, still, I found myself using that feature often to work on POEdit (on new projects) or a simple text editor (especially when doing search&replace). As for PRs, they work, with some caveats: * Need to wait for developers to merge. Having done that for Serbian for months, that's just painful. * Developers don't know our localizers, if people start doing PRs they might also accept changes from "unknown" localizers. So, it works, but that's not a long term solution.
I think any solution that involves the website stopping or changing behavior while sync is occurring is a bad idea. Users shouldn't care that the site is syncing, and we shouldn't be blocking people from being productive based on how fast or slow our syncing code is. What if we just treat resource upload as a way for users to bulk-submit translations? We allow users to upload replacement resources, and they get queued for processing by a worker. When the worker processes the uploaded file, it parses out the entities and updates the database as if the user submitted those translations one at a time via the UI. Since the sync process is already built to handle submitted translations during sync, it doesn't matter if a sync is currently happening. Effectively, we're leveraging our ability to parse l10n files to enable bulk translation submission. The problem is if you want users to be able to edit attributes that Pontoon normally doesn't let them change, like comments. I think it'd be good to, at least for now, say that uploading resources is only for bulk-submitting translations so that we don't have to also change that part of Pontoon.
For what my opinion is worth, I simply *cannot* localize online when there are more than one or two strings to translate. It is a time-consuming activity, the connection is not always stable, searches are slow, I cannot search/replace at will; moreover, in Poedit I have all my translation memory. I simply think that all the localization activities will be seriously delayed if this feature is not implemented. Thanks for your attention.
Priority: P3 → P2
(In reply to Francesco Lodolo [:flod] from comment #0) > The possibility to download and upload projects in .po format is a key > feature for several localizers. I'd suggest to additionally add support for .xliff format, which is a standard format for localization.
Quick update: We have a pull request pending for this bug. It's available for all file formats Pontoon supports (po, lang, xliff, dtd, properties ...).
Some needs and ideas as far as I am concerned: 1. A separate commits instead of a mix commit, don't mix unsynchronized changes. 2. Allow additional comment (Max 1000 characters) to VCS commit. 3. Bulk upload files through zip and 7z (Solid compression advantages). 4. A batch management for suggestions, if the upload results incorrect.
Blocks: 1214260
Commits pushed to master at https://github.com/mozilla/pontoon https://github.com/mozilla/pontoon/commit/de89de0e524a7fb5b807d129e093dea8f43005c5 Bug 1219212: Enable download of translated resources https://github.com/mozilla/pontoon/commit/3239d43ebebfca3ebe40bd379e2d74610b09f918 Merge pull request #280 from mathjazz/bug-1219212-download Bug 1219212: Add ability to download translated resources
Commits pushed to master at https://github.com/mozilla/pontoon https://github.com/mozilla/pontoon/commit/8f7a61788f8f1200932ebb850d0cf41ffa4c4930 Fix bug 1219212: Enable UPLOAD of translated resources https://github.com/mozilla/pontoon/commit/e1e6ecb4a93629489d644fe4535af1ab01182b47 Merge pull request #281 from mathjazz/bug-1219212-upload Bug 1219212: Add ability to UPLOAD translated resources
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
This is now deployed, looking forward to your feedback. Use the profile menu on the translate page to download or upload currenlty translated file. All file formats are supported and attribution for uploaded changes is given to the uploader. The feature is only available to users with translators privileges. No fancy features like .zip or drag & drop upload for now. We can focus on that after issues of higher priority are resolved.
How does attribution work for uploads? Should we document that somewhere?
For each new translation, authorship is attributed to the uploader. Suggestions, which become translations if they are the same as in the uploaded file, keep existing authorship attribution and get approval attribution set to the uploader.
Let me reopen this one. I do not think this solution is what users are requesting. I am suggesting that we can upload/download files in .po (and/or .xliff) formats, *regardless of the original format of the file*. That is, if a file is a .properties (= monolingual), such as in https://pontoon.mozilla.org/ca/fundraising/Emails/, Pontoon now offers to download it as a monolingual .properties (which is good and should be left like that as an option), but it is not what I had in mind. I still want to be able to download a .po/.xliff bilingual version of the file. Pontoon knows the source and the target for each string, so it should be able to produce a valid bilingual .po/.xliff file. Simon from comment #6 says: > For what my opinion is worth, I simply *cannot* localize online when there > are more than one or two strings to translate. It is a time-consuming > activity, the connection is not always stable, searches are slow, I cannot > search/replace at will; moreover, in Poedit I have all my translation memory. Poedit requires a .po file; he will not be able to translate if Pontoon offers to download a monolingual .properties file. So this implementation will not work for him either. Finally, the ability to download a whole project (as a ZIP) is a must to run offline QA checks for the whole project. It is of no use if I have to download file by file in a project that might contain hundreds of files scattered around subdirectories.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Jordi Serratosa from comment #15) > Let me reopen this one. > I do not think this solution is what users are requesting. > I am suggesting that we can upload/download files in .po (and/or .xliff) > formats, *regardless of the original format of the file*. > That is, if a file is a .properties (= monolingual), such as in > https://pontoon.mozilla.org/ca/fundraising/Emails/, Pontoon now offers to > download it as a monolingual .properties (which is good and should be left > like that as an option), but it is not what I had in mind. I still want to > be able to download a .po/.xliff bilingual version of the file. Pontoon > knows the source and the target for each string, so it should be able to > produce a valid bilingual .po/.xliff file. > po/xliff compat is going to come to an end at some point, I don't think it makes sense to restrict ourselves here in what l10n file formats can do. Technically, I think that conversion on download/upload as an option is fine, but I'd make that a different bug, and optional. Same for download/upload of packages.
(In reply to Axel Hecht [:Pike] from comment #16) > po/xliff compat is going to come to an end at some point, I don't think it > makes sense to restrict ourselves here in what l10n file formats can do. xliff is alive and kicking and the only hope (so far) for interoperability between CAT (Computer-Assited Translation) tools. > Technically, I think that conversion on download/upload as an option is > fine, but I'd make that a different bug, and optional. I disagree on "make that a different bug". If you see the bug subject: "Add possibility to download and upload .po files for projects". And re-read the flod's initial request (comment #0): > The possibility to download and upload projects in .po format is a key > feature for several localizers. > > If you can download and upload a .po file with your translation, you can: > * Translate offline, or perform QA checks with scripts. > * Use Pootle as a translation memory and to commit files to repositories > without the need of pull requests. I think he is actually asking exactly what I'm suggesting: up/download a **.po** file [regardless of the original format]
Jordi, a bit of background: we'd probably leave this bug intact if it didn't turn out to be a blocker for Verbatim to Pontoon migration for some people. They use Verbatim primarily as a gateway to repositories, to download and upload translations and to avoid making pull requests and leaving it for developers to decide when and if they should merge them or not Looking at Verbatim, we only use it for .po projects (except for one, which uses .properties) and all these projects only use 1 or 2 files (except for one, which uses 4). Sure, there are other projects in Pontoon, which use other file formats and multiple files (e.g. Firefox & Firefox OS), but we also happen to give write access to their repositories, so there's barely a reason to go through Pontoon if you work offline. I agree we should allow you to download .zip of the entire project and maybe also add support for downloading/uploading .po files for all projects, but I also see them as two separate issues. I'm leaving it for :flod to decide if this bug should be closed or not and if the new bug should be filed or not. But in any case, this should no longer block bug 1214260.
As the reporter, I consider this bug fixed: now I'm able to upload and download files in the original format, and work offline without going through pull requests on GitHub. As for comment 6, it's about a project based on .po files (like 99% projects available on Verbatim). I can totally see how your proposal could be useful, especially for monolingual files like .properties, but that's really another bug (and I'm going to file it to avoid a battle of open/close on this one).
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
(In reply to Francesco Lodolo [:flod] from comment #19) > I can totally see how your proposal could be useful, especially for > monolingual files like .properties, but that's really another bug (and I'm > going to file it to avoid a battle of open/close on this one). And that would be bug 1228718.
Last note (sorry for the bugspam): I realized how the subject and first comment could be misleading, but I filed this bug on behalf of my teammates (one is Simon in comment 6) and all of them work only on .po based projects (SUMO, Marketplace, etc.). .properties files used to be the exception on Verbatim: we only had Hello on Verbatim, now we have donate.mozilla.org too on Pontoon. Amending the subject to avoid any further confusion in case someone searches for bugs.
Summary: Add possibility to download and upload .po files for projects → Add possibility to download and upload files for projects
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.