Closed Bug 960275 Opened 10 years ago Closed 9 years ago

Implement offline mode

Categories

(Webtools Graveyard :: Pontoon, defect)

Product:
Webtools Graveyard
Component:
Pontoon
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mathjazz, Assigned: sandra.shklyaeva)

Details

Some localizers don't have a reliable connection all the time or work from a place where they don't even have a connection at all.

Implement offline mode, that will enable storing project-locale combinations for offline use. When user gets online again, data should be saved to the database.
As far as I can understand, we need to be able to handle situations like the following:
1. A localizer has a connection.
2. He/she opens Pontoon and chooses a project to translate.
3. Pontoon saves translations to the database on the server using ajax calls. Good.
4. A localizer loses or turns off the connection.
5. Pontoon reacts and starts to save all translations locally using HTML5 Local Storage. From the localizer point of view, everything works as usual. Nothing special has happened.
6. The next time the connection is available, Pontoon checks the local storage and save translations to the database, if any.

If a localizer wants to translate another project or another page inside the current project, nothing we can do without a connection. Am I right? I ask this because the phrase "they don't even have a connection at all" confuses me. If a localizer does not have a connection at all, how he/she can open Pontoon at least...

My questions are:
1. Should we implement offline mode to store translations only or there is also another functionality that requires offline mode?
2. Which versions of browsers are supported by Pontoon? I checked all available docs, but I cannot find these specs.
3. Is HTML5 Local Storage ok? Well, I am not sure we have any other options...

I would appreciate if you can provide me with any additional comments (if any) regarding the task.

And the last thing. When I have explored the projects available for translation through Pontoon, I noticed the following:
 - The "Pontoon Intro" project opens its website inside an iframe. It allows us to translate in place.
 - On the other hand, the "Google Play" project allows us to translate it through the sidebar only. No iframe(s). No in-place editing.

Why is the behavior different? Can I test the same behavior (translation through the sidebar only) for my local pontoon-intro? 

In other bug reports, you use "the translators workbench" phrase. Where can I find this "the translators workbench"? :) Is it the sidebar I see clicking on the hamburger icon?
I have figured out why there are different behaviors...
(In reply to sandra.shklyaeva from comment #1)
> As far as I can understand, we need to be able to handle situations like the
> following:
> 1. A localizer has a connection.
> 2. He/she opens Pontoon and chooses a project to translate.
> 3. Pontoon saves translations to the database on the server using ajax
> calls. Good.
> 4. A localizer loses or turns off the connection.
> 5. Pontoon reacts and starts to save all translations locally using HTML5
> Local Storage. From the localizer point of view, everything works as usual.
> Nothing special has happened.
> 6. The next time the connection is available, Pontoon checks the local
> storage and save translations to the database, if any.

Correct.

> If a localizer wants to translate another project or another page inside the
> current project, nothing we can do without a connection. Am I right? I ask
> this because the phrase "they don't even have a connection at all" confuses
> me. If a localizer does not have a connection at all, how he/she can open
> Pontoon at least...

Right. We can only help with pre-loaded projects.

> My questions are:
> 1. Should we implement offline mode to store translations only or there is
> also another functionality that requires offline mode?

Storing translations only. We will also need to display warm "You are offline" messages in places like helpers (history, machinery, etc.) that depend on connection.

> 2. Which versions of browsers are supported by Pontoon? I checked all
> available docs, but I cannot find these specs.

Good question. While Pontoon is only used at Mozilla for now, we're only testing latest versions of Firefox, Chrome and Safari.

> 3. Is HTML5 Local Storage ok? Well, I am not sure we have any other
> options...

This should help:
https://developer.mozilla.org/en/docs/Web/Guide/API/DOM/Storage

> And the last thing. When I have explored the projects available for
> translation through Pontoon, I noticed the following:
>  - The "Pontoon Intro" project opens its website inside an iframe. It allows
> us to translate in place.
>  - On the other hand, the "Google Play" project allows us to translate it
> through the sidebar only. No iframe(s). No in-place editing.
> 
> Why is the behavior different? Can I test the same behavior (translation
> through the sidebar only) for my local pontoon-intro? 

Some projects do not have a website provided for in place translation. In this case we only open the sidebar and localizers translate like in other localization tools. To disable in place translation, you need to clear the Website URL in the project admin.

> In other bug reports, you use "the translators workbench" phrase. Where can
> I find this "the translators workbench"? :) Is it the sidebar I see clicking
> on the hamburger icon?

Yeah, that means the sidebar. Sorry, some of these bugs were filed long time ago.
First implementation of the offline mode is ready.
It is available in the offlinemode-960275 branch:
https://github.com/SandraShklyaeva/pontoon/tree/offlinemode-960275

Currently, the implementation uses existing API to add translations to the server i.e. one translation per AJAX request. If we have a lot of "unsaved" translations in the local storage, we can potentially get dozens of requests to the server. Therefore, I plan to extend the update_translation in the views.py in a way it can accept a list of new translations. Please let me know what you think about this.
Sandra, that looks very promising! I like the way you structure the code and make it simple and readable. I have a few comments to consider before we land this. Please take them as suggestions for improvement.

1. Saving translations
======================
When saving translations in offline mode, they are marked as approved=false. We should use approved=self.user.localizer, which will mark translations approved if submitted by privileged users. We use the same approach when saving translations in online mode.

2. Updating the server
======================
I was unable to save translations submitted while offline back to server. I tested this using the Pontoon Intro project. I wonder if that's related to they way localStorage key is defined. Please note that entity.key is only available for .properties file format. Shall we just use a combination of locale.code and entity.pk (no need for project.pk)?

Also, I don't fully understand why did you decide to store the translation.previous?
I have just updated the branch. Please checkout.

1. Saving translations
======================
Fixed.

2. Key Generation
======================
Done. I used the combination you proposed (locale.code and entity.pk).
Is entity.pk unique across all projects?

3. Updating the server 
======================
I was able to save translations to the database even with the old logic for localStorage key generation.
Therefore, I added some logs to this version. So, if you are still unable to save translations, please let me know what you see in the console. 
Everything works fine for me right now :)

4. What is translation.previous? 
================================
The translation.previous always stores previous translation available in the database. 
We do NOT update it every time a localizer submit a translation in offline mode.

For example, let's have the following chain of translations:
"Title1" (fetched from the server) -> "Title2" (offline mode) -> "Title3" (offline mode)

With the above chain, our translation.previous always equals "Title1".
   
Why do we need it? Let's imagine the following:
- Localizer John translates "Title1" to "Title2" in offline mode and closes his browser.
- Nothing is saved to the database. The translation will be saved the next time John opens the Pontoon. Seems good, but...
- Before the John's next time, another localizer sees "Title1" ('cause we have it in the database) and translates it to "Title3" in online mode. Now our database has "Title3" as the most recent translation.
- When the John's next time comes, John should see "Title3". We can not just silently save "old" translation "Title2" from his local storage.
- Therefore, we use translation.previous. 
  If current translation in the database is the same as translation.previous, the John's local translation is still relevant -> save.
  If it's not (someone pushed a new translation after John), the John's local translation is no longer "valid" -> ignore & clear.
(In reply to sandra.shklyaeva from comment #6)
> 1. Saving translations
> Fixed.

\o/

(In reply to sandra.shklyaeva from comment #6)
> Is entity.pk unique across all projects?

Yes, that's a primary key from the Entity table.

> Therefore, I added some logs to this version. So, if you are still unable to
> save translations, please let me know what you see in the console. 
> Everything works fine for me right now :)

I'll check and report back.

> Why do we need it? Let's imagine the following:
> - Localizer John translates "Title1" to "Title2" in offline mode and closes
> his browser.
> - Nothing is saved to the database. The translation will be saved the next
> time John opens the Pontoon. Seems good, but...
> - Before the John's next time, another localizer sees "Title1" ('cause we
> have it in the database) and translates it to "Title3" in online mode. Now
> our database has "Title3" as the most recent translation.
> - When the John's next time comes, John should see "Title3". We can not just
> silently save "old" translation "Title2" from his local storage.
> - Therefore, we use translation.previous. 
>   If current translation in the database is the same as
> translation.previous, the John's local translation is still relevant -> save.
>   If it's not (someone pushed a new translation after John), the John's
> local translation is no longer "valid" -> ignore & clear.

I see. I wonder what does less damage: overriding translation submited while John was offline (but still keeping it in histroy) or throwing away John's work completely. IMHO first option is better, so I suggest we use it in the first implementation while keeping the "previous" functionality documented here as an idea we can move forward with later on.
I evaluated the updated code.

1. There's a JS error in the else {} block in lines 1251-1253 - it should be moved one line up.

2. Translations are still saved with approved=false. 

3. I'm still unable to update offline translations on server. I'm using the Pontoon intro project and I'm logged in as a user with full (admin) privileges. The steps I'm taking are as follows:

A. File -> Enabled Offline mode.
B. Submit first translation (string "See spatial limitations").
C. File -> Disabled Offline mode.
D. Submit second translation (string "Get instant preview").
E. Click Save anyway on quality check warning.
F. Reload page.

RESULT: First translation (step B) is gone, second translation (step D) is still there.

The console output:

"ADDED TRANSLATION: Prostorske omejitve" translate.js:1273:6
Storage { sl/94258: "{"translation":"Prostorske omejitve"}", length: 1 } translate.js:1274:6
"ENTITIES IN LOCAL STORAGE: 52" translate.js:1227:8
"LOCAL STORAGE IS EMPTY!" translate.js:1251:12
"SYNC TRANSLATION" translate.js:1234:12
Object { translation: "Prostorske omejitve" } translate.js:1235:12
"PREVIOUS TRANSLATION: Prostorske omejitve false" translate.js:1236:0
"ORIGINAL TRANSLATION: See spatial limitations false" translate.js:1238:0
"FAILED!" translate.js:1246:14
" " translate.js:1248:12
"LOCAL STORAGE IS EMPTY!" translate.js:1251:12
I was just about to update :)

1. Removed. Other logs are removed too.
2. Fixed. Feels like magic 'cause I was sure I have it pushed...
3. Fixed.
4. Removed checks for translation.previous. You are right, it is better to use history on the server. Somehow I forgot we have it.
Seems like it works. :-)

Good job! Now I'd like to ask you to make a pull request against pontoon master. Before that, you should also remove all refereces to "previous" in translate.js and pontoon.js.
Done :)
Merged:
https://github.com/mathjazz/pontoon/commit/3959ecffbe37bd66919dcd9e99c5ec2d6bd10d5d

I've also made a few formatting changes to fit the code style we use:
https://github.com/mathjazz/pontoon/commit/80df5799225714b04eb045480b3c85792d4bf3dc

And fixed two bugs:
https://github.com/mathjazz/pontoon/commit/295a65a09ab8b67450e1fdaad8bac077b12ed289
https://github.com/mathjazz/pontoon/commit/50a7fede18e2272a8a0f28c29a3edaf642efda4c

There's a tiny regression I'd like to fix before we close this. If there are two translations available for a string that is localizable in place, when you delete current translation, the next one is selected and the History tab contents visually "jumps" (refreshes). It shouldn't.
(In reply to Matjaz Horvat [:mathjazz] from comment #12)
> There's a tiny regression I'd like to fix before we close this. If there are
> two translations available for a string that is localizable in place, when
> you delete current translation, the next one is selected and the History tab
> contents visually "jumps" (refreshes). It shouldn't.

That has nothing to do with offline mode. Sorry, my bad.

Closing this bug. Thank you for adding your first new feature to Pontoon! Looking forward to more. :-)
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
LOL :) But I am in process to debug it right now. Well, okay...
If you're already working on the fix (it's still needed) feel free to go on. I just wanted to say it's a different bug.
Discussion about flickering moved to bug 1155103.
Assignee: nobody → sandra.shklyaeva
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.