1191718 - The new sync_projects.py script is slow

Status: RESOLVED FIXED

(Webtools Graveyard :: Pontoon, defect)

Description

[Matjaz Horvat [:mathjazz]]

Attachment: timings.txt

There's only one project using the new sync script that is big enough to measure timing (Mozilla.org). It takes 7-8 minutes to sync with the old method and more than double the time with the new one.

Comment 1

[[github robot]]

Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/88af74b2549217370e2ed065d234782ba1fdfa63
Bug 1191718: Track Entity.has_changed status in the database.

Instead of querying for translations every time we need to see if an entity has
changed for a particular locale, we instead store whether the entity has changed
in the database whenever translations are saved. This significantly speeds up
sync_projects by removing a ton of unnecessary database queries.

Since this replaces what Project.last_synced was used for, we remove that field.

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/2ef6a9dc0d800c2328e2f42489901d3d433a851f
Bug 1191718: Only update DB during sync when necessary.

Uses django-dirtyfields to determine if an Entity or Translation has actually
changed, so that we only update the database in cases where changes are
necessary rather than blindly updating all the time.

Comment 3

[Osmose [:osmose, :mkelly]]

According to mathjazz the sync time for Mozilla.org went down to as low as 3 minutes, but that may be dependent on how many strings are being updated or committed. Either way there's been improvement.

The last fix I'm working on before calling this bug resolved is to avoid updating a project/locale altogether by tracking the last VCS revision checked per project (or per locale for projects with multiple repos) and skipping sync if the project has no changes and the revision hasn't changed.

Comment 4

[Matjaz Horvat [:mathjazz]]

Attachment: timings2.png

Yes, the entire sync time for all 7 .lang projects has been reduced significantly. Great work!

Comment 5

[Matjaz Horvat [:mathjazz]]

Attachment: timings2.png

I meant this.

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/53529a8fac1a9724c60d68a39eef9743e408a978
Fix bug 1191718: Don't sync when there are no changes.

Now we track the revision number/hash from the last time we synced, and
use that to determine if VCS has changed since the last sync. If not,
and there are no changes in the DB, we can skip syncing completely.