1951478 - BMO ETL: Add a timestamp to lock table and remove lock if timestamp is greater than a day

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Extensions, enhancement)

Description

[David Lawrence [:dkl]]

Currently if a export fails before the lock table can be cleared, any subsequent exports will not run as the lock will never be removed. Since a new day is a new export in ETL, we can just clear the lock anyway if it is greater than 24 hours and try again even if it fails the same way.

Comment 1

[:glob ✱]

Would it be possible instead to clear the lock even if there's a failure?

Comment 2

[David Lawrence [:dkl]]

(In reply to :glob ✱ from comment #1)

Would it be possible instead to clear the lock even if there's a failure?

Maybe but I would need to put all of the SQL statements in try/catch blocks and create a timer that will kill a query after some period of time has passed. The removal of old locks seemed to be the quickest solution at the time. The problem is the container is stopping a query sometimes in the middle without any error message of any kind. And the container either stays stuck like that or dies completely. The script would not be able to remove the lock without some sort of watchdog.

Comment 3

[BMO Github Automation]

Attachment: [mozilla-bteam/bmo] Bug 1951478 - BMO ETL: Add a timestamp to lock table and remove lock if timestamp is greater than a day (#2398)

Comment 4

[BMO Github Automation]

Authored by https://github.com/dklawren
https://github.com/mozilla-bteam/bmo/commit/fc9ef42ca191a8b235863ea88a77ad45ed1fefc2
[master] Bug 1951478 - BMO ETL: Add a timestamp to lock table and remove lock if timestamp is greater than a day