BMO ETL: Change the way export loops through the row data so as to not allow on total and current count when tables may actually shrink before the table is fully exported
Categories
(bugzilla.mozilla.org :: Extensions, task)
Tracking
()
People
(Reporter: dkl, Assigned: dkl)
References
(Blocks 1 open bug)
Details
Attachments
(1 file)
Creating transactions around the SELECTS in bug 1967247 failed and was reverted. It was generating ERROR: Unable to create lock in database: Another process has set a lock. Exiting errors on different tables after 3 retries of running the job. My original thinking was that wrapping the SELECTs in a transaction, I would get a consistent view of each table regardless of changes being made on the production DB.
The current code performs a SELECT on a table to get the total count of rows and then a variable $count is incremented for each row. When $total == $count we exit the loop and move on to the next table. This would work if we had a consistent snapshot of the current table and the count of rows does not change during the loop.
I reverted the changes in bug 1967247 and propose in this bug to change the way we loop over the data that does not rely on total rows and just exits the loop when there is no more rows to fetch. This way if a tables total rows can decrease over time, such as the flags table, we still exit the loop properly at the right time and not loop forever.
|
||
Comment 1•1 year ago
|
||
|
|
||
Comment 2•1 year ago
|
||
Authored by https://github.com/dklawren
https://github.com/mozilla-bteam/bmo/commit/2e36e0de647095483d3503b691e4a39fa71a7c89
[master] Bug 1968044 - BMO ETL: Change the way export loops through the row data so as to not allow on total and current count when tables may actually shrink before the table is fully exported
Description
•