BMO ETL: Fix logic bug introduced by bug 1968044 that caused the looping of database rows to exit too early leaving out data
Categories
(bugzilla.mozilla.org :: Infrastructure, defect)
Tracking
()
People
(Reporter: dkl, Assigned: dkl)
References
(Blocks 1 open bug)
Details
Attachments
(1 file)
With the landing of bug 1968044. I was trying to fix a looping issue for mainly the flag table where it would shrink while we were pulling blocks of data 1000 rows at at time. It would get stuck in an endless loop since the $total had decreased from when we first read the total count and so $count would never equal $total to break out.
Using transactions to give a snapshot of the flags table, which should have not seen any changes, did not work as it gave locking errors often.
The change in bug 1968044 introduced a new logic bug of its own. We would select rows 100 blocks at a time and then loop as long as their were more rows available. When we hit a 1000, we then would reset and run the query again with the OFFSET incremented by 1000. But we also skip bugs sometimes if they are part of Legal product for example. So in that case the number of rows in @results would be less than 1000 and would not run the query again, the while loop would break and we would never see the rest of the rows.
This fix will always run a new query in all cases and the while loop only exits when there are no more rows in the table regardless if we skipped any to be excluded.
|
||
Comment 1•6 months ago
|
||
|
|
||
Comment 2•6 months ago
|
||
Authored by https://github.com/dklawren
https://github.com/mozilla-bteam/bmo/commit/1aff34d2dc4bc5400db1cbebf097e322156370c3
[master] Bug 1969307 - BMO ETL: Fix logic bug introduced by bug 1968044 that caused the looping of database rows to exit too early leaving out data
Description
•