1967247 - BMO ETL: Create transaction around entire export process with REPEATABLE READ isolation to ensure data integrity

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Extensions, enhancement)

Description

[David Lawrence [:dkl]]

It seems to me that intermittently the flags table fails during export due to the rapid changes the flags table goes through with normal Bugzilla operation. This can cause the while loop that is used when pulling rows from the table to never exit and loop forever. Using a wrapping transaction around the export process with give a more consistent view of the data and see only the changes that have been committed before the transaction began.

Comment 1

[BMO Github Automation]

Attachment: [mozilla-bteam/bmo] Bug 1967247 - BMO ETL: Create transaction around entire export process with REPEATABLE READ isolation to ensure data integrity (#2443)

Comment 2

[David Lawrence [:dkl]]

From the MySQL docs:

---

If the transaction isolation level is REPEATABLE READ (the default level), all consistent reads within the same transaction read the snapshot established by the first such read in that transaction. You can get a fresher snapshot for your queries by committing the current transaction and after that issuing new queries.

With READ COMMITTED isolation level, each consistent read within a transaction sets and reads its own fresh snapshot.

---

The export script reads rows in 1000 chunks so we cannot use READ COMMITTED as it would be a new snapshot for each SELECT and so the total row count could change.

Comment 3

[BMO Github Automation]

Authored by https://github.com/dklawren
https://github.com/mozilla-bteam/bmo/commit/70505ce3dd2cc8b49e0138fdecd5781543ddebd1
[master] Bug 1967247 - BMO ETL: Create transaction around entire export process with REPEATABLE READ isolation to ensure data integrity

Comment 4

[BMO Github Automation]

Authored by https://github.com/dklawren
https://github.com/mozilla-bteam/bmo/commit/23b5c5a54772c2f1d30c04c64f02b13c70f5de0e
[master] Revert "Bug 1967247 - BMO ETL: Create transaction around entire export process with REPEATABLE READ isolation to ensure data integrity"