1308625 - improve active data extraction script to grab referenced partials

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P3)

Description

[[e-mail] (:bhearsum)]

In bug 1284481, a script was developed that extracts all "active" releases from a database, where "active" is defined as "referenced by a rule". However, it does not grab the Releases that *those* Releases reference. Effectively this means that no Releases in a dump can serve partials.

This isn't the end of the world, but it would be better if partials were included as well.

To do this, the script would need to get smart enough to look through the initially extracted Releases, and then select all oft he Releases that those reference in their "from" sections.

Comment 1

[[e-mail] (:bhearsum)]

Ashish is going to take a look at this.

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/18d881ae973d366b990c56fbecd000892556b901
Bug 1308625 - improve active data extraction script to grab referenced partials (#211). r=bhearsum

Comment 3

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/24207fc98b0087b35811445aae07f6de2293e9de
Bug 1308625 : Adding extra arguments to skip dropping and recreating the table again (#221). r=bhearsum

Comment 4

[[e-mail] (:bhearsum)]

We tried this in production today, and the dump it generated gave an error when I tried to import it due to a duplicate release name:
balrogadmin_1  | ERROR 1062 (23000) at line 586: Duplicate entry 'Firefox-43.0.1-build1' for key 'PRIMARY'


My suspicion is that this a release that is both in the initial dump from releases, and also specified as a partial later. MAybe it's possible to generate the full set of releases up front to avoid any other bugs similar to this?

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/312635cdc88cb2b45e3f819aba29c5fa99c68eac
Backout bug 1308625 for now, because it's not working well enough yet. (#222)

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/897c39144caafaa41c53bfdd95c70f71e66b4b03
Bug 1308625 : Preventing same release row data from getting inserted twice in the sql dump file (#226). r=bhearsum

Comment 7

[[e-mail] (:bhearsum)]

This is in production now. I'm going to leave this open until tomorrow, so we can examine the new nightly dump.

Comment 8

[[e-mail] (:bhearsum)]

This is working great in production. Thanks Ashish!