988321 - require human validation for large database changes

Status: RESOLVED FIXED

(Testing Graveyard :: Mozpool, defect)

Description

[Amy Rich [:arr] [:arich]]

In bug 988281 changes were made to inventory which resulted in the data sync deleting all but one (unchanged) panda.  We should put code in place so that massive deletes/additions require human intervention and verification like we do for dns/dhcp.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

What does "massive" mean in this case?  How should a human approve of such changes?

Comment 2

[Amy Rich [:arr] [:arich]]

I'd say we'd want human validation on more than a few chassis worth of changes?  Maybe 35 boards?  I know that with dns updates, it flags the change and notifies oncall (I think) so that they have to run a script to push the changes manually with a --ship-it flag. We should be able to do something similar with the sync script, right?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug988321.patch

The idea here is that someone would need to login to the mobile-imaging admin host and run MOZPOOL_CONFIG=/opt/mozpool/config.ini /opt/mozpool/frontend/bin/mozpool-inventorysync --verbose --ship-it after a substantial modification of inventory.  We'll get a nice test of this when we rename all of the pandas in the move.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug988321-p1.patch

Comment 5

[Jake Watkins [:dividehex]]

Comment on attachment 8397175 [details] [diff] [review]
bug988321.patch

I personally would have opted for ANY changes require a hand run and produce mail till that happens.  But this works also.

>+    # if there are too many changes, bail out
>+    if len(tasks) > max(5, len(from_db) / 10, len(from_inv) / 10) and not ship_it:
>+        raise RuntimeError("%d changes: pass --ship-it to make these changes" % len(tasks))
>+

There should probably be a little more explanation in the comment as to what constitutes "too many changes"
Other than that, looks good.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Will be in 4.2.0