1311003 - Validate that balrog rules correctly implement watersheds

Status: RESOLVED DUPLICATE of bug 1320373

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P3)

Description

[Chris AtLee [:catlee]]

Once we have watershed information published in shipit, we should have checks in Balrog to ensure we're properly implementing the watersheds.

Comment 1

[[e-mail] (:bhearsum)]

This might be better implemented as an update-verify style test instead of in the Balrog backend, but we'll see! We definitely need to do it somewhere.

Comment 2

[Chris AtLee [:catlee]]

Yeah, I don't think it has to be in the backend either. Just filed it here since I couldn't think of a better place :)

Comment 3

[Jordan Lund (:jlund)]

I recently got bit by not having this: 
https://bugzilla.mozilla.org/show_bug.cgi?id=1319167#c30

While porting a bunch of watersheds from central to aurora, I accidently clicked 'update' instead of 'duplicate' on one of the watersheds, effectively killing the existing central watershed!

Having silly humans like me being able to cause such dramatic changes without tests is bad. It would be great to have tests that run on balrog rule changes, and the change only goes into effect if it passes the tests. This could be painful as you would need to update the tests (in most cases) and the actual rules when making changes but it provides a sort of 'are you sure you want to do this' safety mechanism.

Comment 4

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #3)

ben reminded me of this one: Bug 1241766 - Balrog rule changes should inform users of the impact of the rule change

that bug might better help me here as it could show the effect of the change without actual tests or expectations. I see this bug itself being within update verify tests, validating specific watersheds for specific releases.

Comment 5

[[e-mail] (:bhearsum)]

I think this is a subset of bug 1320373.