692501 - run update verify against beta-cdntest for betas

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[[e-mail] (:bhearsum)]

Ironically, the betatest channel serves almost no purpose in beta builds these days. Because we push them to the internal mirrors, and can do so early in the release process, we have the ability to test releasetest in update verify rather than betatest. This is better because releasetest snippets are identical to the actual snippets we'll be pushing to the beta channel users.

Implementing this could be tricky because we have different requirements for 3.6 and final releases.

For 3.6, we don't push to the internal mirrors prior to pushing to the beta channel, so we must run update verify against betatest in advance.

For final releases, we don't push them to the internal mirrors well in advance of the releases, so we should be testing them on betatest as soon as we can.

We've ignored betatest in chemspills recently, too, so it would be good to test releasetest in update verify for those as well.

Implementation wise this is tricky, because we have to change when update verify fires for different scenarios, and possibly change the update verify bump script, too. For 3.6 & finals, it should fire after "updates", and the bump script should use "betatest". For betas and chemspills, it should fire after "push_to_mirrors", and use "releasetest".

Comment 1

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 3

[[e-mail] (:bhearsum)]

Not really sure how much value there is in this, but updating summary to reflect new channel names.

Comment 4

[Rail Aliiev [:rail]]

Attachment: Bug 692501 - run update verify against beta-cdntest for betas

Review commit: https://reviewboard.mozilla.org/r/85930/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/85930/

Comment 5

[Rail Aliiev [:rail]]

https://gist.github.com/rail/da29a4f25c747d166494ff233d53c43a is just a draft of the approach I want to take.

Comment 6

[Justin Wood (:Callek)]

Comment on attachment 8801190 [details]
Bug 692501 - run update verify against beta-cdntest for betas

will be another day or two before I get to this one, feel free to redirect the request if you need it sooner.

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8801190 [details]
Bug 692501 - run update verify against beta-cdntest for betas

https://reviewboard.mozilla.org/r/85930/#review85246

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8801190 [details]
Bug 692501 - run update verify against beta-cdntest for betas

https://hg.mozilla.org/build/tools/rev/153c1d0220f31f549982ff28e923a906ef829157

Comment 9

[Rail Aliiev [:rail]]

Another idea here is to keep the same approach and run update verify tests only against 1 channel. The main goal of update verify tests is to make sure that we can apply updates.

Running update verify against other channels is repeating the same test, because we "should" serve the same binaries. To make sure we serve the same binaries we can compare them (candidates vs releases) and make sure that the balrog rules are the same. This way we won't need to rerun the same expensive tests again and again.

Still, in some cases (beta) it's worth skipping the localtest channel and test against cdntest.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Rail Aliiev [:rail] ⌚️ET (away until Nov 21) from comment #9)
> binaries we can compare them (candidates vs releases) and make sure that the
> balrog rules are the same. This way we won't need to rerun the same
> expensive tests again and again.

Given that we talked about this in our chat last week when we compared update verify tests and firefox-ui-update tests, it would mean that if you decide to take this route, fx-ui-update tests would become more important to run for those other channels.

The question here is which of those tests are more expensive.

Comment 11

[Rail Aliiev [:rail]]

I don't think it's related. I'm not going to change the amount of tests that we run. What I'm saying is that we don't want to run the same test multiple times for no good reason. Once we run update verify tests on one channel, the results should be the same on any channel if the update rules and the binaries are the same. We just need to make sure that the binaries and the rules are the same.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sure, that is clear. But you can argue the same for firefox-ui tests. That's why I'm asking where we could spend machine hours while increasing our coverage more effectively.

Comment 13

[Rail Aliiev [:rail]]

Yup, we just need to make sure we properly verify binaries and the rules. Then we can stop testing on multiple channels regardless of the test types (UI or update verify)

Comment 14

[Rail Aliiev [:rail]]

Attachment: Bug 692501 - run update verify against beta-cdntest for betas

Review commit: https://reviewboard.mozilla.org/r/95072/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/95072/

Comment 15

[Rail Aliiev [:rail]]

Attachment: beta-uv-tools.diff

Comment 16

[Rail Aliiev [:rail]]

Attachment: PR for releasetasks

Nothing can possibly go wrong. Testing in production is fine :)

Comment 17

[Jordan Lund (:jlund)]

Comment on attachment 8813651 [details] [diff] [review]
beta-uv-tools.diff

Review of attachment 8813651 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with one comment to be addressed or commented back on.

::: buildfarm/release/releasetasks_graph_gen.py
@@ +104,5 @@
>          "publish_to_balrog_channels": release_config["publish_to_balrog_channels"],
>          "postrelease_mark_as_shipped_enabled": release_config["postrelease_mark_as_shipped_enabled"],
>          # TODO: use [] when snaps_enabled is landed
>          "snap_enabled": release_config.get("snap_enabled", False),
> +        "update_verify_channel": release_config.get("update_verify_channel"),

for releasetasks_graph_gen.py, iirc I tried to make every config item required so a config file explicitly set every item and we didn't have in-script lookups. With that in mind, could we change this to something like:

"update_verify_channel": release_config["update_verify_channel"]
"update_verify_requires_cdn_push": release_config["update_verify_requires_cdn_push"]

then in each rel config (e.g. https://github.com/mozilla/releasetasks/blob/master/releasetasks/release_configs/prod_mozilla-release_firefox_rc_graph_2.yml)

have:
update_verify_channel: ""
update_verify_requires_cdn_push: false

might have yaml syntax wrong but I believe that is safe given your condition here: https://github.com/mozilla/releasetasks/pull/208/files#diff-3734a77e4781962e80fcb966440e29f8R49

Comment 18

[Jordan Lund (:jlund)]

Comment on attachment 8813650 [details]
Bug 692501 - run update verify against beta-cdntest for betas

https://reviewboard.mozilla.org/r/95072/#review95368

so I guess this means that nothing will care about (verify) localtest within releng release automation for betas.

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8813651 [details] [diff] [review]
beta-uv-tools.diff

https://hg.mozilla.org/build/tools/rev/409f510a288eb137f3495ceee9adb6831e7402f1

Comment 20

[Rail Aliiev [:rail]]

I addressed the "get" vs "[]" issue.

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 8813650 [details]
Bug 692501 - run update verify against beta-cdntest for betas

https://hg.mozilla.org/build/buildbot-configs/rev/829424621d03eba64d5d38a968100b13b41193d9

We should see this in next beta

Comment 22

[Rail Aliiev [:rail]]

This worked fine in Firefox 51.0b4.

Next step will be separating integrity verification and balrog verification logic into 2 separate tests. In this case we could test MAR file even before we upload them to the candidates directory.