run update verify against beta-cdntest for betas

RESOLVED FIXED

Status

defect
P3
normal
RESOLVED FIXED
8 years ago
3 years ago

People

(Reporter: bhearsum, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2707] [automation][releases][updates])

Attachments

(4 attachments)

Ironically, the betatest channel serves almost no purpose in beta builds these days. Because we push them to the internal mirrors, and can do so early in the release process, we have the ability to test releasetest in update verify rather than betatest. This is better because releasetest snippets are identical to the actual snippets we'll be pushing to the beta channel users.

Implementing this could be tricky because we have different requirements for 3.6 and final releases.

For 3.6, we don't push to the internal mirrors prior to pushing to the beta channel, so we must run update verify against betatest in advance.

For final releases, we don't push them to the internal mirrors well in advance of the releases, so we should be testing them on betatest as soon as we can.

We've ignored betatest in chemspills recently, too, so it would be good to test releasetest in update verify for those as well.

Implementation wise this is tricky, because we have to change when update verify fires for different scenarios, and possibly change the update verify bump script, too. For 3.6 & finals, it should fire after "updates", and the bump script should use "betatest". For betas and chemspills, it should fire after "push_to_mirrors", and use "releasetest".
Whiteboard: [automation][releases]
Priority: -- → P3
Whiteboard: [automation][releases] → [automation][releases][updates]
Mass move of bugs to Release Automation component.
Component: Release Engineering → Release Engineering: Automation (Release Automation)
No longer blocks: hg-automation
Duplicate of this bug: 748815
Blocks: daily-beta
Product: mozilla.org → Release Engineering
Whiteboard: [automation][releases][updates] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2700] [automation][releases][updates]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2700] [automation][releases][updates] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2707] [automation][releases][updates]
Not really sure how much value there is in this, but updating summary to reflect new channel names.
QA Contact: bhearsum
Summary: run update verify against releasetest for betas → run update verify against beta-cdntest for betas
Assignee: nobody → rail
https://gist.github.com/rail/da29a4f25c747d166494ff233d53c43a is just a draft of the approach I want to take.
Comment on attachment 8801190 [details]
Bug 692501 - run update verify against beta-cdntest for betas

will be another day or two before I get to this one, feel free to redirect the request if you need it sooner.
Attachment #8801190 - Flags: review?(bugspam.Callek) → review?(mtabara)
Comment on attachment 8801190 [details]
Bug 692501 - run update verify against beta-cdntest for betas

https://reviewboard.mozilla.org/r/85930/#review85246
Attachment #8801190 - Flags: review?(mtabara) → review+
Comment on attachment 8801190 [details]
Bug 692501 - run update verify against beta-cdntest for betas

https://hg.mozilla.org/build/tools/rev/153c1d0220f31f549982ff28e923a906ef829157
Attachment #8801190 - Flags: checked-in+
Another idea here is to keep the same approach and run update verify tests only against 1 channel. The main goal of update verify tests is to make sure that we can apply updates.

Running update verify against other channels is repeating the same test, because we "should" serve the same binaries. To make sure we serve the same binaries we can compare them (candidates vs releases) and make sure that the balrog rules are the same. This way we won't need to rerun the same expensive tests again and again.

Still, in some cases (beta) it's worth skipping the localtest channel and test against cdntest.
(In reply to Rail Aliiev [:rail] ⌚️ET (away until Nov 21) from comment #9)
> binaries we can compare them (candidates vs releases) and make sure that the
> balrog rules are the same. This way we won't need to rerun the same
> expensive tests again and again.

Given that we talked about this in our chat last week when we compared update verify tests and firefox-ui-update tests, it would mean that if you decide to take this route, fx-ui-update tests would become more important to run for those other channels.

The question here is which of those tests are more expensive.
I don't think it's related. I'm not going to change the amount of tests that we run. What I'm saying is that we don't want to run the same test multiple times for no good reason. Once we run update verify tests on one channel, the results should be the same on any channel if the update rules and the binaries are the same. We just need to make sure that the binaries and the rules are the same.
Sure, that is clear. But you can argue the same for firefox-ui tests. That's why I'm asking where we could spend machine hours while increasing our coverage more effectively.
Yup, we just need to make sure we properly verify binaries and the rules. Then we can stop testing on multiple channels regardless of the test types (UI or update verify)
Attachment #8813651 - Flags: review?(jlund)
Posted file PR for releasetasks
Nothing can possibly go wrong. Testing in production is fine :)
Attachment #8813652 - Flags: review?(jlund)
Comment on attachment 8813651 [details] [diff] [review]
beta-uv-tools.diff

Review of attachment 8813651 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with one comment to be addressed or commented back on.

::: buildfarm/release/releasetasks_graph_gen.py
@@ +104,5 @@
>          "publish_to_balrog_channels": release_config["publish_to_balrog_channels"],
>          "postrelease_mark_as_shipped_enabled": release_config["postrelease_mark_as_shipped_enabled"],
>          # TODO: use [] when snaps_enabled is landed
>          "snap_enabled": release_config.get("snap_enabled", False),
> +        "update_verify_channel": release_config.get("update_verify_channel"),

for releasetasks_graph_gen.py, iirc I tried to make every config item required so a config file explicitly set every item and we didn't have in-script lookups. With that in mind, could we change this to something like:

"update_verify_channel": release_config["update_verify_channel"]
"update_verify_requires_cdn_push": release_config["update_verify_requires_cdn_push"]

then in each rel config (e.g. https://github.com/mozilla/releasetasks/blob/master/releasetasks/release_configs/prod_mozilla-release_firefox_rc_graph_2.yml)

have:
update_verify_channel: ""
update_verify_requires_cdn_push: false

might have yaml syntax wrong but I believe that is safe given your condition here: https://github.com/mozilla/releasetasks/pull/208/files#diff-3734a77e4781962e80fcb966440e29f8R49
Attachment #8813651 - Flags: review?(jlund) → review+
Comment on attachment 8813650 [details]
Bug 692501 - run update verify against beta-cdntest for betas

https://reviewboard.mozilla.org/r/95072/#review95368

so I guess this means that nothing will care about (verify) localtest within releng release automation for betas.
Attachment #8813650 - Flags: review?(jlund) → review+
Attachment #8813652 - Flags: review?(jlund) → review+
Attachment #8813652 - Flags: checked-in+
I addressed the "get" vs "[]" issue.
Comment on attachment 8813650 [details]
Bug 692501 - run update verify against beta-cdntest for betas

https://hg.mozilla.org/build/buildbot-configs/rev/829424621d03eba64d5d38a968100b13b41193d9

We should see this in next beta
Attachment #8813650 - Flags: checked-in+
This worked fine in Firefox 51.0b4.

Next step will be separating integrity verification and balrog verification logic into 2 separate tests. In this case we could test MAR file even before we upload them to the candidates directory.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.