588396 - run update verify on nightlies

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[[e-mail] (:bhearsum)]

Omnijar was backed out because updates were broken. To help reduce the frequency of breaking nightly updates and the window in which their broken we should run update verify tests on nightlies.

Ideally, we'd run this post-build but prior to publishing the update. If update_verify fails, the update should not be published, I think.

Comment 1

[[e-mail] (:bhearsum)]

A bit more detail: Specifically, we'd be doing the "compare update applied against previous build to current build" part of update verify. We wouldn't be able to use AUS.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Given the discussion on bug 498425 would that be the right place to have Mozmill tests in place for nightly builds? Which means that those test can be run before we offer the updates on the nightly channel? As given the info from coop we do not have a test channel for nightlies yet. It's probably something we could add here?

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ben, we will have a CI system up shortly which let us run all kind of tests against daily builds of Firefox. It doesn't match your criteria by doing it before we publish the update to our users, but would there be an option we could do that, ie. by using a specific update channel?

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) from comment #3)
> Ben, we will have a CI system up shortly which let us run all kind of tests
> against daily builds of Firefox. It doesn't match your criteria by doing it
> before we publish the update to our users, but would there be an option we
> could do that, ie. by using a specific update channel?

Not at this time. Right now, the only place you could test updates prior publishing is as part of the build =(.

Comment 5

[Chris AtLee [:catlee]]

Can we rescope this slightly to run a modified update-verify process for nightlies?

We can fetch the current and previous mar files directly given information in the taskgraph. We can fetch the linux64 updater binaries from the linux64 build, and then use it to verify that all the nightly updates can be applied and result in the correct end-state. In this way we could avoid a dependency on AUS, and be able to block publishing updates until they've been tested.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Chris AtLee [:catlee] from comment #5)

Can we rescope this slightly to run a modified update-verify process for
nightlies?

We can fetch the current and previous mar files directly given information
in the taskgraph. We can fetch the linux64 updater binaries from the linux64
build, and then use it to verify that all the nightly updates can be applied
and result in the correct end-state. In this way we could avoid a dependency
on AUS, and be able to block publishing updates until they've been tested.

This probably wouldn't be very hard. https://dxr.mozilla.org/mozilla-central/source/tools/update-verify/release/common/check_updates.sh wants the path to two MARs on disk (as well as a few other things), and it should run the test and give you pass/fail.

Comment 7

[Aki Sasaki (not active)]

We have startup tests for nightlies.