588398 - run mozmill update tests on nightly updates

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P5)

Comment 1

[[e-mail] (:bhearsum)]

Once we're capable of running Mozmill in Buildbot it would be great to have it to test nightly updates. We'd have to test them after they are published, but it would let us catch busted updates a lot sooner.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We already run those tests on our QA lab machines for en-US builds on all platforms. Ben, which improvements you would like to see and which resources does RelEng have?

Personally I have already targeted to run update tests for tier 1 locales too. Would be great if we could combine those efforts on RelEng hardware which is probably better suited for those tasks as our own machines. Not to talk about the system integration scripts.

The main problem I still see here is that even with Mozmill running tests on buildbot, those will only be en-US builds. But we want to target to run tests against nearly all locales. So we would still have to run update tests on our own boxes to have recent l10n builds available.

I don't want to duplicate efforts but finding a proper solution is probably the best choice we can make before starting to work on this bug.

Comment 3

[[e-mail] (:bhearsum)]

That's great to hear that QA is already running tests on nightly builds. What I'd like to see happen with this bug is that we start triggering such tests immediately after the en-US nightly finishes and that if the tests fail, we get alerted so we can pull the updates.

Spotchecks on l10n nightlies would be good too, I don't think it's worthwhile testing every one though. I don't see why Mozmill couldn't test l10n nightlies once we've got it running in Buildbot. Is there a hidden blocker I'm missing?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #3)
> That's great to hear that QA is already running tests on nightly builds. What
> I'd like to see happen with this bug is that we start triggering such tests
> immediately after the en-US nightly finishes and that if the tests fail, we get
> alerted so we can pull the updates.

Sounds good. Would there be a way for us to get alerted too? Or does that only work internally on your boxes?

> Spotchecks on l10n nightlies would be good too, I don't think it's worthwhile
> testing every one though. I don't see why Mozmill couldn't test l10n nightlies
> once we've got it running in Buildbot. Is there a hidden blocker I'm missing?

Nope. AFAIK we only test en-US builds on buildbot. So I wonder if we want or better can go this step. It will take a long time to run all those tests for each locale. Having those tests on a nightly basis would probably a better idea. Shall we spun-off that discussion into another bug?

Comment 5

[[e-mail] (:bhearsum)]

(In reply to comment #4)
> (In reply to comment #3)
> > That's great to hear that QA is already running tests on nightly builds. What
> > I'd like to see happen with this bug is that we start triggering such tests
> > immediately after the en-US nightly finishes and that if the tests fail, we get
> > alerted so we can pull the updates.
> 
> Sounds good. Would there be a way for us to get alerted too? Or does that only
> work internally on your boxes?

Test results will end up showing up on TBPL, as all other tests do. QA can be alerted directly by e-mail too, if you so choose.

> 
> > Spotchecks on l10n nightlies would be good too, I don't think it's worthwhile
> > testing every one though. I don't see why Mozmill couldn't test l10n nightlies
> > once we've got it running in Buildbot. Is there a hidden blocker I'm missing?
> 
> Nope. AFAIK we only test en-US builds on buildbot. So I wonder if we want or
> better can go this step. It will take a long time to run all those tests for
> each locale. Having those tests on a nightly basis would probably a better
> idea. Shall we spun-off that discussion into another bug?

Let's start with en-US then. I'm certain we can figure out how to make l10n builds through Buildbot/Mozmill work, though.


This bug is weeks away from having anything done on it, though.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #5)
> Test results will end up showing up on TBPL, as all other tests do. QA can be
> alerted directly by e-mail too, if you so choose.

Nice! For the time being lemme ask how we could detect when new nightly builds and updates are available? Which trigger are you using? Right now we fire our test-runs at 8am each day. 

> Let's start with en-US then. I'm certain we can figure out how to make l10n
> builds through Buildbot/Mozmill work, though.
> 
> This bug is weeks away from having anything done on it, though.

Sure. Shall I file the work which is needed for l10n builds already as a new bug or would you like to wait until we have marked this bug as fixed?

Comment 7

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> (In reply to comment #5)
> > Test results will end up showing up on TBPL, as all other tests do. QA can be
> > alerted directly by e-mail too, if you so choose.
> 
> Nice! For the time being lemme ask how we could detect when new nightly builds
> and updates are available? Which trigger are you using? Right now we fire our
> test-runs at 8am each day.

Buildbot can trigger these types of tests when the nightly finishes.

> 
> > Let's start with en-US then. I'm certain we can figure out how to make l10n
> > builds through Buildbot/Mozmill work, though.
> > 
> > This bug is weeks away from having anything done on it, though.
> 
> Sure. Shall I file the work which is needed for l10n builds already as a new
> bug or would you like to wait until we have marked this bug as fixed?

Let's wait until we've got this fixed and then see what we think we need.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #7)
> > Nice! For the time being lemme ask how we could detect when new nightly builds
> > and updates are available? Which trigger are you using? Right now we fire our
> > test-runs at 8am each day.
> 
> Buildbot can trigger these types of tests when the nightly finishes.

Is there a way to send external triggers? Until we have a final solution on the releng side we want to execute our Mozmill tests as soon as possible updates are available. But we would need a trigger.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #8)
> > Buildbot can trigger these types of tests when the nightly finishes.
> 
> Is there a way to send external triggers? Until we have a final solution on the
> releng side we want to execute our Mozmill tests as soon as possible updates
> are available. But we would need a trigger.

Would be great to get the information if that would also work with testing outside of the releng framework.

Comment 10

[Jeff Hammel]

(In reply to comment #9)
> (In reply to comment #8)
> > > Buildbot can trigger these types of tests when the nightly finishes.
> > 
> > Is there a way to send external triggers? Until we have a final solution on the
> > releng side we want to execute our Mozmill tests as soon as possible updates
> > are available. But we would need a trigger.
> 
> Would be great to get the information if that would also work with testing
> outside of the releng framework.

You can always poll FTP for new builds.  While in the long term it would be nice to have this be an event, I'm not sure if this is a priority.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Until RelEng can work on this bug I have filed bug 610915 to make it possible to trim our QA horus box to execute testruns as soon as possible builds/updates are available.

Comment 12

[Aki Sasaki (not active)]

This seems tied at the hip to bug 498425.

Comment 13

[Aki Sasaki (not active)]

Aiui, we're testing the latest nightly here, and the latest release in bug 498425, for the ability to update to anything.

[17:06] <nthomas>   there are quite a few tests in the tree now, if we're paying attention to the results
[17:06] <nthomas>   say on nightly builds we could do a special check run that calls those tests, and fail the nightly if any tests fail
[17:09] <nthomas>   prior to uploading it that is

I'm not sure if running these tests would obsolete this bug and bug 498425.

Assuming we continue on this path, I have a wip script that downloads a list of firefoxes and updates them against a list of channels.
http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness/file/f79f1af5f2b7/scripts/mozmill_updates.py

However, since we're talking the latest release/nightly here, there will be nothing to update to on any official channels.  We could potentially use betatest or releasetest, or a new nightlytest. Or, we could:

* create a fake snippet on disk or on a fake update server
* set app.update.url.override to file:// or point to the fake update server
* have that snippet pointing at (the latest? a generic?) complete mar
* attempt to update and post results.
* if this happens before pushing the snippets/mars live, then prevent that push if the update fails.

We could also do this against a subset, or the entirety, of locales as well.

If that's the approach here, I have a few questions:

* How do I add app.update.url.override to the prefs.js used by testrun_update.py ? Or should we add that capability, or should I call the mozmill extension directly after creating a profile?
* Is it better to try to update to the current complete.mar, or to a fake complete.mar of an aging version of Firefox?
** The current complete.mar seems more accurate, but is more work to set up.
** Do we need to test partial updates here, or is knowing we can update at all acceptable?

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

See also bug 586350.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Urgh, make that bug 587344.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Aki, which specific things we should test to ensure the build has been updated correctly. If you could supply the version and buildid to the update script we could do those checks internally. Is there anything else we should be aware of?

Comment 17

[Aki Sasaki (not active)]

(In reply to comment #16)
> Aki, which specific things we should test to ensure the build has been
> updated correctly. If you could supply the version and buildid to the update
> script we could do those checks internally. Is there anything else we should
> be aware of?

If, for the release update testing, we use the current complete.mar to update the current build, the version and buildid won't change at all, in which case I don't know what to check for other than maybe timestamps or if there's an update log.

We could potentially use the previous build's complete.mar for nightlies and releases. I'm worried about what that means -- updating 7.0b1 -> 6.0, for example. Or 9.0a1 -> 8.0a1. If we don't foresee that being an issue, I can go that route.

Do you know how I can set app.update.url.override through mozmill-automation?

Comment 18

[[e-mail] (:bhearsum)]

(In reply to comment #13)
> Aiui, we're testing the latest nightly here, and the latest release in bug
> 498425, for the ability to update to anything.
> 
> [17:06] <nthomas>   there are quite a few tests in the tree now, if we're
> paying attention to the results
> [17:06] <nthomas>   say on nightly builds we could do a special check run
> that calls those tests, and fail the nightly if any tests fail
> [17:09] <nthomas>   prior to uploading it that is
> 
> I'm not sure if running these tests would obsolete this bug and bug 498425.

Seems like we should find out if those tests are enough before investing any more time here.

On the assumption they aren't....

> we could:
> 
> * create a fake snippet on disk or on a fake update server
> * set app.update.url.override to file:// or point to the fake update server
> * have that snippet pointing at (the latest? a generic?) complete mar
> * attempt to update and post results.
> * if this happens before pushing the snippets/mars live, then prevent that
> push if the update fails.

Can we set app.update.url.override to the normal URL, except with the n-2 buildid (that is, the last one with an update available) hardcoded in? If we did that, we'd test in the most real world way possible (no fake mar or snippet, etc.).

> ** Do we need to test partial updates here, or is knowing we can update at
> all acceptable?

I don't think it matters whether we test a complete or a partial. In terms of "does the application updater work", they're the same IMHO.

Comment 19

[Robert Strong (they/them - no direct email)]

This isn't so much about verifying the updater works as it is about verifying all steps for an update work. The things I want this to test that can't be tested in the normal tests are:

Verify that the aus certificate is valid by performing the check against aus3. The app.update.url preference - not the app.update.url.override preference - would need to be set and can be done during test run by just setting the default preference via code. Would likely need to have the ability to advertise the update snippet to a test url on aus3 that returns the same update snippet when it is advertised to the public.

Test the partial update and verify the files / directories after the update are the same as the build. This will catch cases where files aren't added due to not being included in the app's package.manifest. I still have some work to do to get the directory removal story 100% but it should be good to go with mars generated since Firefox 5.0.

Test the complete update against the build the partial was applied to and verify the files / directories after the update are the same as the build. This should catch issues like bug 670468 as well as test that the new updater will work for the next nightly build (should already be covered by bug 601518 but this is important enough to test here as well IMO).

There are also other items this will cover due to the operations performed to accomplish the above items that I haven't listed. For example, verification that the hash in the update snippet matches the mar downloaded.

Comment 20

[Aki Sasaki (not active)]

(In reply to comment #19)
> Verify that the aus certificate is valid by performing the check against
> aus3. The app.update.url preference - not the app.update.url.override
> preference - would need to be set and can be done during test run by just
> setting the default preference via code. Would likely need to have the
> ability to advertise the update snippet to a test url on aus3 that returns
> the same update snippet when it is advertised to the public.

Aiui, we can't change it via code while still testing a real build that we will eventually push to a nightly/beta/release audience. Is that incorrect?

> Test the partial update and verify the files / directories after the update
> are the same as the build. This will catch cases where files aren't added
> due to not being included in the app's package.manifest. I still have some
> work to do to get the directory removal story 100% but it should be good to
> go with mars generated since Firefox 5.0.

This sounds like bug 588396, which seems to be "apply partial MAR against previous nightly and verify it looks like tonight's nightly".

> Test the complete update against the build the partial was applied to and
> verify the files / directories after the update are the same as the build.
> This should catch issues like bug 670468 as well as test that the new
> updater will work for the next nightly build (should already be covered by
> bug 601518 but this is important enough to test here as well IMO).

This seems to be testing the previous night's build, rather than this build, right? Which is valid, but I think this bug and bug 498425 are about "make sure the current build can update to the next thing, if/when the next thing is built and makes valid updates available".

> There are also other items this will cover due to the operations performed
> to accomplish the above items that I haven't listed. For example,
> verification that the hash in the update snippet matches the mar downloaded.

I can certainly expand the suite of tests we can run to cover all the above, and I think it's valuable to. However, is the current scope of "verify this current build I'm about to publish can update to anything" valuable? If so, I'm thinking about continuing with updating to the current complete.mar or perhaps with Ben's suggestion in comment 18.

Comment 21

[Robert Strong (they/them - no direct email)]

(In reply to comment #20)
> (In reply to comment #19)
> > Verify that the aus certificate is valid by performing the check against
> > aus3. The app.update.url preference - not the app.update.url.override
> > preference - would need to be set and can be done during test run by just
> > setting the default preference via code. Would likely need to have the
> > ability to advertise the update snippet to a test url on aus3 that returns
> > the same update snippet when it is advertised to the public.
> 
> Aiui, we can't change it via code while still testing a real build that we
> will eventually push to a nightly/beta/release audience. Is that incorrect?
For the purpose of verifying aus3 and the aus3 certificate I think this would be acceptable.

When I discussed this with a few people a couple of years ago my goal was to make sure we have end to end testing of applying an update. Since this can't be done in the tree and there are mozmill tests that do this for the most part already it seemed to me that it would be best to leverage the existing mozmill tests. As far as I am concerned I'm fine with the tests being broken out into individual tests instead as long as the parts of the process that can't be tested in the tree are tested using whatever test harness you want.

Comment 22

[Robert Strong (they/them - no direct email)]

s/end to end testing of applying an update/end to end testing of applying a *real* update/

Specifically, we do test applying an update with the updater but it isn't possible to test all of the moving parts that are public facing.

Comment 23

[Aki Sasaki (not active)]

Found in triage; not actively working on this.

Comment 24

[[e-mail] (:bhearsum)]

I think mozmill update tests are dead?

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Mozmill is dead for about 3 years, and we stopped firefox-ui-update tests not that long ago and developers request. So yes, we don't need that for the moment.