1549417 - Testing of profile upgrades from previous versions needed in automation

Status: NEW

(Toolkit :: Startup and Profile System, enhancement, P1)

Description

[Randell Jesup [:jesup] (needinfo me)]

The latest issue with the LSNG changes needing to be backed out (bug 1549362) and nightly respun points out that we don't have good (any?) coverage of profile-upgrade in the tree.

When something like this changes, there should be smoketests in automation that verify that profiles last touched in Release/Beta/Nightly/ESR/previous-ESR/last-N-releases don't fail in some basic manner.

This would not include any downgrade testing (users are no longer supposed to do that, and if they force it that's on them).

We can't run the full suite of tests for this, and may not want to run it on every checking, but some basic test of "the upgrade succeeded with no unexpected errors thrown, and the browser can still browse and the DBs are all still sane with no lost data" would be good.

Doing this at least once before merge-to-m-c (and perhaps a fuller test before merge-m-c-to-beta) would be good.

It might be nice-to-have to be able to run (most of?) the automation suite with a provided profile instead of an effectively new profile, which would allow extensive tests to be run before a merge-to-beta or release. This should be a follow-on bug if we think it's useful.

(it's also possible there might be perf issues that only exist with a previous profile with real data in it)

Comment 1

[Dave Townsend [:mossop]]

Yeah I've been thinking a little about this lately too. A challenge is that to be a useful test the old profile needs to have some data in it, and all of our tests in automation right now assume a fairly clean profile, adding any test data they want at the start of the test. So this would probably involve fixing a lot of tests to cope with data they don't expect right now as well as adding a bunch of tests to verify that the upgraded profile contains the data expected from the old profile.

As you say though, at the bare minimum, a check that the browser can still browse would certainly be helpful.

Not entirely sure where this bug should live but Testing seems like a reasonable place to start.

Comment 2

[Jan Varga [:janv]]

A generic solution would be great, but for now I'll just use existing profile upgrade testing infrastructure which we already have for QuotaManager, IndexedDB, LSNG and other storage related stuff.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Yes, this would be great to have, particularly rolling nightly profiles. Bug 1458320 is another relevant example where successive changes to nightly disk format (bug 1434308 then bug 1456604) resulted in failures for nightly users updating regularly but was not a problem for fresh profiles or profiles older than a few days.

This would likely also be a useful thing for evaluating potential backouts. LSNG took great care to handle downgrade/backout scenarios so the backout was not a problem. But as more and more data is stored on disk (ex: rkv), it seems possible that other storage-using code might have a bad landing that wants to be backed out, but could end up making things worse.

Comment 4

[Markus Stange [:mstange]]

I'm QF-minusing this bug because I think this bug should be about functionality tests and not about performance tests. We can file a new bug if we really want to test the performance of profile upgrades.

Comment 5

[Selena Deckelmann :selenamarie :selena]

Having a test here would have prevented a very unpleasant regression where Nightly didn't render. I'd like a test that renders a page with text and an image from a local webserver, and validates that some load time occurred. Not sure what's required to ensure we trigger storage bugs, but I imagine that we need a profile that's loaded the page previously at a minimum. Given that this needs some scoping and design, I'd like engineering to own this implementation.