1414571 - there should be continuous integration tests for the profile manager

Status: NEW

(Toolkit :: Startup and Profile System, enhancement)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

We just shipped a crash bug on nightly (bug 1414513) that I believe crashes all users who see the profile manager on startup.  I suspect that these users are stranded on the crashing build unless they do something to disable the profile manager and start up, or manually download nightly again.

It seems bad that starting up via profile manager isn't tested in continuous integration.  It should be.

Can somebody own this?

Comment 1

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking this for 58 to make sure we follow up (if not for 58, maybe for 59)

Comment 2

[Dave Townsend [:mossop]]

I'm adding a limited set of profile service tests in bug 1373244 but it doesn't cover the full case of opening the profile manager on startup. This is extremely difficult to test properly, we don't have any of the usual options like injecting add-ons etc. into the running process to act as a test harness. We might need to write custom test code on all three platforms in order to do this properly. Either that or embed the test code in the product and activate it with an environment variable or something.

Do you have any thoughts here Jim?

Comment 3

[Jim Mathies [:jimm]]

(In reply to Dave Townsend [:mossop] from comment #2)
> I'm adding a limited set of profile service tests in bug 1373244 but it
> doesn't cover the full case of opening the profile manager on startup. This
> is extremely difficult to test properly, we don't have any of the usual
> options like injecting add-ons etc. into the running process to act as a
> test harness. We might need to write custom test code on all three platforms
> in order to do this properly. Either that or embed the test code in the
> product and activate it with an environment variable or something.
> 
> Do you have any thoughts here Jim?

I don't know much about the boot strapping of the profile manager. I do know we don't officially support it, and we have this in-tab version that we were moving toward but isn't complete.

Long term I think we should look at deprecating the dialog based profile manager and (re)prioritize the in-tab version which should be more test framework friendly.

For selecting profiles and restarts and such, why not rely on a good set of manual QA test cases? There's not much to test here. Authoring custom testing infra for this seems like overkill.

Comment 4

[Dave Townsend [:mossop]]

(In reply to Jim Mathies [:jimm] from comment #3)
> (In reply to Dave Townsend [:mossop] from comment #2)
> > I'm adding a limited set of profile service tests in bug 1373244 but it
> > doesn't cover the full case of opening the profile manager on startup. This
> > is extremely difficult to test properly, we don't have any of the usual
> > options like injecting add-ons etc. into the running process to act as a
> > test harness. We might need to write custom test code on all three platforms
> > in order to do this properly. Either that or embed the test code in the
> > product and activate it with an environment variable or something.
> > 
> > Do you have any thoughts here Jim?
> 
> I don't know much about the boot strapping of the profile manager. I do know
> we don't officially support it, and we have this in-tab version that we were
> moving toward but isn't complete.
> 
> Long term I think we should look at deprecating the dialog based profile
> manager and (re)prioritize the in-tab version which should be more test
> framework friendly.

Unless we want to stop supporting it we still need the dialog version to a.low users who want to to select a profile on startup.

> For selecting profiles and restarts and such, why not rely on a good set of
> manual QA test cases? There's not much to test here. Authoring custom
> testing infra for this seems like overkill.

The downside with bug 1373244 is that the startup behaviour becomes more complicated so the number of QA tests goes up. And we can't run QA tests on every nightly which is what would have been required to catch the bug that this was filed to try to prevent.

Comment 5

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Update_affecteds

Comment 6

[Julien Cristau [:jcristau]]

(In reply to David Baron :dbaron: ⌚️UTC-8 from comment #0)
> Can somebody own this?

It doesn't sound like anyone's in a hurry to do that :/

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.