456877 - Can we use MozRunner and JsBridge to do upgrade + mochitest runs?

Status: RESOLVED FIXED

(Testing Graveyard :: Mozmill, defect)

Description

[cmtalbert]

We have a host of issues that only seem to occur and appear after doing an upgrade of the application.  Could we find a way to run our standard set of unit tests in this scenario?

My thought is that somehow we can do the following using the existing mochitest and XPCShell frameworks in combination with jsbridge and mozrunner from the Gristmill project.

1. Install version n-1 with an already seeded profile with some data
2. Restart application
3. Install version n
4. Run version n using profile from step 1, and use it to run mochitests and xpcshell (and in the future, mozmill scripts)
5. Report results

= A note on an idea =
we keep this infrastructure in a VM that we haul out at release time, change the version numbers it points at for n and n-1 and click the proverbial "go" button.  Once we upgrade it to the next version, we snapshot at that point, and save the VM for next time.

= Here is the IRC conversation that spawned the idea =
 You rejoined the room.
[2:33pm] tracy: yeah, what abillings  said
[2:33pm] juanb: smoke/bft updated builds vs. testing around specific bugs from the fixed bugs list
[2:33pm] Tomcat_xp: yeah i mean, well with the current timestamp
[2:34pm] tracy: I'm actually amazed we don't see more update caused bugs like this.
[2:35pm] tracy: and not the "lost bookmarks' type.. but obscure stuff.
[2:35pm] dmose joined the chat room.
[2:36pm] tracy: tchung: please toss this topic in for tomorrows meeting..  we should toss it around a bit.
[2:36pm] tchung: tracy: yep, good one
[2:36pm] Tomcat_xp: tracy: well there is the norton issue that cause some of this bookmark problems
[2:37pm] Aleksej2 joined the chat room.
[2:37pm] Tomcat_xp: when this is resolved we can more analyze whats now norton or whats other issues
[2:37pm] tracy: Tomcat_xp:  that's not the usual "lost bookmark" issue.
[2:38pm] Aleksej2 is now known as IRCMonkey23279.
[2:38pm] abillings: you realize that going forward, the people that will be working on these releases are me, JuanB and whatever third person volunteers.
[2:38pm] bretr joined the chat room.
[2:38pm] tracy: yeah, abillings: it certainly would be a huge hit to QA to add more extensive testing on the post-update side...
[2:38pm] abillings: If people want to sizably increase our testing, they should consider volunteering to help on the releases. 
[2:39pm] IRCMonkey23279 left the chat room. (Quit: ChatZilla 0.9.83 [Firefox 3.0.2/2008091618])
[2:39pm] abillings: mochitests and unit tests in general should be doable, perhaps.
[2:39pm] tracy: yeah, I was just thinking that would be a big step forward in itself
[2:39pm] tchung: we can always worry about resourcing later.  but planning stiffer qa work is always in good discussion
[2:41pm] ctalbert: abillings: you mean mochitests for update testing?
[2:41pm] ctalbert: i.e. migration testing from version to version
[2:41pm] tracy: ctalbert: Not for, just run post-update
[2:41pm] abillings: ways to find bugs like the 3.0.3 bug
[2:42pm] ctalbert: Ah, that'd be a good step, though I think in this particular case there is no mochitest for this bug.  But in general, that'd be a great idea.  I can help with that very easily once Jmaher finishes the debug build VM's he's working on 
[2:42pm] ctalbert: I think anyway...
[2:43pm] tracy: 3.0.x passes, 3.0.x+1 pass. not run update 3.0.x -> 3.0.x+1 to ensure it still passes
[2:43pm] tracy: s/not/now/
[2:43pm] coop is now known as coop_away.
[2:44pm] tracy: passes means any automation tests we can easily push at it should pass.
[2:44pm] ctalbert: is 3.0.x still on CVS?
[2:45pm] abillings: you all realize that neither smoketests or bfts would have caught the bug that caused 3.0.3, right?
[2:45pm] ctalbert: yes, I do
[2:45pm] tracy: yes, I said that already
[2:45pm] abillings: so running our existing tests post-update wouldn't catch things necessarily
[2:46pm] abillings: this is what the betas are supposed to catch
[2:46pm] ctalbert: but in the interest of general stability I can do a unit test run on the 3.0.3 build, if ya'll want.  But doesn't tinderbox already do that though?
[2:46pm] tracy: but if we had an appropriate test in litmus and ran it both before and after update, then it would get cuaght.. but thats a very expensive man hours thing to do.
[2:46pm] Aleksej: Is that about the "lost passwords" complaints?
[2:46pm] abillings: for this one problem but we'll be addressing the existing problem, not the next one
[2:46pm] tracy: ctalbert, no.  start with 3.0.1 then update to 3.0.3, then run tests
[2:47pm] ctalbert: tracy: ah...with an a profile with some interesting stuff in it too.
[2:47pm] tracy: this particular bug only happens when 3.0.1 updated to 3.0.2.  it's fine in 3.0.1 and fine in 3.0.2 wiht fresh profiles
[2:48pm] ctalbert: I'll think about this.  This requires a bit more thought.
[2:48pm] tracy: ctalbert, yep.  a nonascii host name associated to a passowrd
[2:48pm] tracy: starting from a given profile isn't a problem.
[2:49pm] ctalbert: All the mochitests and unit tests automatically start in their own shiny new profile
[2:49pm] tracy: oh really?  they can't be forced to start from a designated profile?
[2:50pm] ctalbert: no.  That's where mozrunner and jsbridge can come in
[2:50pm] ctalbert: It can be done, but it requires some hackery.
[2:50pm] Tomcat_xp: heh
[2:50pm] Tomcat_xp: yeah
[2:50pm] ctalbert: fusing the existing tools together
[2:50pm] tracy: that does pose a problem in this case then.
[2:50pm] ctalbert: requires some thought.
[2:50pm] Tomcat_xp: thats the problem i run with the extension mochikit leak test
[2:51pm] Major|Broke joined the chat room.
[2:51pm] ctalbert: I think that we can use mozrunner and jsbridge to help solve this (two of the pieces of Gristmill).  mozrunner can start in any designated profile with any set of extensions
[2:51pm] ctalbert: And jsbridge can allow us the remote control needed to kick off the tests once it's launched
[2:52pm] ctalbert: and to automate the upgrade/restart of the browser
[2:52pm] tracy: cool, I think we'd get a good win from that sort of testing post-update.

Comment 1

[Mikeal Rogers [:mikeal] (mrogers@)]

There aren't any technical hurdles I can see in getting jsbridge to open the browser and getting MozMill to update the browser then restarted and running something.

The big hurdle here is the mochitests. Currently they don't run against a release build, they have to run against special "test" builds. The MozMill test framework that will be dropping in RC1 can handle integration like this in to another test framework, so we could theoretically modify all the mochitest framework stuff to call in MozMill which would offer simple debugging and calls back to jsbridge.

-Mikeal

Comment 2

[Mikeal Rogers [:mikeal] (mrogers@)]

We need to talk about how we want to handle this.

I see two paths for adding this kind of functionality

1) Add restart support to mozmill 
   - Add an API function to trigger a restart
   - Serialize most of the framework state and send to jsbridge
   - jsbridge gets a restart event, kills the browser, starts back up, sends serialized state information

2) Create another small test harness for restart specific testing. This could still use mozmill for the actually pre and post restart testing, it would just need to seperate the pre and post tests in to different isolated test directories.

The first option is harder but notably easier to write and integrate tests in. The second option is easier but creates another way of writing tests.

My main question is this. Will all the restart tests be separated from the other functional tests we run?

Restart tests will take longer to run and be a little more error prone than other tests, so part of me thinks that restart test runs may be separated from the other functional tests.

Comment 3

[Tracy Walker [:tracy]]

For running functionality test cases,, since it will be massively difficult to have MozMill detect what a test case failure has caused, I suggest a restart of the app to get the app back into a clean functional state before proceeding to the next test case. This should just happen without having to write something specific into the test case (other than passing a result back to the test harness if necessary)

Comment 4

[cmtalbert]

I think that Tracy's idea in comment 3 can be adequately handled if we have option 1 (add restart functionality to mozmill api) and those are placed in the teardown wrapper of the testcases that are likely to cause this.  We already have startup/teardown wrappers, so it seems like the best approach. Adding yet another way to write test cases seems like more complexity for less benefit.  Also, we can always separate the tests that we know have restarts (like session restore tests) into their own subgroup during a test run anyway regardless of which option we take in comment 2.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mikeal, what's the current status of this bug? Can you set a priority?

Comment 6

[Mikeal Rogers [:mikeal] (mrogers@)]

So, we have the mozmill restart testing now which should enable this testing.

I don't see any reason to use mochitest at all, this can be done solely with the mozmill-restart testing as I understand this problem.

As noted by Henrik the current limitation is that it only supports a single restart per test.

Is there anything else that is required for this bug?

Comment 7

[cmtalbert]

(In reply to comment #6)
> So, we have the mozmill restart testing now which should enable this testing.
> 
> I don't see any reason to use mochitest at all, this can be done solely with
> the mozmill-restart testing as I understand this problem.
> 
The reason to use mochitest was to run a base of TUnit on the updated build, which can be done if we have the "Update" process performed through mozmill, and we have Ted's "Run TUNit on packaged builds" infrastructure done.

There is nothing required for this from a mozmill perspective once we have restart testing.

Comment 8

[Mikeal Rogers [:mikeal] (mrogers@)]

Fixed with the new restart testing rig.