If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Use VMware VMs to run mochitests, optionally record and repeat until they fail

RESOLVED FIXED

Status

Testing
Mochitest
RESOLVED FIXED
8 years ago
3 years ago

People

(Reporter: Ben Turner (not reading bugmail, use the needinfo flag!), Assigned: Ben Turner (not reading bugmail, use the needinfo flag!))

Tracking

Trunk
x86
Windows 7
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 2 obsolete attachments)

Created attachment 432430 [details] [diff] [review]
Patch

Attached patch runs mochitests over and over again under VMware until they fail. Recordings are deleted after each successful run so that they don't build up. Requires the patch from bug 551256 as well.
There are still a few little loose ends to work out so not requesting review just yet. Right now this patch assumes that you've got the sharing set up so that C:\ in the host maps to Z:\ in the guest. We could do all that dynamically, I guess, but it seems complicated (not only do you have to tell vmware what to do to make the shared folder but then you have to tell windows to map it to a drive letter...). Also I have the shutdown stuff disabled for the moment. It works but it's annoying to restart the VM every time during testing.
Assignee: nobody → bent.mozilla
Status: NEW → ASSIGNED
Oh, so, to use this you use runtests.py just like normal but you specify additional options:

--with-vmware-vm=/path/to/vmx
    Tells runtests to launch the vm. It assumes vmrun.exe is in the standard spot.
--with-wmrun-executable=/path/to/vmrun.exe
    Not necessary unless vmrun.exe is in not in the default path.

--repeat-until-failure
    Optional, it runs over and over until the tests fail. The VM is not
    restarted between runs, but the guest mochitest framework is.

--use-vmware-recording
    From bug 551256, records execution of guest mochitests. Really this
    should always be required if --repeat-until-failure is specified,
    otherwise you won't know anything except that the tests eventually
    failed.
Looks cool!

runProgramInVM is probably misnamed, since commands like "start" aren't actually running a program in the VM.

My preferred drive-letter setup is to have C: and Z: partitions in the host, and mount host Z: as guest Z:. Most of my data lives on Z:. I have scripts that copy the code and data needed for running tests from Z: into a directory on C:, and I run those scripts in both the host and guest. (I could skip doing this and just run the tests from Z: but that's awfully slow and leads to much larger recordings.) This guarantees that file paths on host and guest are always identical, which I find helps a lot.

Another option here would be to have a separate wrapper script that launches tests in a VM and optionally repeats them.
Woo, this recorded bug 552054 on run #50!
Created attachment 432844 [details] [diff] [review]
Patch, v1

Ok, this is cleaned up slightly. Ted, what do you think? Is this too much for runtests.py to handle?
Attachment #432430 - Attachment is obsolete: true
Attachment #432844 - Flags: review?(ted.mielczarek)
(In reply to comment #3)
> runProgramInVM is probably misnamed, since commands like "start" aren't
> actually running a program in the VM.

Hm... Yeah. "runVMCommand"?

> I have scripts that
> copy the code and data needed for running tests 

Yeah, that's an option too. I find that the nightly builds run just as fast over the VMware network as if I had copied them. Source builds, on the other hand, take forever. No idea what the difference is.
(In reply to comment #6)
> (In reply to comment #3)
> > runProgramInVM is probably misnamed, since commands like "start" aren't
> > actually running a program in the VM.
> 
> Hm... Yeah. "runVMCommand"?

Sure.
Comment on attachment 432844 [details] [diff] [review]
Patch, v1

After looking at this, I'm not wild about having it in runtests.py. It's a lot of added complexity.

What you could do instead, since jmaher went and refactored it all into classes, is create a new file that subclasses the MochitestOptions and Mochitest classes from runtests.py, and adds your own options/logic. That I would fully support. Maybe runtestsvmware.py?
Attachment #432844 - Flags: review?(ted.mielczarek) → review-
Created attachment 437433 [details] [diff] [review]
Patch, v2

Ok, here it is as a separate runtestsvmware.py script.
Attachment #432844 - Attachment is obsolete: true
Attachment #437433 - Flags: review?
Attachment #437433 - Flags: review? → review?(ted.mielczarek)
Ted, what's the status on this?  I for one would love to see it in the tree, and given everything else I have to do I likely won't get to orange-hunting until it is.
Sorry, lost track of my remaining review requests over paternity leave. Trying to catch up, but just haven't gotten there yet. I'll try to clear it out this week.
Comment on attachment 437433 [details] [diff] [review]
Patch, v2

Thanks, that's a lot nicer as a separate script! There are still some hacky bits in there, but since it's a pretty special-purpose script they don't bother me very much.

You should land this and get some feedback from other people on how it fits their workflow.
Attachment #437433 - Flags: review?(ted.mielczarek) → review+
Thanks Ted!

http://hg.mozilla.org/mozilla-central/rev/cf5dcc522934
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Sorry to revive this :)

So, because of the structured logging patch (bug 886570), runtestsvmware.py would probably require some changes (as it's currently parsing strings) to work. I don't have a Windows machine and can't test it, but seeing how it was last updated in 2010, the chances this script is still working are really small.

Should we try to fix it up or remove it from the tree? (seeing how no one complained after the patch landed, I'm not sure anyone uses it anymore)
Flags: needinfo?(bent.mozilla)
VMware no longer supports record and replay.  (We should try to get http://rr-project.org/ working on the build machines at some point!)
Yeah, it's not worth keeping this script going.
Flags: needinfo?(bent.mozilla)
You need to log in before you can comment on or make changes to this bug.