346785 - Firefox needs a performance testing framework for Windows

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Annie Sullivan]

I've been working on one for an extension, so I thought I might as well post it so other people can use it too.

Comment 1

[Annie Sullivan]

Attachment: First pass at performance testing framework for Win32

I'm not sure exactly where this should go. I put it under testing/performance/win32 hoping that we can factor out the platform-independent parts into testing/performance/ and put other modules that implement the same functions in testing/performance/linux/ and testing/performance/osx

There are a few known issues with this patch:
1. It's a little flaky. It always runs great on the machine I developed it on, but sometimes Firefox doesn't pick up the new profile I create or I don't get all of the dump() output on other machines. Hopefully having it checked in will allow more people to run it on their machines and help fix it up.

2. I checked in some mozilla websites (old copies of lxr.mozilla.org and bugzilla.mozilla.org) as sample data for the page load test. Hopefully we'll be able to get better data soon.

Comment 2

[Annie Sullivan]

Attachment: Sample report generated with the sample.config file

The graph isn't very good because there aren't enough test pages for the page load test yet.

Comment 3

[Annie Sullivan]

Somebody asked me how this is different from Tinderbox. There are two major differences. First, it doesn't build, it just runs the performance test given a path to the executable. This is helpful if you're testing the performance of an extension or a build from another server. (You could build on a fast server, and then run performance tests on a machine with low memory). Second, it measures performance characteristics while it's running the pageload tests--you can track cpu speed, memory, or any of the other counters listed here: http://technet2.microsoft.com/WindowsServer/en/Library/86b5d116-6fb3-427b-af8c-9077162125fe1033.mspx?mfr=true

Comment 4

[Annie Sullivan]

I'm marking this "fixed" since the first pass is checked in. We'll file other bugs for refactoring, feature additions, and bug fixing as they happen.