Closed Bug 97712 Opened 23 years ago Closed 18 years ago

Check in perl script to run startup-test

Categories

(SeaMonkey :: General, defect)

Product:
SeaMonkey
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mcafee, Assigned: jrgmorrison)

Details

Attachments

(2 files)

[win32] script for starting mozilla and timing startup; optionally can run in a loop
3.54 KB, text/plain
Details
comment in script should say ms, not sec
496 bytes, text/plain
Details
Reminder to check in perl script to run startup-test. mozilla/builds/startup-test.html needs a small perl wrapper to start mozilla with the html file and the current time as an ? argument. Snippet from tinderbox test is a basic start (jrgm has xp and time resolutions to address in a better form): # Generate URL of form file:///<cwd>/startup-test.html?begin=986869495000 # Where begin value is current time. my ($time, $url, $cwd, $cmd); $time = time() . "000"; # looks stupid, but 'time()*1000' returns a negative $cwd = Cwd::getcwd(); print "cwd = $cwd\n"; $url = "\"file:$binary_dir/startup-test.html?begin=$time\""; print "url = $url\n";
Another simple way that I run startup test is : - Create a default profile 'Startup Test Profile' - save this as say /usr/tmp/quit.html <html><body onLoad="window.close()"></body></html> - run $ time ./mozilla -P 'Default User' file:///usr/tmp/quit.html
Chris, here's a tweak for the tinderbox script to allow it to conditionally use high resolution time from perl on linux (I'm punting on making this share with win32 for now; would need to much perl-fu to deal with all the different win32-only modules used to launch a process). Add this section to the same block where you have that other code: # Linux/Unix version # Use high resolution routines on Linux if installed, using # eval as try/catch block around import of gettimeofday # Get 'Time::HiRes' at <http://search.cpan.org/search?dist=Time-HiRes> eval "use Time::HiRes qw(gettimeofday);"; my $timesub = $@ ? sub { time() . "000"; } : sub { my @t = gettimeofday(); $t[0]*1000 + int($t[1]/1000); }; and then make this change to the current tinderbox code - $time = time() . "000"; # ... + $time = &$timesub;
Attachment #47781 - Attachment description: script for starting mozilla and timing startup; optionally can run in a loop → [win32] script for starting mozilla and timing startup; optionally can run in a loop
You know, the more I look at dp's way, the more I like it. Unfortunately it seems to crash-on-exit for me about half the time (some event thing in nspr), but you can still get numbers simply. Including shutdown time in the measurement appears to add about 13% to the overall time (startup dominates). Alternatively, the attached script (for windows) is pretty simple (if you don't look at it) and only includes startup to the first onload event. Tomato, Tomahto.
dp's way times startup and shutdown, right?
Yup. my method times startup and shutdown. So it is not accurate if you are looking for startup time. But usually the shutdown component of this timing is a constant with and without most perf improvements. So this works heuristically mos to the time. The one thing I have learnt is the simpler the method, the more easier and widely will it be adopted. People are not interested in learning complicated methods for doing performance timing.
Yeah, I like dp's method. Easy to setup (well, except Mac), and the shutdown cost is a minor portion of the total time measured.
win32 script has some errors for me on rh7.1. Attaching a simple script that works on linux, standalone version of what tinderbox is doing.
Comment on attachment 48018 [details] comment in script should say ms, not sec attachment is a file, not a patch, testing new bugzilla feature
Attachment #48018 - Attachment is patch: false
Comment on attachment 48018 [details] comment in script should say ms, not sec >#!/usr/bin/perl > ># ># Script to time mozilla startup. ># > >require 5.003; > >use strict; >use Cwd; > ># ># mozilla file:/foo/startup-test.html?begin=T ># where T = ms since 1970, e.g.: ># mozilla file:/foo/startup-test.html?begin=999456977000 ># > >{ > # Build up command string. > my $cwd = Cwd::getcwd(); > my $time = time() . "000"; # time = sec, *1000 for ms. > my $cmd = "mozilla \"file:$cwd/startup-test.html?begin=" . $time . "\""; > print "cmd = $cmd\n"; > > # Run the command. > exec $cmd; >}
Attachment #48018 - Attachment description: linux script to time mozilla startup → comment in script should say ms, not sec
adding smfr for some mac help, mac should also have a wrapper script that can do this.
jrgm has copies of my Mac perl scripts.
just a quick comment: has anyone looked at the distribution of startup times collected using this method? In another words, I'm wondering what the error associated with this measurement method has been in your expersience. I'd have some data from two W2K machines and statistical evaluation, if there is some interest. The data indicates that the measurement error might be in the 10% range. Has anyone else seen this or am I missing something?
http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey openwound is running the startup test (warm) on the front page of tbox, the numbers are pretty consistant. 22 +/- 1.5 seconds it looks like, something like that?
Chris, thanks for pointing this out! Seems like openwound is roughly three times more accurate than my W2K machines. I assembled a quick sampling of 60 data points and popped it into http://school.discovery.com/homeworkhelp/webmath/adata.html. I verified that the data sample is normally distributed and it seems like the margin of error is 3% with a 95% confidence interval. average value: 21.95 standard deviation: 0.351 margin of error (95%): 2x0.351/21.95 I was just trying to raise the issue of accuracy, since I'm struggling to get an idea of the performance gain in bug 97175 and the margin of error I have been able to achieve locally was consistently around 10%.
Unless you install certain perl modules, you will get the default available time resolution for perl which is one second. That is the source of error. If you use dp's method on win32, you can use the cygwin version of 'time' to get finer resolution.
>Unless you install certain perl modules, you will get the default available >time resolution for perl which is one second. That is the source of error. jrgm, could you please elaborate on this? What modules are needed and where does one find them? I believe it to be helpful for anyone attempting to use this metrics locally. As to the 1 s resolution: my average startup time was 5.6 s with a standard deviation of 280 ms, which might be consistent with your statement since the error band (95% confidence level) is 4 * 0.28 s ~ 1 s. This would also imply higher accuracy on slower machines, right? If my startup time is say 4 times longer (~ 20 s), when ignoring all other sources of noise I'd arrive at 4 times higher accuracy. I assume that you used said modules on openwound, but in theory the improved accuracy could also be a result of longer startup time...
On *nix systems, millisecond time is available in perl with the Time::HiRes module. On win32, there is no direct call available, but there is a way to call the win32 GetLocalTime directly, by using the Win32::API module. Win32::API is available at <http://www.activestate.com/PPMPackages/zips/5xx-builds-only/Win32-API.zip> Time::HiRes is available at <http://search.cpan.org/search?dist=Time-HiRes> and attachment http://bugzilla.mozilla.org/attachment.cgi?id=47781&action=view has a module that uses those modules if it finds them. But, I find dp's way to be the more direct path to getting a measurement (on win32 and linux). The other stuff will be useful for doing a more detailed test, but I've not done the work to make this happen. Yes, as the machine is slower, then 1 second becomes less of a percentage error. I believe openwound is an old, slower machine (possibly a 166MHz), and that's why it is more 'accurate'.
I checked in jrgm's Time::PossiblyHiRes script into tinderbox: mozilla/tools/tinderbox/gettime.pl and restarted openwound to use this script. I think it's working, let's watch the numbers now. Thanks to jrgm for doing most of the guts work here.
please check files into mozilla/tools/performance/startup
checked in copy of gettime.pl, and new file startup-unix.pl @ mozilla/tools/performance/startup. jrgm, could you edit your win32 version to possibly use gettime.pl here?
what's left to do on this bug? Linux is done, win32/mac ?
Product: Browser → Seamonkey
I think this bug is mostly fixed, file specific bugs if some parts still need to happen.
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: