386359 - need more Ts runs in Talos

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rob Campbell [:rc] (:robcee)]

Currently we're only running Ts 5 times and averaging the results after throwing one away. We should double or quadruple the number of runs to get a more accurate reading on startup time.

Comment 1

[alice nodelman [:alice] [:anode]]

Attachment: upped ts number of cycles

Increased the number of ts runs to 20.  Also, a minor fix to the base profile to set the size of the browser to 1024x768 (in case -height and -width command line options fail).

Comment 2

[Rob Campbell [:rc] (:robcee)]

checkin log:

[e-mail]>, r+=me"
cvs commit: Examining .
cvs commit: Examining talos
cvs commit: Examining talos/base_profile
cvs commit: Examining talos/base_profile/Cache
cvs commit: Examining talos/base_profile/bookmarkbackups
cvs commit: Examining talos/page_load_test
cvs commit: Examining talos/startup_test
cvs commit: Examining win32
cvs commit: Examining win32/base_profile
cvs commit: Examining win32/base_profile/Cache
cvs commit: Examining win32/base_profile/bookmarkbackups
cvs commit: Examining win32/page_load_test
cvs commit: Examining win32/page_load_test/base
cvs commit: Examining win32/page_load_test/base/bugzilla.mozilla.org
cvs commit: Examining win32/page_load_test/base/bugzilla.mozilla.org/res
cvs commit: Examining win32/page_load_test/base/lxr.mozilla.org
cvs commit: Examining win32/page_load_test/base/lxr.mozilla.org/res
cvs commit: Examining win32/page_load_test/base/vanilla-page
cvs commit: Examining win32/startup_test
Checking in talos/config.py;
/cvsroot/mozilla/testing/performance/talos/config.py,v  <--  config.py
new revision: 1.4; previous revision: 1.3
done
Checking in talos/base_profile/localstore.rdf;
/cvsroot/mozilla/testing/performance/talos/base_profile/localstore.rdf,v  <--  localstore.rdf
new revision: 1.3; previous revision: 1.2
done

Comment 3

[alice nodelman [:alice] [:anode]]

Attachment: actual fix to get talos to run more ts runs

Okay, I had updated the TS_NUM_RUNS but then failed to use it as the actual variable to set the number of ts runs.  This fixed that oversight.

Comment 4

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 270757 [details] [diff] [review]
actual fix to get talos to run more ts runs

that looks promising :)

Comment 5

[Reed Loden [:reed]]

Checking in ts.py;
/cvsroot/mozilla/testing/performance/talos/ts.py,v  <--  ts.py
new revision: 1.5; previous revision: 1.4
done

Comment 6

[Myk Melez [:myk] [@mykmelez]]

Note that wikimo describes the Ts test differently.  It says: "The page is loaded 10 times, and the shortest startup time is used as the final score."

i.e. it runs the test 10 times instead of 20 and uses the shortest time rather than the average as the result of the test.

http://wiki.mozilla.org/Performance:Tinderbox_Tests#Ts:_Startup_time

Comment 7

[(not currently active) Ted Mielczarek]

Mass move of Core:Testing bugs to mozilla.org:Release Engineering:Talos. Filter on RelEngTalosMassMove to ignore.