create new tp pageset that is newer and smaller

RESOLVED FIXED

Status

defect
P2
normal
RESOLVED FIXED
11 years ago
6 years ago

People

(Reporter: anodelman, Assigned: anodelman)

Tracking

Dependency tree / graph

Firefox Tracking Flags

(status1.9.1 ?)

Details

Attachments

(8 attachments, 3 obsolete attachments)

7.90 KB, patch
catlee
: review+
anodelman
: checked-in+
Details | Diff | Splinter Review
40.50 KB, patch
catlee
: review+
anodelman
: checked-in+
Details | Diff | Splinter Review
2.01 KB, patch
catlee
: review+
anodelman
: checked-in+
Details | Diff | Splinter Review
4.21 KB, patch
catlee
: review+
anodelman
: checked-in+
Details | Diff | Splinter Review
6.96 KB, patch
catlee
: review+
anodelman
: checked-in+
Details | Diff | Splinter Review
62.93 KB, patch
catlee
: review+
anodelman
: checked-in+
Details | Diff | Splinter Review
1.18 KB, patch
catlee
: review+
Details | Diff | Splinter Review
5.25 KB, patch
catlee
: review+
anodelman
: checked-in+
Details | Diff | Splinter Review
Replace the currently used Tp3 pageset with one based upon the current alexa top 500, also attempt to get under 100 pages to decrease the cycle time of talos boxes without reducing the trustworthiness of the data collected.
That's one way to reduce Tp, I suppose .. ;p
(In reply to comment #2)
> Replace the currently used Tp3 pageset with one based upon the current alexa
> top 500, also attempt to get under 100 pages to decrease the cycle time of
> talos boxes without reducing the trustworthiness of the data collected.

Some Talos suites give a wide variety in performance results. To compensate, for those suites, we run the suite multiple times and then average the results. But this means running the entire test multiple times!! :-(

1) Can we find+fix why some pages caused that much variance?
2) If some pages by design will always have lots of variance, maybe we should consider removing those specific pages from the performance suites? 

If this was done, we could reduce the number of iterations of the complete set we have to run when running Talos. Reduced iterations == reduced time == happy! :-) 


/me pushes bug over to Alice, as this is one of her Q1 goals.
Assignee: nobody → anodelman
Priority: -- → P3
Assignee

Updated

11 years ago
Priority: P3 → P2
Assignee

Comment 3

11 years ago
Grabbing a new set of the alexa top 100 (minus all the google duplicates).  I'll clean it up and have it running on talos stage - from there we'll be able to determine if we can get as good regression tracking out of smaller set of pages.

This is an initial easiest-thing-that-could-possibly-work solution, instead of attempting to pick and choose web pages just limit us to a total of 100 instead of 500.
Assignee

Comment 4

11 years ago
As a reference for 'interesting' pages see bug 419105.
Assignee

Comment 5

10 years ago
Whittled down the top 500 pages to 100 pages using a simple scheme:

- remove copies of google (seem to have about 40 copies in the set from different locales)
- remove broken pages (lost layout, missing images, broken links, redirects, popups, etc)
- remove pages that are just simple login screens
- favor pages with lots of content
- favor pages in different languages, attempting to get the greatest variety in the final page set

I'll be putting the new page set on all the Talos staging boxes and letting it run for a while.  Hopefully they'll be a regression and we'll be able to observe if it appears in the new page set (the only time that I hope for a performance regression).

Also, from my initial runs on linux the 100 page set takes approx. 20 minutes to run to completion.  This is still doing 10 full cycles through the 100 pages.
Assignee

Comment 7

10 years ago
Hitting a snag with the new pageset on tiger/leopard - seems to freeze up on some international pages.  Will have to investigate if this is the pageset or the browser.
Assignee

Updated

10 years ago
Depends on: 483902
Assignee

Comment 8

10 years ago
There was a Tp regression observed yesterday, it also shows up on the linux box using the new page set:

http://graphs-stage.mozilla.org/graph.html#show=454230&sel=1238294359,1238613420

I think that this is good proof that the new page set collects the same sort of data as the old page set - but far more quickly.
Assignee

Updated

10 years ago
Depends on: 492671

Updated

10 years ago
Assignee: anodelman → nobody
Component: Release Engineering: Talos → Release Engineering

Updated

10 years ago
Assignee: nobody → anodelman
Assignee: anodelman → nobody
Assignee: nobody → anodelman
Assignee

Comment 9

10 years ago
Current status:

- new pageset runs to completion on all 5 supported talos platforms as long as upgraded flash is installed.

What we are waiting on for roll-out:

- solution to getting new flash on the talos slaves, hopefully through bug 475383
(Mentioned this to Vlad today, actually, figured he'd appreciate the cc)
Will need to be landed with matching buildbot changes that will have pagesets downloaded per run into page_load_test/tp3 and page_load_test/tp4 directories, also with updates to download newer flash plugins into the base_profile directory.
Attachment #384155 - Flags: review?(catlee)
Attachment #384153 - Flags: review?(catlee) → review+
Comment on attachment 384153 [details] [diff] [review]
[Checked in]add tp4 test to graph server

>diff -r b13b2f40f565 sql/data.sql
>--- a/sql/data.sql	Wed Jun 10 11:38:35 2009 -0700
>+++ b/sql/data.sql	Fri Jun 19 13:59:21 2009 -0700
>@@ -278,6 +278,7 @@
> insert into pagesets values (NULL,"SVG");
> insert into pagesets values (NULL,"Dromaeo");
> insert into pagesets values (NULL,"SunSpider");
>+insert into pagesets values (NULL, "Tp February, 2009 (100 pages)");

Can we call this "Tp4 February, 2009 (100 pages)", or "Tp #4 February, 2009 (100 pages)"?
Attachment #384155 - Flags: review?(catlee) → review+
Make it so that we can download and unzip plugins and pagesets per-talos run.
Attachment #384994 - Flags: review?(catlee)
Also using defaults for location of plugins/pagesets - each talos buildbot will have a local directory called 'zips' where these things can be stored.
Attachment #384994 - Attachment is obsolete: true
Attachment #384997 - Flags: review?(catlee)
Attachment #384994 - Flags: review?(catlee)
Just for moz-central to start and do some baking.
Attachment #384999 - Flags: review?(catlee)
Comment on attachment 384999 [details] [diff] [review]
turn on tp4 on moz-central exlusively 

>+# How many tp4 tests per build to run, and whether to merge build requests
>+BRANCHES['mozilla-central']['tp4'] = (1,True)

This needs to be set for the other branches, and maybe should be called 'tp4_tests' to be consistent with the other flags.

>diff -r 3bf03608f457 talos-pool/master.cfg
>--- a/talos-pool/master.cfg	Wed Jun 24 18:56:34 2009 +1200
>+++ b/talos-pool/master.cfg	Wed Jun 24 16:00:53 2009 -0700
>@@ -172,6 +172,40 @@
>                 c['builders'].append(builder)
>                 branch_builders.append(builder['name'])
> 
>+            if tp4_tests:

You need to be setting tp4_tests somewhere, so you probably need a line like

tp4_tests, tp4_merge = branch_config['tp4_tests']
Attachment #384999 - Flags: review?(catlee) → review-
Attachment #384997 - Flags: review?(catlee) → review+
Comment on attachment 384998 [details] [diff] [review]
same updates for those talos set ups not using buildbotcustom

Is UnpackFile available to use in talos-staging, talos, and talos-try?
Attachment #384999 - Attachment is obsolete: true
Attachment #385465 - Flags: review?(catlee)
Attachment #385465 - Flags: review?(catlee) → review+
Ah, cut and paste errors.  Should be fixed now.
Attachment #384998 - Attachment is obsolete: true
Attachment #385473 - Flags: review?(catlee)
Attachment #384998 - Flags: review?(catlee)
Attachment #385473 - Flags: review?(catlee) → review+
Assignee

Updated

10 years ago
Attachment #384153 - Attachment description: add tp4 test to graph server → [Checked in]add tp4 test to graph server
Attachment #384153 - Flags: checked‑in+ checked‑in+
Comment on attachment 384153 [details] [diff] [review]
[Checked in]add tp4 test to graph server

836396688321
2009-06-24 16:29 -0700
Comment on attachment 384155 [details] [diff] [review]
[Checked in]add tp4 test to talos 

/cvsroot/mozilla/testing/performance/talos/sample.config,v  <--  sample.config
new revision: 1.30; previous revision: 1.29
done
Removing page_load_test/manifest.txt;
/cvsroot/mozilla/testing/performance/talos/page_load_test/manifest.txt,v  <--  manifest.txt
new revision: delete; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/tp3.manifest,v
done
Checking in page_load_test/tp3.manifest;
/cvsroot/mozilla/testing/performance/talos/page_load_test/tp3.manifest,v  <--  tp3.manifest
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/tp4.manifest,v
done
Checking in page_load_test/tp4.manifest;
/cvsroot/mozilla/testing/performance/talos/page_load_test/tp4.manifest,v  <--  tp4.manifest
initial revision: 1.1
done
Attachment #384155 - Attachment description: add tp4 test to talos → [Checked in]add tp4 test to talos
Attachment #384155 - Flags: checked‑in+ checked‑in+
Comment on attachment 384997 [details] [diff] [review]
[Checked in]fix for name of downloaded fast cycle manifest

changeset:   353:567121bf725b
Attachment #384997 - Attachment description: fix for name of downloaded fast cycle manifest → [Checked in]fix for name of downloaded fast cycle manifest
Attachment #384997 - Flags: checked‑in+ checked‑in+
Comment on attachment 385465 [details] [diff] [review]
[Checked in]turn on tp4 on moz-central exclusively (take 2)

changeset:   1247:1f6fc287876d
Attachment #385465 - Attachment description: turn on tp4 on moz-central exclusively (take 2) → [Checked in]turn on tp4 on moz-central exclusively (take 2)
Attachment #385465 - Flags: checked‑in+ checked‑in+
Comment on attachment 385473 [details] [diff] [review]
[Checked in]same updates for those talos set ups not using buildbotcustom (take 2)

changeset:   1248:ce8bec434f9f
Attachment #385473 - Flags: checked‑in+ checked‑in+
Assignee

Updated

10 years ago
Attachment #385473 - Attachment description: same updates for those talos set ups not using buildbotcustom (take 2) → [Checked in]same updates for those talos set ups not using buildbotcustom (take 2)
Bustage fix for space issues on fast talos boxes:

changeset:   1249:f9b009d30b6a
tag:         tip
user:        Alice Nodelman <alice@mozilla.com>
date:        Tue Jun 30 19:24:00 2009 -0700
summary:     Bug 473821 - create new tp pageset that is newer and smaller (p/r=anodelman) bustage fix for space problems on some talos boxes
Same bustage fix for try

changeset:   1250:0f065cc25be4
tag:         tip
user:        Alice Nodelman <alice@mozilla.com>
date:        Tue Jun 30 20:27:39 2009 -0700
summary:     Bug 473821 - create new tp pageset that is newer and smaller (p/r=anodelman) bustage fix for downloading zip on try talos
Tp4 now up and running on moz-central.
Fixes behavior on machines where we are not downloading the pageset per-run.
Attachment #386336 - Flags: review?(catlee)
Attachment #386336 - Flags: review?(catlee) → review+
Comment on attachment 386336 [details] [diff] [review]
[Checked in]use installed pageset for running tp3 instead of downloaded

Checking in tp3.manifest;
/cvsroot/mozilla/testing/performance/talos/page_load_test/tp3.manifest,v  <--  tp3.manifest
new revision: 1.2; previous revision: 1.1
done
Attachment #386336 - Attachment description: use installed pageset for running tp3 instead of downloaded → [Checked in]use installed pageset for running tp3 instead of downloaded
Attachment #386336 - Flags: checked‑in+ checked‑in+
Comment on attachment 386368 [details] [diff] [review]
don't download plugins.zip/pagesets/zip except for tp4 builders

Please expand the tabs into spaces.

Looks good otherwise!
Attachment #386368 - Flags: review?(catlee) → review+
Comment on attachment 386368 [details] [diff] [review]
don't download plugins.zip/pagesets/zip except for tp4 builders

>diff -r 0f065cc25be4 talos-pool/master.cfg
>--- a/talos-pool/master.cfg	Tue Jun 30 20:27:39 2009 -0700
>+++ b/talos-pool/master.cfg	Wed Jul 01 14:22:23 2009 -0700
>@@ -117,6 +117,8 @@
>                     configOptions=TALOS_CONFIG_OPTIONS,
>                     talosCmd=TALOS_CMD,
>                     fetchSymbols=branch_config['fetch_symbols'],
>+		    plugins='',
>+		    pagesets='',

I think this parameter is called 'pageset' in the Factory.
As bug#483902 is approved+landed, we'll also need to approval+land this other part of the tp4 setup on 1.9.1.x.
Flags: wanted1.9.1.x?
Bug 475383 is the one we need landed on the 1.9.1 branch.
Comment on attachment 385465 [details] [diff] [review]
[Checked in]turn on tp4 on moz-central exclusively (take 2)

Can you make a corresponding patch for talos-staging-pool?
Bug 475383 landed on mozilla-1.9.1 yesterday.  Time to turn this on in the talos pool?
Tp4 for all of talos-staging-pool, along with Tracemonkey and Firefox3.5
Attachment #390329 - Flags: review?(catlee)
Comment on attachment 390329 [details] [diff] [review]
[Checked in]tp4 everywhere but 1.9.0

TP4 FTW!
Attachment #390329 - Flags: review?(catlee) → review+
Comment on attachment 390329 [details] [diff] [review]
[Checked in]tp4 everywhere but 1.9.0

changeset:   1398:5c47b32a4364
Attachment #390329 - Attachment description: tp4 everywhere but 1.9.0 → [Checked in]tp4 everywhere but 1.9.0
Attachment #390329 - Flags: checked-in+
status1.9.1: --- → ?
Flags: wanted1.9.1.x?
Assignee

Updated

10 years ago
Depends on: 509879
Assignee

Updated

10 years ago
Depends on: 509282
All that is left here is rolling out to mobile devices, but tp4 is considered final and is in production so this bug is considered complete.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.