413713 - Need rapidly cycling Talos boxes to replace old perf test machines

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rob Campbell [:rc] (:robcee)]

Before we can mothball the old perf test machines, we need to provide a rapidly-cycling talos box that can reproduce as closely as possible what the old machines provide.

The requirements are: 10-30m cycle times.

Pared down pageset for running Tp.

Other tests include: Ts, Tdhtml, and Txul.

Comment 1

[Johnathan Nightingale [:johnath]]

I've been spouting for a little while to whomever would listen that a pared down talos pageset should be possible, even one that tracks the "real" talos numbers.  My thinking is that if we can get our hands on a stats software package, we oughta be able to run a regression analysis on each of the ~400 pages, and find the N pages that, taken together, correlate well with real Tp over the history we currently have.

N could be 50, or 30, or 10 -- whatever tradeoff we want to make between rapid sniff tests and probability of catching regressions.  I'm thinking that we name a cycle-time, 10 minutes say, and keep running down the list of "next most useful page" until we start hitting that.

We'll want to be clear that this isn't the real Tp, that full Talos Tp results win all arguments.  Tp_snifftest?  Tquickcheck?  Something.

What are the odds that we could get some kind of straight DB dump that has historical scores for every page?  Maybe the CSV dumps from graph server are enough...

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

The CSV dumps would be enough, but you'd have a hard time going through enough historical data to get them.  To do this kind of analysis you really just want a copy of the database.

That said, I bet someone could spend an afternoon and manually identify pages that would be useful to have in the set.

Comment 3

[Mike Schroepfer]

Can we just get a mysql dump of the db Aravind?

Comment 4

[Aravind Gottipati [:aravind]]

(In reply to comment #3)
> Can we just get a mysql dump of the db Aravind?
> 

Yup, people.mozilla.com:/tmp/graphs_mozilla_org.2008.01.24.sql.gz

Comment 5

[alice nodelman [:alice] [:anode]]

From a meeting today with me, schrep, robcee & joduinn our current thinking is as follows:

- we want rapidly cycling talos boxes so that we can retire tinderbox perf test boxes
- if terms of the fastest route to get us to that point we need to have tests that run as quickly as the tinderbox perf tests do
- thus, to start, we will just set up a set of talos boxes (win/mac/linux) to run the same tests that the current tinderbox perf test boxes are running (tp with the historic pageset, ts, tdhtml, txul, ts)

This has the advantage of not having to do any statistical analysis of the new page set to determine an interesting sub set of pages, along with pretty much exactly replacing the machines that we want to retire.

We can still pursue the concept of a small scale talos machine that runs pared down versions of all the tests we have (including tgfx, tsvg, etc) - but that would considered a separate project.

Comment 6

[Johnathan Nightingale [:johnath]]

That's much easier, I agree.  :)

Comment 7

[Mike Schroepfer]

(In reply to comment #5)
> From a meeting today with me, schrep, robcee & joduinn our current thinking is
> as follows:

++ to this plan.   

Comment 8

[Mike Schroepfer]

If it is *super* easy it would be great to get a set of 1.8 boxes running the same tests - I'm curious on how we are doing on the old Tp pageset on branch vs. trunk.   

Comment 9

[Mikeal Rogers [:mikeal] (mrogers@)]

Created new task for mini talos run;

https://bugzilla.mozilla.org/show_bug.cgi?id=419105

Comment 10

[alice nodelman [:alice] [:anode]]

We now have both a set of trunk fast and branch fast machines reporting.  See Firefox & Mozilla1.8 waterfalls.

Comment 11

[(not currently active) Ted Mielczarek]

Mass move of Core:Testing bugs to mozilla.org:ReleaseEngineering. Filter on RelEngMassMove to ignore.