419105 - Create mini talos run

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Mikeal Rogers [:mikeal] (mrogers@)]

Task to create a shorter talos run that can still find the max amount of regressions in the shortest total run time.

This task requires;

-Find the "most interesting" pages from the existing talos runs
-Determine the optimum amount of pages to satisfy usefullness and speed requirements
-Run tests in isolation for a time to determine whether they are in fact effective.

Comment 1

[Mikeal Rogers [:mikeal] (mrogers@)]

This looks like the best 10 to start with;

www.washingtonpost.com
www.pconline.com.cn
www.mobile.de
www.wretch.cc
cn.msn.com
www.wangyou.com
www.wikipedia.org
www.invisionfree.com
www.tianya.cn
www.qihoo.com

Comment 2

[Johnathan Nightingale [:johnath]]

(In reply to comment #1)
> This looks like the best 10 to start with;

Looks like a nice list, multiple languages, multiple layout styles, but I wonder if you could give any insight into what made these the 10 best?  Was it purely statistical, were there explicit criteria?

Comment 3

[Rob Campbell [:rc] (:robcee)]

these were based on the largest variances across all platforms for data in graphs.mozilla.org.

Comment 4

[Mikeal Rogers [:mikeal] (mrogers@)]

To be more specific, I pulled info out of the graph server in to a local db scheme that was significantly richer. First I took just the mini- hardware data did the following.

1. Calculated the average time to fall runs per page. 
2. Took the highest 100 and lowest 100 results of each page, averaged it, and stored the differential between that number and the average.
3. Then I look through the biggest differentials on the high and low end, and took 7 of the 10 sites in that list.

Then I took the rest of the tp_loadtime data out of the graph server, no just the mini- hardware, and computed the same as before but did it per hardware.

I sorted through all that data and picked out the 3 that were showing up the most and weren't showing up in the mini- only hardware.

We may increase the amount of sites in this list, there is plenty of data if we ever want to pull more. I just want to see how long these 10 take to run and see what kind of stuff they're catching first.

Comment 5

[Mikeal Rogers [:mikeal] (mrogers@)]

typo:

1. Calculated the average time to for all runs per page.

Comment 6

[Mikeal Rogers [:mikeal] (mrogers@)]

For refernces, the _next_ 20 "most interesting" sites would be:

www.asahi.com
www.nikkansports.com
www.verycd.com
www19.dantri.com.vn
www.nikkei.co.jp
www.pcpop.com
www.jrj.com.cn
www.terra.com.br
www.expedia.com
www.yandex.ru
www.typepad.com
www.webs-tv.net
www.cj.com
www.webs-tv.net
www.kooora.com
www.weather.com
tag.seesaa.jp
www.it168.com
www.265.com
www.clarin.com
www.qihoo.com

Comment 7

[Mikeal Rogers [:mikeal] (mrogers@)]

I've uploaded a tarball of mess of code I wrote to get these sites in case we ever need to come back to it.

It's in my user directory on the office fileserver.

Comment 8

[alice nodelman [:alice] [:anode]]

Instead of working on mini talos I've constructed a new 100 page set (bug 473821) - that will give us good perf data and run in far less time than the 400 page set currently in use.

Comment 9

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.