696864 - Double check that regressions of rev3 10.6 matches to rev4 10.6

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

The last rev4 email regression that I have seen if from the 14th. 
We should check that this is correct since 10.6.2 rev3 machines have sent emails for at least 15 times since that day.

It seems that both rev3 and rev4 are reporting to graphs:
http://graphs-new.mozilla.org/graph.html#tests=[[115,1,21],[115,1,17],[112,1,21],[112,1,17],[104,1,21],[104,1,17]]&sel=none&displayrange=7&datatype=running

NOTICE: Non-PGO branches are not considered for Mac.

Assigned to catlee as he is going to look into it.

Comment 1

[Chris AtLee [:catlee]]

I've checked a few of the recent regression mails from the rev3 machines vs the rev4 machines. If you look at the graphs for the two platforms, the two platforms usually follow the same rise and falls on the graph. e.g. 

http://graphs-new.mozilla.org/graph.html#tests=[[77,63,21],[77,63,17]]&sel=1319950573361,1320037991429&displayrange=7&datatype=running

The regression detection script isn't picking all of the same regressions on the rev4 systems as it is for the rev3 ones though.

Comment 2

[Armen [:armenzg]]

(In reply to Chris AtLee [:catlee] from comment #1)
> The regression detection script isn't picking all of the same regressions on
> the rev4 systems as it is for the rev3 ones though.

I can't think of anything else for this.

Comment 3

[Armen [:armenzg]]

For the same changeset (11-10-30 7:05 PM):
Talos Improvement! Ts Shutdown, MIN Dirty Profile decrease 10.5% on MacOSX 10.6 (rev4) Mozilla-Inbound
Talos Improvement! Ts Shutdown, MED Dirty Profile decrease 11.9% on MacOSX 10.6 (rev4) Mozilla-Inbound
Talos Improvement! Ts Shutdown decrease 9.74% on MacOSX 10.6 (rev4) Mozilla-Inbound
Talos Improvement! Codesighs decrease 4.77% on MacOSX 10.6.2 (rev3) Mozilla-Inbound
Talos Regression :( Dromaeo (String/Array/Eval/Regex) decrease 2.19% on MacOSX 10.6.2 (rev3) Mozilla-Inbound

Rev3 noticed Codesighs and Dromaeo changes while rev4 detected TS shutdown variances.

For another changeset (11-10-27 7:25 AM):
Talos Regression :( V8 increase 2.82% on MacOSX 10.6 (rev4) Mozilla-Inbound
Talos Regression :( Dromaeo (String/Array/Eval/Regex) decrease 4.63% on MacOSX 10.6 (rev4) Mozilla-Inbound
Talos Regression :( V8 increase 3.39% on MacOSX 10.6.2 (rev3) Mozilla-Inbound
Talos Regression :( Dromaeo (String/Array/Eval/Regex) decrease 4.94% on MacOSX 10.6.2 (rev3) Mozilla-Inbound

Both detect the same variances. Slightly different percentages.

I believe we can conclude that different HW specs and slightly different OS (10.6.8 vs 10.6.6) can make some suites vary and/or different percentages.

RESOLVED VERIFIED?

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

Yes!  Thanks for looking into this Armen.