698513 - talos-r4-snow is one hour off from talos-r3-snow in end-to-end per OS

Status: RESOLVED INCOMPLETE

(Testing Graveyard :: GoFaster, defect)

Description

[Armen [:armenzg]]

If you load up http://brasstacks.mozilla.com/gofaster/#/endtoend/per_os
and only select the osx10.6 and osx10.6-r4 you will see that r3-snow is approximately one hour up.

If I look at [1] I can see that "Rev3" and "rev4" test jobs start at the same time (by clicking on each pair of jobs).

If you look at [2] you will see that "10.5", "10.6.2" (r3-snow) and "10.6" (r4-snow) jobs are triggered at the same time (a.k.a. both jobs are triggered by the same builds).

[1] https://tbpl.mozilla.org/?noignore=1&jobname=10.6&rev=271b14e0c0b
[2] http://brasstacks.mozilla.com/gofaster/buildchart.html#/builds/6068559dc65c4ac69daacc1151066f14

Comment 1

[Jonathan Griffin (:jgriffin)]

Basically, what's happening is that we have a group of tests on osx10.6-r4 which don't have a matching build; that is, both the 10.6 and 10.6-r4 tests are kicked off with the same build - the 10.6 build.

The algorithm that aggregates the data expects that tests and build will be performed on the same os; it doesn't handle the case where tests occur with a different os string than the build.

What this means is that currently, the 10.6-r4 line on the E2E graphs are showing the total E2E time for tests only.  We'll need to add a mapping to correlate the two, so we can get a true E2E number.  Just eyeballing the individual buildcharts, it looks like the E2E times for 10.6 and 10.6-r4 are nearly identical.