410869 - be able to test a single build with talos multiple times

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Robert Helmer [:rhelmer]]

Currently, we cannot run more than one perf test per build because we don't have a way to correlate checkin time (revision) to specific test runs except for the start time.

So, all of the perf test machines lie about the start time, so that they line up on the build and graph server with the start time of the build (which is as close to the latest checkin as possible). This isn't the right thing to do; we should:

1) run one perf test per revision, for accurate regression testing
2) be able to run as many test runs against that revision as we like

For point #1 we need to store more than one depend build at a time

I think that Tinderbox server and the graph servers would need to have explicit support for this.

Comment 1

[Robert Helmer [:rhelmer]]

(In reply to comment #0)
> 1) run one perf test per revision, for accurate regression testing

Slight ammendment .. run *at least one* perf test per revision. That is, do queuing more like the tryserver (one patch, one build) and not like the way buildbot does checkins by default (4 checkins since last build = one build).

Comment 2

[Rob Campbell [:rc] (:robcee)]

I agree. I've been hankering for some additional data on the graphserver for awhile. I'd like to see a proper buildid stored and a column for run id so we can store multiple runs for the same build on the same machine. Some additional graphing options would be nice, for example, "Show me all the runs for this build from this machine".

I think ben's work on massaging the buildid got us part of the way there. We're generating the buildtime from the changetime on tinderbox for the talos run.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Any update on this? It came up in triage today.

Comment 4

[Axel Hecht [:Pike]]

Reliably associating test results with builds is going to come up for l10n testing, too.

Comment 5

[Armen [:armenzg]]

How does this bug block Bug 411629?
Can I remove the dependency and be able to change the status of the blocked bug fixed?

Comment 6

[alice nodelman [:alice] [:anode]]

This will require a few things:

- graph server work to be able to display information concerning multiple test runs on a single build
- buildbot work to be able to trigger a secondary test run for a single build

Comment 7

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> This will require a few things:
> 
> - graph server work to be able to display information concerning multiple test
> runs on a single build

I'm not sure if this part is done. Alice or catlee, either of you know?

> - buildbot work to be able to trigger a secondary test run for a single build

This part is done.

Comment 8

[Chris AtLee [:catlee]]

(In reply to comment #7)
> (In reply to comment #6)
> > This will require a few things:
> > 
> > - graph server work to be able to display information concerning multiple test
> > runs on a single build
> 
> I'm not sure if this part is done. Alice or catlee, either of you know?

AFAIK, no.  I think it will record and display multiple runs of the same build on different machines, but if you run the same build on the same machine then the additional test runs are not displayed.

Comment 9

[alice nodelman [:alice] [:anode]]

Created bug 519836 against the graph server which covers all the remaining work for this bug.

Comment 10

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.