be able to test a single build with talos multiple times

RESOLVED WONTFIX

Status

Release Engineering
General
P3
normal
RESOLVED WONTFIX
10 years ago
5 years ago

People

(Reporter: rhelmer, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

10 years ago
Currently, we cannot run more than one perf test per build because we don't have a way to correlate checkin time (revision) to specific test runs except for the start time.

So, all of the perf test machines lie about the start time, so that they line up on the build and graph server with the start time of the build (which is as close to the latest checkin as possible). This isn't the right thing to do; we should:

1) run one perf test per revision, for accurate regression testing
2) be able to run as many test runs against that revision as we like

For point #1 we need to store more than one depend build at a time

I think that Tinderbox server and the graph servers would need to have explicit support for this.
(Reporter)

Updated

10 years ago
Depends on: 291167
(Reporter)

Updated

10 years ago
Blocks: 401936
(Reporter)

Comment 1

10 years ago
(In reply to comment #0)
> 1) run one perf test per revision, for accurate regression testing

Slight ammendment .. run *at least one* perf test per revision. That is, do queuing more like the tryserver (one patch, one build) and not like the way buildbot does checkins by default (4 checkins since last build = one build).

I agree. I've been hankering for some additional data on the graphserver for awhile. I'd like to see a proper buildid stored and a column for run id so we can store multiple runs for the same build on the same machine. Some additional graphing options would be nice, for example, "Show me all the runs for this build from this machine".

I think ben's work on massaging the buildid got us part of the way there. We're generating the buildtime from the changetime on tinderbox for the talos run.
(Reporter)

Updated

10 years ago
Blocks: 411629
No longer blocks: 401936
Assignee: nobody → anodelman
Any update on this? It came up in triage today.
Component: Testing → Release Engineering
Priority: -- → P3
Product: Core → mozilla.org
Version: unspecified → other
QA Contact: testing → release

Comment 4

10 years ago
Reliably associating test results with builds is going to come up for l10n testing, too.

Comment 5

10 years ago
How does this bug block Bug 411629?
Can I remove the dependency and be able to change the status of the blocked bug fixed?
This will require a few things:

- graph server work to be able to display information concerning multiple test runs on a single build
- buildbot work to be able to trigger a secondary test run for a single build
Assignee: anodelman → nobody
Summary: separate revision from test start time → be able to test a single build with talos multiple times
Component: Release Engineering: Talos → Release Engineering: Future
(In reply to comment #6)
> This will require a few things:
> 
> - graph server work to be able to display information concerning multiple test
> runs on a single build

I'm not sure if this part is done. Alice or catlee, either of you know?

> - buildbot work to be able to trigger a secondary test run for a single build

This part is done.
(In reply to comment #7)
> (In reply to comment #6)
> > This will require a few things:
> > 
> > - graph server work to be able to display information concerning multiple test
> > runs on a single build
> 
> I'm not sure if this part is done. Alice or catlee, either of you know?

AFAIK, no.  I think it will record and display multiple runs of the same build on different machines, but if you run the same build on the same machine then the additional test runs are not displayed.
Created bug 519836 against the graph server which covers all the remaining work for this bug.
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → WONTFIX
Moving closed Future bugs into Release Engineering in preparation for removing the Future component.
Component: Release Engineering: Future → Release Engineering
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.