419487 - change buildbot & talos to use buildtime, not test-run time

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

There is no change needed to graph server. We just need to change buildbot & talos to pass buildtime instead of test-runtime.

Comment 2

[alice nodelman [:alice] [:anode]]

Attachment: patches perfrunner to send build timestamp to talos machines

Comment 3

[alice nodelman [:alice] [:anode]]

Attachment: patches PerfConfigurator to handle new timestamp format

Add support to PerfConfigurator:

- pull buildid from application.ini of talos
- use a testdate as an argument for timestamping the results
- have the option of using the buildid as a timestamp (might still be useful for baseline work)
- handle dates that are either 10 digits or 12 digits long (YmdH or YmdHM)

Comment 4

[alice nodelman [:alice] [:anode]]

Currently talos reports results to the graph server timestamped with test start time.  This makes it hard to determine regression ranges as the test time has no connection to when the build being tested was created.

If we go with this solution the talos machines will start to use timestamps that reflect the build start time as used on the waterfall - so talos results would line up with waterfall results.

This should make it easier to track regressions and determine bonsai ranges for a given test.

We would _not_ be using the BuildID as stored in the application.ini for timestamping, it would just be stored in the graph server database as an extra piece of information.

Comment 5

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 318699 [details] [diff] [review]
patches PerfConfigurator to handle new timestamp format

this seems to work ok on mac. I wish the error-handling were a bit better and more informative, but that's probably worth filing another bug to fix it.

Comment 6

[alice nodelman [:alice] [:anode]]

Adding a dependency to bug 291167 - there's no point in changing the time stamp to buildtime until we can guarantee that we are pulling the correct build.

Comment 7

[Johnathan Nightingale [:johnath]]

+1 to this change if it happens after FF3 release, with a little bit of advance notice in the newsgroups as to when the actual "flag day" will be.

It will absolutely make it easier to track down regressions in the future, especially now that we have multiple boxes reporting, so that we can play off near-adjacent machines against each other to build a small check-in window.

Probably want to do this on a day that the tree's been green (well, perf-flat, I guess I mean) for a solid 24 hours or so, so that we're unlikely to be looking for regressions across the jump.

Comment 8

[alice nodelman [:alice] [:anode]]

Adding blocking on bug 388685 (graph server should handle duplicate data better).  If we switch to using build time there is a not entirely unlikely situation where we would end up serving the same build to a single machine to test twice - resulting in an attempt to send data with the same time stamp to the graph server and thus corrupting the graph server database.

The simplest solution as I see it would be to have the graph server simply delete any previous data collected for a machine/timestamp key and replace it with the new data.  In future we'd like to collect the information in a more sensible manner, but this would get us where we want to be for now.

Comment 9

[alice nodelman [:alice] [:anode]]

We now have a system in place for dealing with duplicate data in the graph server (the bug remains open waiting for a check in to hg).  We are also in a slow(ish) state after ff3 - this is probably the best time that we are going to get to land this and see what sort of fall out there is.

Next scheduled outage we should include this patch.  It should only be pushed when all trees are closed.

Comment 10

[alice nodelman [:alice] [:anode]]

Attachment: patch perfrunner for both talos and talos try

Missed out on the change to talos try.  Should push all of this at the same time for things to not go red.

Comment 11

[alice nodelman [:alice] [:anode]]

talos change:
Checking in PerfConfigurator.py;
/cvsroot/mozilla/testing/performance/talos/PerfConfigurator.py,v  <--  PerfConfigurator.py
new revision: 1.4; previous revision: 1.3
done

buildbot changes:
Checking in perfmaster/perfrunner.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/perfrunner.py,v  <--  perfrunner.py
new revision: 1.18; previous revision: 1.17
done
Checking in tryperfmaster/perfrunner.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/tryperfmaster/perfrunner.py,v  <--  perfrunner.py
new revision: 1.4; previous revision: 1.3
done

Comment 12

[alice nodelman [:alice] [:anode]]

Filed bug 441837 for bustage after pushing patches.  

Comment 13

[alice nodelman [:alice] [:anode]]

This is up and working correctly on production.