438752 - moz2 talos builds sometimes line up with a build other than the one they tested

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

Talos builds on http://tinderbox.mozilla.org/showbuilds.cgi?tree=Mozilla2 have start times that line up with the build that they allegedly tested, which allows figuring out what code they tested.

However, I think bug 437391 comment 8 through bug 437391 comment 12 provide evidence that they aren't actually lining up the way they claim to be.  That should be fixed, since when we diagnose performance regressions we should be able to determine regression windows accurately.

Comment 1

[alice nodelman [:alice] [:anode]]

Based upon https://bugzilla.mozilla.org/show_bug.cgi?id=437391#c14 this is invalid.  I see no evidence that talos is incorrectly reporting to the waterfall.

Comment 2

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

Reopened based on bug 437391 comment 15.

Comment 3

[alice nodelman [:alice] [:anode]]

My going theory now is that we are getting some fallout from merging test requests (change sets in buildbot) when machines are busy.  Looks like when we merge we end up using the new changeset, but timestamp with some changeset from earlier in the queue.

Ccing bhearsum to see if my theory has any validity. 

Comment 4

[alice nodelman [:alice] [:anode]]

Here's the discrepancy:

- talos pulls a build to test based upon the date of the _first change in the list of merged changes
- buildbot dates builds sent to the waterfall with the _last change in the list of merged changes

We just need to decide what behavior we consider to be correct and go with it.

Comment 5

[alice nodelman [:alice] [:anode]]

Attachment: always test the last build in a list of merged changes

To go with the same logic that we use for time stamping the results on the waterfall we should always test the last build in a given list of buildbot changes.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 324740 [details] [diff] [review]
always test the last build in a list of merged changes

Makes sense to me. I'm a little surprised we didn't catch this bug sooner, though!

Comment 7

[alice nodelman [:alice] [:anode]]

Checking in perfrunner.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/talos/perfmaster/perfrunner.py,v  <--  perfrunner.py
new revision: 1.17; previous revision: 1.16
done

Comment 8

[alice nodelman [:alice] [:anode]]

I've pushed this to production.  I want to watch if for a couple of days to make sure that everything is working correctly.

When I had a closer look at the logs this misalignment was occurring all the time on the 1.8 waterfall - as we only have a single talos machine to test builds per platform resulting in a lot of merging.  Should be able to see if everything is okay by careful observation once the 1.8 builds are back up after spinning 2.0.0.15.

Comment 9

[alice nodelman [:alice] [:anode]]

I've looked over talos machines on the 1.8 waterfall and everything is now lining up correctly, so if talos is testing a given brower its result lines up on the waterfall with the build start time that produced that browser.