1284236 - TC Linux64 opt (tier-1) builds are not indexed

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 02/17 - 02/21)]

Just figured out that we do not index the builds since we promoted the Linux64 opt builds on TC to tier-1:

https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.revision.39dffbba764210b25bfc1e749b4f16db77fa0d46.firefox/gecko.v2.mozilla-central.revision.39dffbba764210b25bfc1e749b4f16db77fa0d46.firefox.linux64-opt

All latest entries only contain buildbot builds.

Comment 1

[Michael Shal [:mshal]]

So it seems that if both buildbot and Taskcluster are tier-1, then buildbot wins. They both have the same rank in this case, so I believe it just depends on which job gets to the index last. We can probably work around that by making buildbot use "pushdate epoch minus one" as its rank so that Taskcluster would win in this situation.

However, I would normally expect that when a Taskcluster job is deemed worthy of promoting from tier-2 to tier-1, that we would subsequently turn off the equivalent buildbot job (within a week or so). :sdeckelmann, is there a reason the buildbot Linux64 opt job hasn't been disabled yet? I tried looking for a bug, but I only found information about Linux64 debug.

Comment 2

[Selena Deckelmann :selenamarie :selena]

(In reply to Michael Shal [:mshal] from comment #1)
> So it seems that if both buildbot and Taskcluster are tier-1, then buildbot
> wins. They both have the same rank in this case, so I believe it just
> depends on which job gets to the index last. We can probably work around
> that by making buildbot use "pushdate epoch minus one" as its rank so that
> Taskcluster would win in this situation.
> 
> However, I would normally expect that when a Taskcluster job is deemed
> worthy of promoting from tier-2 to tier-1, that we would subsequently turn
> off the equivalent buildbot job (within a week or so). :sdeckelmann, is
> there a reason the buildbot Linux64 opt job hasn't been disabled yet? I
> tried looking for a bug, but I only found information about Linux64 debug.

This is more of a question for :coop. Redirecting...

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Selena Deckelmann :selenamarie :selena from comment #2)
> (In reply to Michael Shal [:mshal] from comment #1)
> > So it seems that if both buildbot and Taskcluster are tier-1, then buildbot
> > wins. They both have the same rank in this case, so I believe it just
> > depends on which job gets to the index last. We can probably work around
> > that by making buildbot use "pushdate epoch minus one" as its rank so that
> > Taskcluster would win in this situation.
> > 
> > However, I would normally expect that when a Taskcluster job is deemed
> > worthy of promoting from tier-2 to tier-1, that we would subsequently turn
> > off the equivalent buildbot job (within a week or so). :sdeckelmann, is
> > there a reason the buildbot Linux64 opt job hasn't been disabled yet? I
> > tried looking for a bug, but I only found information about Linux64 debug.
> 
> This is more of a question for :coop. Redirecting...

The key sticking point here is whether we trust either system to produce valid nightlies without corresponding CI builds. 

We run Linux64 opt jobs still in buildbot (and more specifically, opt PGO) because they are used to select the known-good changeset used for buildbot nightlies. If we strive for absolute parity here before switching from buildbot->TC, we will need to stand up *everything* in TC before we switch over.

Part of standing up any build in TC is a QA step to ensure parity with buildbot. If we trust that parity step, I would like is to have all builds/tests running in TC as tier 1, and (until we're ready) have buildbot make its decision about nightlies based on those TC builds. That would allow us to turn off the corresponding buildbot CI builds.

Comment 4

[Michael Shal [:mshal]]

Do we need linux64-opt in buildbot if we have linux64-pgo there, or can we turn it off?

I'm also still a bit confused by the tier distinction. What should be available in the index if both the buildbot job and the Taskcluster job are tier-1? Should the linux64-opt Taskcluster build be tier-2 while its tests are tier-2?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

The one shred of sanity I'll stand on here is that tier-2 jobs should have index.rank=0.  To that end, I'm marking things that are tier-2 in treeherder as tier-2 in the in-tree YAML (as part of bug 1286075).  For a few tier-2 builds, that will change the index.rank from pushdate to 0, but I think that's the right thing to do.  The question of what to do when we have TC and BB both at tier-1 remains open (and I'll keep my opinions on it to myself).