1355939 - Adding new jobs for Linux64 talos jobs does not currently work

Status: RESOLVED INVALID

(Taskcluster :: General, enhancement)

Description

[Armen [:armenzg]]

I believe the way this works is that TH fetches the latest task-graph.json from the index for that repo and it displays the jobs in the UI.
From looking at a task-graph.json from m-c [1] I can see talos tasks (e.g.  "label": "test-linux64/opt-talos-chrome”), however, the TH UI does not show them as jobs that can be selected. Well, it shows BB versions of them, however, they don't work because they require their associated Builbot Linux 64 builder which does not exist. mozci/pulse_actions barfs everytime we try to request this (bug 1355812).

I believe we need to determine why the talos BBB jobs don't show up on the runnable jobs API even though other TaskCluster tasks do show up. 

[1] https://public-artifacts.taskcluster.net/K35sIfN9RfSO_bnDNvP5QA/0/public/task-graph.json

Comment 1

[Brian Stack [:bstack]]

The reason the jobs don't show up in runnable_jobs is https://github.com/mozilla/treeherder/blob/master/treeherder/etl/runnable_jobs.py#L145

The test-linux64/opt-talos-chrome does not have any "extra" in the task definition. Why that is is a question as well I suppose. Would you like me to try to track that down or is that helpful as it stands?

Comment 2

[Greg Arndt [:garndt]]

If I'm understanding the problem correctly, the talos BBB tasks do not have task.extra, and thus do not show up on TH intentionally.  Otherwise we will have both the scheduled buildbot job for that BBB task as well as the BBB task showing up on TH.  we want the buildbot job to show up because it is the one that actually did the work and has the results attached to it.  The BBB tasks is merely for scheduling purposes.

If we didn't filter the BBB tasks out, then it would show up something like "Talos tc(Talos)" on treeherder where Talos might be the symbol for the buildbot talos job, and tc(talos) is the BBB task.

Comment 3

[Brian Stack [:bstack]]

I think this has left my pay grade and moved into releng world at this point. It seems like everything is working as expected from the TC+TH side. The question now is how can we make that job show up from an in-tree standpoint, right?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

yeah, I hit this adding new jobs in linux64-stylo talos :(

:bhearsum, I believe you are familiar with BBB, can you help out here?

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Joel Maher ( :jmaher) from comment #4)
> yeah, I hit this adding new jobs in linux64-stylo talos :(
> 
> :bhearsum, I believe you are familiar with BBB, can you help out here?

I'll have to defer to Rail at this point.

Comment 6

[Armen [:armenzg]]

We should call this the deferance bug! :P (Adding TH devs to be in the loop)

I see and I understand the decisions taken. Thanks for explaining it!

A quick summary, the issue happens because there are talos jobs on BB that do not have a BB builder to trigger them. Instead we have TC schedule those talos jobs via BBB. Since both the BB talos job and the TC talos job could show up on TH "Add new jobs" UI we decided to not add the "extra" attribute in the TC talos definitions to avoid repeated talos jobs (one representing a BB job and one a TC job).

Some ideas that could work:
* We can modify TH to blacklist the BB talos jobs and add extra to the TC talos jobs (hardcoding)
** We would update this blacklist every time we disable a build job on the BB side
* Instead of blacklisting I can add some code on TH to determine if a talos job does not have an associated build job.
* We could also add a parameter on allthethings.json (a change inside of buildbot-configs) that would allow TH to determine that the BB builder should not be shown.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

as a note, all of osx debug tests are run via BBB; soon to be osx opt as well.

Comment 8

[Ed Morley [:emorley]]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #6)
> Some ideas that could work:
> * We can modify TH to blacklist the BB talos jobs and add extra to the TC
> talos jobs (hardcoding)
> ** We would update this blacklist every time we disable a build job on the
> BB side

This seems prone to getting out of date.

> * Instead of blacklisting I can add some code on TH to determine if a talos
> job does not have an associated build job.

I'd prefer to reduce complexity on the Treeherder side.

> * We could also add a parameter on allthethings.json (a change inside of
> buildbot-configs) that would allow TH to determine that the BB builder
> should not be shown.

This seems preferable.

Comment 9

[Rail Aliiev [:rail]]

I don't think that Buildbot Bridge plays any special role here. It's pretty dumb, takes from TC and schedules something *already existing* in Buildbot.

Comment 10

[Armen [:armenzg]]

I will be working on this next week. I will be grabbing three reviewers:
* buildbot-configs (maybe tools)
* treeherder
* TC in-tree configs

I will be filing bugs for each component.

Comment 11

[Armen [:armenzg]]

Filed all three bugs. More next week.

Comment 12

[Armen [:armenzg]]

I'm going to refocus on some Quantum related work. I won't be able to look at this for a while.

The BB bug has patches that are going to be reviewed.
I will ask for help on the TH and TC bugs.

Comment 13

[Armen [:armenzg]]

Changing plans after speaking with the TC team.
We need to fix this in mozci/pulse_actions.
I can't tackle this at the moment.

Comment 14

[Armen [:armenzg]]

I will be looking to determine the way forward.

Comment 15

[Armen [:armenzg]]

This is a TaskCluster bug.
I will use bug 1355812 instead