1395724 - When adding new jobs, talos job symbols don't appear in their treeherder groups

Status: RESOLVED INVALID

(Tree Management :: Treeherder, defect)

Description

[Robert Wood [:rwood]]

Unsure if this is a treeherder or taskcluster issue. Adding new talos jobs on try is failing. The gecko-1-decision opt 'A' job goes red with:

[taskcluster:error] (HTTP code 404) no such container - invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"/builds/worker/bin/run-task\\\": stat /builds/worker/bin/run-task: no such file or directory\"\n" 

See log [1].

[1] https://public-artifacts.taskcluster.net/CwDv6igUSbaeuHfPGaD80A/0/public/logs/live_backing.log

One strange thing I notice, when I choose 'Add new jobs', the talos test jobs don't show up in their treeherder groups i.e. I'm trying to add 'g1' which should appear inside the "T-e10s" treeherder group, but when I do 'Add new jobs' that group only contains the job(s) originally added in the try syntax; and the other talos jobs are all shown across the top in no group at all.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

oh, this is an issue that should have been fixed via bug 1394779.  Possibly your base tree is out of date?

Comment 2

[Robert Wood [:rwood]]

Ahh, yes you are correct my local branch didn't have the latest. Will re-open if it still doesn't work on my next try push after I update, but probably will be fine, thanks!

Comment 3

[Robert Wood [:rwood]]

This still doesn't work for me, even with the latest repo pulled. For my latest try push [1] I want to add talos jobs for Linux x64 opt, but when I choose 'add new jobs', the 'T-e10s' group doesn't even show up. Also same thing if I try to add linux x64 opt talos jobs on one of :jmaher's latest try pushes [2].

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=910a531495792ec9cd53339335e43087a710be3b

[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b1339b6c7198361a8e059d2075d695a4ce29536

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

adding in the expert :bstack :)

do you see the job symbols in general?  I have seen odd stuff on tree related to stylo-disabled

Comment 5

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #4)
> adding in the expert :bstack :)
> 
> do you see the job symbols in general?  I have seen odd stuff on tree
> related to stylo-disabled

Ah yes so I do see the symbols along the top not inside any group and multiplied (i.e. regular job, stylo job, etc). I chose the one I need and added that and will see if that works... weird they're not in any group though.

Comment 6

[Robert Wood [:rwood]]

It looks like that worked, I was able to add the new job on Linux x64 opt and it is now pending. So I'm not sure if there's a bug here, but IMO when adding new jobs, all the symbols should be showing up in their groups and not just all listed across the top... but perhaps that is expected behaviour now for some reason (:bstack)?

Comment 7

[Brian Stack [:bstack]]

The borders of the kingdoms involved with the runnable jobs stuff are a bit confusing and this is actually a bit outside my lands. However, I'm not sure who owns the land over there, so I've poked around a bit and here's what I've found:

All of that logic with putting things into job groups happens in [0]. The results return from the treeherder api indeed only have a job group for some platforms [1]. Jobs are loaded from the th api via [2] and [3]. All of the taskcluster jobs appear to have their build_group info set correctly, but none of the ones from the buildbot list appear to have job_group info set correctly. I think with that stuff the best bet is to find somebody who knows more about buildbot+treeherder than I do :p


[0] https://github.com/mozilla/treeherder/blob/master/ui/js/models/resultsets_store.js#L1174

[1] https://pastebin.mozilla.org/9031949

[2] https://github.com/mozilla/treeherder/blob/master/treeherder/etl/runnable_jobs.py#L121

[3] https://github.com/mozilla/treeherder/blob/master/treeherder/etl/runnable_jobs.py#L173

Comment 8

[Robert Wood [:rwood]]

(In reply to Brian Stack [:bstack] from comment #7)
> The borders of the kingdoms involved with the runnable jobs stuff are a bit
> confusing and this is actually a bit outside my lands. However, I'm not sure
> who owns the land over there, so I've poked around a bit and here's what
> I've found:
> 
> All of that logic with putting things into job groups happens in [0]. The
> results return from the treeherder api indeed only have a job group for some
> platforms [1]. Jobs are loaded from the th api via [2] and [3]. All of the
> taskcluster jobs appear to have their build_group info set correctly, but
> none of the ones from the buildbot list appear to have job_group info set
> correctly. I think with that stuff the best bet is to find somebody who
> knows more about buildbot+treeherder than I do :p
> 
> 
> [0]
> https://github.com/mozilla/treeherder/blob/master/ui/js/models/
> resultsets_store.js#L1174
> 
> [1] https://pastebin.mozilla.org/9031949
> 
> [2]
> https://github.com/mozilla/treeherder/blob/master/treeherder/etl/
> runnable_jobs.py#L121
> 
> [3]
> https://github.com/mozilla/treeherder/blob/master/treeherder/etl/
> runnable_jobs.py#L173

Thank you Brian! Perhaps you have some insight here, Kim?

Comment 9

[Kim Moir [:kmoir] ET]

Are you adding new talos jobs that run on through buildbot bridge?  If this is the case, if the new job names are not specified, they'll won't run because they need to specified on buildbot side.

Comment 10

[Robert Wood [:rwood]]

(In reply to Kim Moir [:kmoir] ET from comment #9)
> Are you adding new talos jobs that run on through buildbot bridge?  If this
> is the case, if the new job names are not specified, they'll won't run
> because they need to specified on buildbot side.

Thanks Kim. The jobs did schedule and run fine afterall so I think maybe it's a treeherder issue. When choosing the 'add new jobs' menu on a try link, the talos jobs don't show up in their corresponding groups (i.e. T-e10s) and instead the individual job symbols show up along the top of the platform group on their own. I'll attach a screenshot example.

Comment 11

[Robert Wood [:rwood]]

Attachment: Screen Shot 2017-09-13 at 9.17.50 AM.png

Screenshot example

Comment 12

[Ed Morley [:emorley]]

Hi! Is this still happening? If so, do you have links to more recent examples? :-)

Comment 13

[Robert Wood [:rwood]]

(In reply to Ed Morley [:emorley] from comment #12)
> Hi! Is this still happening? If so, do you have links to more recent
> examples? :-)

No sir, just checked again it's not happening anymore. Thanks for bringing this to my attention :)

Comment 14

[Ed Morley [:emorley]]

Thank you for confirming - perhaps the move away from buildbot / buildbot bridge since this was filed is what fixed the issue.