1389524 - Not all stylo jobs are showing up in the JobPriority table

Status: RESOLVED FIXED

(Tree Management Graveyard :: Treeherder: SETA, defect)

Description

[Armen [:armenzg]]

I'm not going to fix this before EOD. I will deal with it on Monday.

For some reason there are some stylo jobs containing the word 'style' and SETA is ignoring them.

I can't find style jobs in here:
https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=style&group_state=expanded&exclusion_profile=false

I don't find 'style' used under taskcluster/:
https://dxr.mozilla.org/mozilla-central/search?q=style+path%3Ataskcluster&redirect=true
or in Treeherder (Python files):
https://github.com/mozilla/treeherder/search?l=Python&q=style&type=&utf8=%E2%9C%93

Neither on a full-task-graph.json:
https://public-artifacts.taskcluster.net/aKQW4gPGSqC3W9hFSaXs1w/0/public/full-task-graph.json
nor a label-to-taskid.json artifact:
https://public-artifacts.taskcluster.net/aKQW4gPGSqC3W9hFSaXs1w/0/public/label-to-taskid.json

---------------

If I take a step back, I cannot see any "mochitest stylo" jobs in here:
https://treeherder.mozilla.org/api/project/autoland/seta/job-priorities/?build_system_type=taskcluster

>>> JobPriority.objects.filter(testtype__contains="stylo").filter(testtype__contains="mochitest").count()
0

Let's count stylo jobs.

There are 112 Windows7 stylo task labels without counting pgo and linux
armenzg@armenzg-mbp ~$ cat full-task-graph.json | grep -E 'windows7.*stylo/.*{' | grep "{" | grep "windows7" | grep -E -v '(awsy|pgo|linux)' | sort | wc -l
     112

If I ignore mochitests from that set I get to 78 which matches this DB query:
https://sql.telemetry.mozilla.org/queries/15985/source
armenzg@armenzg-mbp ~$ cat full-task-graph.json | grep -E 'windows7.*stylo/.*{' | grep "{" | grep "windows7" | grep -E -v '(awsy|pgo|linux|mochitest)' | sort | wc -l
      78

I think all the mochitest jobs have been conflated into 1 mochitest *style* job priority per category.

You can see all 18 of them in here:
https://sql.telemetry.mozilla.org/queries/15987/source

------------------------

The code that excludes this "unnacceptable" jobs is in here:
https://github.com/mozilla/treeherder/blob/fd8c377eb2af2ddc23c0a0b26f55e43cb293db90/treeherder/seta/job_priorities.py#L49-L54

We build the testtype in here:
https://github.com/mozilla/treeherder/blob/fd8c377eb2af2ddc23c0a0b26f55e43cb293db90/treeherder/seta/update_job_priority.py#L70-L75
https://github.com/mozilla/treeherder/blob/fd8c377eb2af2ddc23c0a0b26f55e43cb293db90/treeherder/etl/seta.py#L63-L88



== Jobs matching on the production database ==
>>> JobPriority.objects.filter(testtype__contains="style")
<QuerySet [<JobPriority: taskcluster,mochitest-style,opt,linux64-stylo-sequential>, <JobPriority: taskcluster,mochitest-style-e10s,opt,linux64-stylo-sequential>, <JobPriority: taskcluster,mochitest-style,debug,linux64-stylo-sequential>, <JobPriority: taskcluster,mochitest-style-e10s,debug,linux64-stylo-sequential>, <JobPriority: taskcluster,mochitest-style-e10s,opt,macosx64-stylo>, <JobPriority: taskcluster,mochitest-style,opt,macosx64-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,debug,macosx64-stylo>, <JobPriority: taskcluster,mochitest-style,debug,macosx64-stylo>, <JobPriority: taskcluster,mochitest-style-e10s,debug,macosx64-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,opt,macosx64-stylo>, <JobPriority: taskcluster,mochitest-style-e10s,debug,windows10-64-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,debug,windows10-64-stylo>, <JobPriority: taskcluster,mochitest-style-e10s,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,opt,windows10-64-stylo>, <JobPriority: taskcluster,mochitest-style-e10s,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-style-e10s,opt,windows10-64-stylo>]>
>>> JobPriority.objects.filter(testtype__contains="style").count()
18

== *Localhost* output after loading the job-priorities API ==
[2017-08-11 13:35:23,656] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-style-e10s,debug,linux64-stylo-sequential) not found in accepted jobs list
[2017-08-11 13:35:23,663] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-style-e10s,debug,windows7-32-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,666] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-chrome-style,opt,windows10-64-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,667] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-style-e10s,opt,windows10-64-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,668] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-chrome-style,debug,macosx64-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,668] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-style-e10s,opt,linux64-stylo-sequential) not found in accepted jobs list
[2017-08-11 13:35:23,670] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-style-e10s,debug,windows10-64-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,670] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-style-e10s,opt,macosx64-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,672] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-chrome-style,debug,windows10-64-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,679] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-chrome-style,opt,windows7-32-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,686] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-chrome-style,opt,macosx64-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,687] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-style-e10s,opt,windows7-32-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,688] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-chrome-style,debug,windows7-32-stylo) not found in accepted jobs list
[2017-08-11 13:35:23,691] WARNING [treeherder.seta.job_priorities:54] Job priority (taskcluster,mochitest-style-e10s,debug,macosx64-stylo) not found in accepted jobs list

Comment 1

[Armen [:armenzg]]

I don't know what on this world introduced 'style' jobs but I decided to delete them and refresh the table.

Steps taken:
* delete style job priorities
* update job priority table (110 new records)
* set priority to 5 and removed expiration_date

I can see Windows mochitest stylo jobs with priority 5 in the API:
https://treeherder.mozilla.org/api/project/autoland/seta/job-priorities/?build_system_type=taskcluster&priority=5

>>> from treeherder.seta.models import JobPriority;
>>> JobPriority.objects.filter(testtype__contains="style")
<QuerySet [<JobPriority: taskcluster,mochitest-style,opt,linux64-stylo-sequential>, <JobPriority: taskcluster,mochitest-style-e10s,opt,linux64-stylo-sequential>, <JobPriority: taskcluster,mochitest-style,debug,linux64-stylo-sequential>, <JobPriority: taskcluster,mochitest-style-e10s,debug,linux64-stylo-sequential>, <JobPriority: taskcluster,mochitest-style-e10s,opt,macosx64-stylo>, <JobPriority: taskcluster,mochitest-style,opt,macosx64-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,debug,macosx64-stylo>, <JobPriority: taskcluster,mochitest-style,debug,macosx64-stylo>, <JobPriority: taskcluster,mochitest-style-e10s,debug,macosx64-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,opt,macosx64-stylo>, <JobPriority: taskcluster,mochitest-style-e10s,debug,windows10-64-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,debug,windows10-64-stylo>, <JobPriority: taskcluster,mochitest-style-e10s,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-chrome-style,opt,windows10-64-stylo>, <JobPriority: taskcluster,mochitest-style-e10s,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-style-e10s,opt,windows10-64-stylo>]>
>>> JobPriority.objects.filter(testtype__contains="style").delete()
(18L, {u'seta.JobPriority': 18L})
>>> JobPriority.objects.filter(testtype__contains="style")
<QuerySet []>
>>> from treeherder.seta.update_job_priority import update_job_priority_table
>>> update_job_priority_table()
(110, 0, 0)
>>> JobPriority.objects.filter(testtype__contains="style")
<QuerySet []>
>>> from treeherder.seta.preseed import load_preseed
>>> load_preseed()
>>> JobPriority.objects.filter(testtype__contains="style")
<QuerySet []>
>>> JobPriority.objects.filter(platform="windows7-32-stylo", testtype__contains='mochitest')
<QuerySet [<JobPriority: taskcluster,mochitest-media-e10s-1,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-media-e10s-2,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-e10s-2,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-e10s-4,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-gl-e10s-2,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-media-e10s-3,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-browser-screenshots-e10s,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-gpu-e10s,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-e10s-4,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-e10s-3,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-e10s-1,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-media-e10s-3,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-media-e10s-1,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-e10s-5,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-e10s-2,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-gl-e10s-3,opt,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-gl-e10s-1,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-gl-e10s-2,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-gl-e10s-3,debug,windows7-32-stylo>, <JobPriority: taskcluster,mochitest-gpu-e10s,debug,windows7-32-stylo>, '...(remaining elements truncated)...']>

>>> JobPriority.objects.filter(platform="windows7-32-stylo", testtype__contains='mochitest', expiration_date__isnull=False).update(expiration_date=None)
26L
>>> JobPriority.objects.filter(platform="windows10-64-stylo", testtype__contains='mochitest', expiration_date__isnull=False).update(expiration_date=None)
26L
>>> JobPriority.objects.filter(platform="windows10-64-stylo", priority=1)
<QuerySet []>
>>> JobPriority.objects.filter(platform="windows7-32-stylo", priority=1)
<QuerySet []>
>>> update_job_priority_table()
(0, 0, 0)