Closed Bug 1338102 Opened 8 years ago Closed 8 years ago

SETA: many warnings in papertrail about not finding jobs from jobpriority table in runnable jobs

Categories

(Tree Management Graveyard :: Treeherder: SETA, defect)

Product:
Tree Management Graveyard
Component:
Treeherder: SETA
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jmaher, Assigned: rwood)

References

Details

Attachments

(1 file)

[treeherder] rwood-moz:bug1338102 > mozilla:master
47 bytes, text/x-github-pull-request
jmaher
: review+
Details | Review
Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,633] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-devtools-chrome-e10s-7) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,634] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-devtools-chrome-e10s-6) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,634] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-devtools-chrome-e10s-5) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,634] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-devtools-chrome-e10s-3) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,634] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-devtools-chrome-e10s-2) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,634] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-devtools-chrome-e10s-9) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,634] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-devtools-chrome-e10s-8) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,634] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-10) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,635] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-11) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,635] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-19) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,635] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-webgl-2) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,635] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-webgl-3) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,636] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-webgl-1) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,636] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-38) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,636] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,android-api-15-gradle,opt,mochitest-chrome) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,636] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-9) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,636] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-browser-chrome-e10s-3) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,636] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-32) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,637] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-33) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,637] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-40) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,637] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-browser-chrome-e10s-5) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,637] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-14) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,637] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-13) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,638] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-18) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,638] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,android-4-3-armv7-api15,debug,mochitest-webgl-10) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,638] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-6) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,638] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-5) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,639] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,osx-10-10,debug,jsreftest-e10s-2) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,639] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,asan,mochitest-devtools-chrome-e10s-10) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,639] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-devtools-chrome-e10s-1) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,639] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-39) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,640] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-devtools-chrome-e10s-3) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,640] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-webgl-3) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,640] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-webgl-2) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,640] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-webgl-1) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,640] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,windows10-64-vm,debug,external-media-tests) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,640] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-6) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,640] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-3) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,641] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-1) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,641] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,external-media-tests) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,641] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-devtools-chrome-e10s-10) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,641] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-7) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,641] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-35) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,641] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-7) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,641] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-4) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,642] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-5) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,642] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-2) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,642] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-3) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,642] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-1) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,642] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-8) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,642] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-8) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,642] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-9) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,643] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-10) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,643] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-11) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,643] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-12) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,643] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux32,opt,external-media-tests) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,644] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux32,opt,mochitest-devtools-chrome-e10s-10) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,644] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux32,debug,external-media-tests) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,644] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux32,opt,mochitest-devtools-chrome-e10s-9) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,644] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux32,debug,mochitest-browser-chrome-e10s-12) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,644] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux32,debug,mochitest-browser-chrome-e10s-11) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,644] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux32,debug,mochitest-browser-chrome-e10s-10) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,644] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux32,debug,mochitest-browser-chrome-e10s-9) in the list of accepted jobs Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,645] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux32,debug,mochitest-browser-chrome-e10s-8) in the list of accepted jobs there are many other messages like this- I suspect we are having a small issue either in conversion of data, or comparing fields. Looking at some examples above, we have data in the database for mochitest-valgrind-*, and it is not finding it in runnable_jobs. I downloaded runnable jobs data from here: https://treeherder.mozilla.org/api/project/autoland/runnable_jobs/?decision_task_id=a1kukcMeQUGcbNrRJ2JdWQ&format=json and parsed it for valgrind-x, and I see: test-linux64/opt-mochitest-valgrind-31 test-linux64/opt-mochitest-valgrind-35 test-linux64/opt-mochitest-valgrind-34 test-linux64/opt-mochitest-valgrind-37 test-linux64/opt-mochitest-valgrind-36 valgrind-linux64-valgrind/opt test-linux64/opt-mochitest-valgrind-28 test-linux64/opt-mochitest-valgrind-26 test-linux64/opt-mochitest-valgrind-27 test-linux64/opt-mochitest-valgrind-24 test-linux64/opt-mochitest-valgrind-25 test-linux64/opt-mochitest-valgrind-20 test-linux64/opt-mochitest-valgrind-21 test-linux64/opt-mochitest-valgrind-17 test-linux64/opt-mochitest-valgrind-16 test-linux64/opt-mochitest-valgrind-15 test-linux64/opt-mochitest-valgrind-13 test-linux64/opt-mochitest-valgrind-12 test-linux64/opt-mochitest-valgrind-11 test-linux64/opt-mochitest-valgrind-10 test-linux64/opt-mochitest-valgrind-19 test-linux64/opt-mochitest-valgrind-18 test-linux64/opt-mochitest-valgrind-40 test-linux64/opt-mochitest-valgrind-3 test-linux64/opt-mochitest-valgrind-29 test-linux64/opt-mochitest-valgrind-23 test-linux64/opt-mochitest-valgrind-14 test-linux64/opt-mochitest-valgrind-22 test-linux64/opt-mochitest-valgrind-30 test-linux64/opt-mochitest-valgrind-33 test-linux64/opt-mochitest-valgrind-1 test-linux64/opt-mochitest-valgrind-32 test-linux64/opt-mochitest-valgrind-39 test-linux64/opt-mochitest-valgrind-38 test-linux64/opt-mochitest-valgrind-5 test-linux64/opt-mochitest-valgrind-4 test-linux64/opt-mochitest-valgrind-7 test-linux64/opt-mochitest-valgrind-6 test-linux64/opt-mochitest-valgrind-2 test-linux64/opt-mochitest-valgrind-9 test-linux64/opt-mochitest-valgrind-8 When we get the runnable jobs, we build a list of jobs via ref_data_names: https://github.com/mozilla/treeherder/blob/f6ac262dd07b8cac14c5fe38c1f3bd6c642e5af7/treeherder/etl/seta.py#L171 we take the names and translate them into standardized tuples (build, type, test) where test is standardized to the buildbot convention. https://github.com/mozilla/treeherder/blob/f6ac262dd07b8cac14c5fe38c1f3bd6c642e5af7/treeherder/etl/seta.py#L184 what we use for the runnable jobs is a mapping of key/value pairs where: key = sanitized tuple value = raw ref_data_name from runnablejobs. this only works for build_system_types that match (which in this case we filter on 'taskcluster' or '*'). So given the raw_data_names_map with key/value pairs, we need to compare the key against the data from the jobpriorities database. in this papertrail message: Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,642] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,opt,mochitest-valgrind-8) in the list of accepted jobs the jobpriorities (sanitized) entry is: (taskcluster,linux64,opt,mochitest-valgrind-8) so we are trying to compare (mochitest-valgrind-8,linux64,opt) from jobpriorities to the ref_data_names_map keys and not finding it. I am not sure why this is the case, from my debugging it should be in there. On another note, there are cases of these failures: Feb 08 16:45:44 treeherder-stage app/web.1: [2017-02-08 21:45:44,642] WARNING [treeherder.seta.job_priorities:37] We did not find job priority (taskcluster,linux64,debug,mochitest-browser-chrome-e10s-8) in the list of accepted jobs when sanitized we would have: key = (mochitest-e10s-browser-chrome-8,debug,linux64) and we would find that in the jobpriorities table, except we try to unsanitize the buildbot names to taskcluster: https://github.com/mozilla/treeherder/blob/f6ac262dd07b8cac14c5fe38c1f3bd6c642e5af7/treeherder/seta/job_priorities.py#L25 so what we are looking for is: (mochitest-browser-chrome-e10s-8,debug,linux64) note how we change /e10s-browser-chrome/browser-chrome-e10s/ I believe an easy fix is to remove lines 25-30: https://github.com/mozilla/treeherder/blob/f6ac262dd07b8cac14c5fe38c1f3bd6c642e5af7/treeherder/seta/job_priorities.py#L25 that would reduce some of the pain, but not all of the messages.
Component: Treeherder → Treeherder: SETA
Blocks: 1326102
Also note we are getting the 'we did not find job priority...' message for jobs from platforms and jobtypes that SETA doesn't support i.e. those listed in: https://github.com/mozilla/treeherder/blob/master/treeherder/seta/settings.py This is because in job_priorities:_process, we are going through the jobs in the JP table and processing them to make sure they're in the latest list of runnable jobs: https://github.com/mozilla/treeherder/blob/master/treeherder/seta/job_priorities.py#L18 At that point we've weeded them out of the ref_data_map (list of runnable jobs) that we're searching in but the unsupported jobs are still in the job priorities list that we're feeding in, so obviously they can't be found (as expected). We should remove the unsupported jobs first before sending it into _process so that we can avoid getting these warning messages in the log for jobs that we don't even support.
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Attachment #8838698 - Flags: review?(jmaher)
Comment on attachment 8838698 [details] [review] [treeherder] rwood-moz:bug1338102 > mozilla:master wow, that was so simple!
Attachment #8838698 - Flags: review?(jmaher) → review+
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: Tree Management → Tree Management Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: