1243123 - Re-write of SETA and support of TaskCluster

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Armen [:armenzg]]

We currently only support Buildbot jobs by grabbing the data from Ouija.

We might need to see if we need the data on Ouija or if we should be using ActivateData instead.

Comment 1

[Armen [:armenzg]]

We can see the slave info ('unknown') and testtype being messed up.

This is for jobs in here https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=7e32d3531b96&filter-searchStr=tc&group_state=expanded
<jmaher> select slave,result,duration,platform,buildtype,testtype from testjobs where revision='7e32d3531b96' and platform='linux64';
<jmaher>  unknown                  | success    |      840 | linux64  | debug     | e4b1937c1441c60f8d18e5c435c3457a37ccea20 |
<jmaher> | unknown                  | success    |     1776 | linux64  | debug     | 32922949cd054fc09aeac83a7a34727f5ab31ad8 |
<jmaher> | tst-linux64-spot-1768    | success    |     2214 | linux64  | opt       | jittest-2                                |
<jmaher> | tst-linux64-spot-1190    | success    |     2017 | linux64  | debug     | mochitest-1                              |


Given a revision 7e32d3531b96 we can use this api [1]
We can then determine the resulset_id (25761) [2]
We can then use job_type_symbol and job_group_name to clasify.
This is the code which uploads the data from treeherder [3]

We have data classified as [4] which a sample piece of data is [5] (Jit1)

Maybe:
> If buildbot:
>   testtype = r[i("ref_data_name")].split()[-1]
> elif taskcluster:
>   testtype = r[i("job_type_name")].split()[-1]

[1] https://treeherder.mozilla.org/api/project/mozilla-inbound/resultset/?format=json&full=true&revision=7e32d3531b96&with_jobs=true
[2] https://treeherder.mozilla.org/api/project/mozilla-inbound/jobs/?count=2000&result_set_id=25761&return_type=list
[3] https://github.com/dminor/ouija/blob/master/src/updatedb.py#L130

[4]
    "job_property_names": [
        "submit_timestamp",
        "build_system_type",
        "machine_name",
        "job_group_symbol",
        "job_group_name",
        "platform_option",
        "job_type_description",
        "result_set_id",
        "build_platform_id",
        "result",
        "id",
        "machine_platform_architecture",
        "job_type_id",
        "build_platform",
        "job_guid",
        "job_type_name",
        "ref_data_name",
        "platform",
        "state",
        "running_eta",
        "end_timestamp",
        "build_os",
        "who",
        "option_collection_hash",
        "failure_classification_id",
        "job_type_symbol",
        "reason",
        "job_group_description",
        "tier",
        "job_coalesced_to_guid",
        "machine_platform_os",
        "start_timestamp",
        "build_architecture",
        "last_modified",
        "signature",
        "job_group_id"
    ],

[5]
        [
            1453759874,
            "taskcluster",
            "unknown",
            "tc",
            "Desktop jittests",
            "debug",
            "",
            25761,
            106,
            "success",
            20450192,
            "-",
            4076,
            "linux64",
            "397ba94c-cf15-443e-a51b-d88b01026eb6/0",
            "[TC] Linux64 jittests 1",
            "931637ca8ab1de3354099d809fa083087270d3cf",
            "linux64",
            "completed",
            2253,
            1453765054,
            "-",
            "[e-mail]",
            "32faaecac742100f7753f0c1d0aa0add01b4046b",
            1,
            "Jit1",
            "scheduled",
            "fill me",
            2,
            null,
            "-",
            1453763117,
            "-",
            "2016-01-25T23:37:35",
            "931637ca8ab1de3354099d809fa083087270d3cf",
            179
        ],

Comment 2

[Armen [:armenzg]]

This is what active builders look like for Buildbot:
http://alertmanager.allizom.org/data/setadetails/?date=2016-01-10&buildbot=1&branch=mozilla-inbound&active=1

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I have initial work to get taskcluster job names into ouija.  This will require additional parsing of the 'testtype', but as it stands we have useful data.  Some hacky debugging locally yields this data:

tc: testtype: linux64: debug:  mochitest-browser-chrome M(bc7)
tc: testtype: linux64: debug:  web-platform-tests-e10s 7
tc: testtype: linux64: debug:  jsreftest 2
tc: testtype: linux64: debug:  mochitest-browser-chrome M(bc5)
tc: testtype: linux64: debug:  mochitest-browser-chrome e10s M(bc3)
tc: testtype: linux64: debug:  web-platform-tests 7
tc: testtype: linux64: debug:  mochitest-browser-chrome e10s M(bc5)
tc: testtype: linux64: debug:  mochitest-chrome 1
tc: testtype: linux64: debug:  mochitest-a11y
tc: testtype: linux64: debug:  mochitest-chrome 2
tc: testtype: linux64: debug:  mochitest-chrome 3


you can see the testtype has M(bcX) in some cases, and others just a chunk number.  I would like to fix that if possible in taskcluster to just be the chunk number- at the very least uniform !!

Comment 4

[Armen [:armenzg]]

For now, let's use this bug for the re-write as explained on https://public.etherpad-mozilla.org/p/seta-rewrite

Comment 5

[Armen [:armenzg]]

Once we get to wanting to modify the in-tree scheduling we should see where dustin is standing with the changes he's planning to do on Q2:
https://github.com/djmitche/taskcluster-in-tree-taskgraph

Comment 6

[Armen [:armenzg]]

From MikeLing:
> I found the taskcluster jobs (e.g jobs on Linux debug64) can't been displayed on SETA 
> is because we don't know its job group symbol and job type symbol. But we do have job
> group symbol for them on treeherder, like M for  mochitest-browser-chrome-7 and W-e10s 
> or  web-platform-tests-e10s-3. So do I need hard code them on SETA in order to display
> job name appropriately?

No, do not hard code. What is the data source you're using? I thought SETA used data from Treeherder. jmaher knows SETA. I don't really.
All TaskCluster jobs have the information you need.

> And I can't find job like [TC] JS Shell Hazard Linux, [TC] Spidermonkey Non-Unified Debug
> and [TC] Browser Hazard Linux on treeherder which ran on linux64 debug platform, 
> could you tell me why? or I miss something?

Where are you looking for those jobs? I see one of them on m-c [1]
What are looking to find about them?

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=JS%20Shell%20Hazard%20Linux&selectedJob=4115248

Comment 7

[MikeLing]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #6)
> From MikeLing:
> > I found the taskcluster jobs (e.g jobs on Linux debug64) can't been displayed on SETA 
> > is because we don't know its job group symbol and job type symbol. But we do have job
> > group symbol for them on treeherder, like M for  mochitest-browser-chrome-7 and W-e10s 
> > or  web-platform-tests-e10s-3. So do I need hard code them on SETA in order to display
> > job name appropriately?
> 
> No, do not hard code. What is the data source you're using? I thought SETA
> used data from Treeherder. jmaher knows SETA. I don't really.
> All TaskCluster jobs have the information you need.

So, I found we need to update this file: https://github.com/dminor/ouija/blob/master/src/treecodes.py to make sure each job have its correspond symbol.

> > And I can't find job like [TC] JS Shell Hazard Linux, [TC] Spidermonkey Non-Unified Debug
> > and [TC] Browser Hazard Linux on treeherder which ran on linux64 debug platform, 
> > could you tell me why? or I miss something?
> 
> Where are you looking for those jobs? I see one of them on m-c [1]
> What are looking to find about them?
> 
> [1]
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-
> searchStr=JS%20Shell%20Hazard%20Linux&selectedJob=4115248

I have discussed this with Joel on IRC and he told me:
>  those are build jobs, not test jobs- SETA is only designed to work with tests
So, I think it's ok to ignore them :)

Comment 8

[Armen [:armenzg]]

(In reply to MikeLing from comment #7)
> (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> comment #6)
> > From MikeLing:
> > > I found the taskcluster jobs (e.g jobs on Linux debug64) can't been displayed on SETA 
> > > is because we don't know its job group symbol and job type symbol. But we do have job
> > > group symbol for them on treeherder, like M for  mochitest-browser-chrome-7 and W-e10s 
> > > or  web-platform-tests-e10s-3. So do I need hard code them on SETA in order to display
> > > job name appropriately?
> > 
> > No, do not hard code. What is the data source you're using? I thought SETA
> > used data from Treeherder. jmaher knows SETA. I don't really.
> > All TaskCluster jobs have the information you need.
> 
> So, I found we need to update this file:
> https://github.com/dminor/ouija/blob/master/src/treecodes.py to make sure
> each job have its correspond symbol.
> 
More than updating I would prefer it if we could generate it dynamically. It does not seem to difficult to do.

> > > And I can't find job like [TC] JS Shell Hazard Linux, [TC] Spidermonkey Non-Unified Debug
> > > and [TC] Browser Hazard Linux on treeherder which ran on linux64 debug platform, 
> > > could you tell me why? or I miss something?
> > 
> > Where are you looking for those jobs? I see one of them on m-c [1]
> > What are looking to find about them?
> > 
> > [1]
> > https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-
> > searchStr=JS%20Shell%20Hazard%20Linux&selectedJob=4115248
> 
> I have discussed this with Joel on IRC and he told me:
> >  those are build jobs, not test jobs- SETA is only designed to work with tests
> So, I think it's ok to ignore them :)

Cool!

Comment 9

[MikeLing]

Attachment: PR for make linux64 debug job displayable

Comment 10

[Armen [:armenzg]]

Comment on attachment 8764213 [details] [review]
PR for make linux64 debug job displayable

I would merge the code as-is but I would prefer it if we created the patched data structure to be generated automatically.

FYI jmaher is on PTO today.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8764213 [details] [review]
PR for make linux64 debug job displayable

thanks for this MikeLing!  I have merged this in github.  I agree a dynamic solution will be required, ideally from the runnable api.  For now, this gets us to completion of showing the jobs.

Comment 12

[MikeLing]

Attachment: PR for using runnable api instead of uniquejobs table(buildbot for now)

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8765171 [details] [review]
PR for using runnable api instead of uniquejobs table(buildbot for now)

I left a few comments in the PR- ideally Armen can comment on the use of runnable api if he sees it being misused or not robust enough.

Thanks for getting this code posted!

Comment 14

[Armen [:armenzg]]

Comment on attachment 8765171 [details] [review]
PR for using runnable api instead of uniquejobs table(buildbot for now)

I left feedback in the PR.
Please flag me again once addressed all issues.

Comment 15

[MikeLing]

Comment on attachment 8765171 [details] [review]
PR for using runnable api instead of uniquejobs table(buildbot for now)

Ask feedback again, thank you :)

Comment 16

[Armen [:armenzg]]

Comment on attachment 8765171 [details] [review]
PR for using runnable api instead of uniquejobs table(buildbot for now)

Did you mean to add feedback flags for jmaher and I?

I'm setting mine just in case that is what you intended to do.

Comment 17

[MikeLing]

Comment on attachment 8765171 [details] [review]
PR for using runnable api instead of uniquejobs table(buildbot for now)

Yeah, sorry I forget to add the requestee. Thank you for the remind :)

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8765171 [details] [review]
PR for using runnable api instead of uniquejobs table(buildbot for now)

I see a lot of small changes to make, but this looks like a great overall direction.  I like the seta.js cleanup with underscore.js, that is excellent.

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8765171 [details] [review]
PR for using runnable api instead of uniquejobs table(buildbot for now)

thanks for updating the patch and addressing the concerns.

Comment 20

[MikeLing]

Attachment: separate buildbot jobs from taskcluster job when we query it

This PR is for separating buildbot jobs from taskcluster jobs when we query it. So we can make Gecko decision task only based on taskcluster jobs to make schedule.

Comment 21

[Armen [:armenzg]]

Comment on attachment 8770058 [details] [review]
separate buildbot jobs from taskcluster job when we query it

The patch looks good but we have to switch to using task labels.
dustin will be landing his code soon (bug 1281004).

Comment 22

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8770058 [details] [review]
separate buildbot jobs from taskcluster job when we query it

outside of the feedback :armenzg provided, I don't have much else.

Comment 23

[MikeLing]

Comment on attachment 8770058 [details] [review]
separate buildbot jobs from taskcluster job when we query it

https://github.com/mozilla/ouija/pull/101

Comment 24

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8770058 [details] [review]
separate buildbot jobs from taskcluster job when we query it

merged the pull request!