1339141 - consider whitelisting certain task definition that we do not support

Status: RESOLVED FIXED

(Tree Management Graveyard :: Treeherder: SETA, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we currently search for certain types of tasks (builds, lint, etc.) and ignore them from the data we use for SETA.

We do some attempt at sanitizing data, I would like to make this a single point of sanitzing, and maybe find a way to do this in-tree:
https://github.com/mozilla/treeherder/blob/master/treeherder/etl/seta.py#L59
https://github.com/mozilla/treeherder/blob/master/treeherder/etl/seta.py#L117
https://github.com/mozilla/treeherder/blob/master/treeherder/etl/seta.py#L94


In looking at a production log, I see a list of possibly job types that we should look into and make sure we are skipping:
[.] We were unable to parse android-checkstyle/*
[.] We were unable to parse android-api-15-gradle-dependencies/*
[.] We were unable to parse android-lint/*
[.] We were unable to parse spidermonkey-sm-*
[.] We were unable to parse clang-format-3.9/*
[.] We were unable to parse hazard-linux64-*
[.] We were unable to parse valgrind-linux64-valgrind/opt/*
[.] We were unable to parse artifact-build-linux64-artifact/opt/*

these below have odd names, possibly these are older, or different systems, or..?

[.] We were unable to parse [TC] MacOSX64 Opt/c3d69c732469e454d70ce6f42ae3745dd91bee6d 
[.] We were unable to parse [TC] MacOSX64 Static Analysis Opt/478845970968835752629e2ab231c3ac778f9402 
[.] We were unable to parse [TC] MacOSX64 Static Analysis Opt/a72d81e805cafc2b0e9b7221022549c5ee31a94f 
[.] We were unable to parse [TC] MacOSX64 Debug/aa14049e8cae018b8106542375152a0f934d1c31 
[.] We were unable to parse [TC] MacOSX64 Debug/648e1924e985cf9300e0a1bf837e97f341fb7eb0 

[.] We were unable to parse Bogo tests/1f2b7d048767ea737cd95ecb3d3c16398ddd535e 
[.] We were unable to parse Bogo tests/65f22443263c84bf2d0e72db4ae6f86d2a85bf77 
[.] We were unable to parse Bogo tests/cb0781ad0d0fe4df2c09da3dde36296066ad7d2a 
[.] We were unable to parse Bogo tests/8969a18d3524fee8f16ace223393d11504869c1e 
[.] We were unable to parse Bogo tests/3819aaeef4bee9037d1326237f6bbed943aac253 

[.] We were unable to parse [TC] Linux64 Debug/2af27ed0086325d199b14e37e0f3a5a847fd5d53 
[.] We were unable to parse [TC] Linux64 Debug/b18afde45076f2417bb47a69597392b09bb57be2 
[.] We were unable to parse [TC] Linux64 ASan Opt/0965a3986932d15daa99456b295164b6bbbfc9b7 
[.] We were unable to parse [TC] Linux64 ASan Opt/4a5bef54eef7013239fed181e59036524b5e089f 
[.] We were unable to parse [TC] Linux64 ASan Debug/c19913a56bf76d1f6cd2f918d1b597b5a9ba8cfc 
[.] We were unable to parse [TC] Linux64 ASan Debug/33080b8d3089ade806d2dee66502fa78ed73839d 
[.] We were unable to parse [TC] Linux64 Static Analysis Opt/ce04c1feb6b0a114f2c95ede8b02c7dc2e4aa5ea 
[.] We were unable to parse [TC] Linux64 Static Analysis Opt/aecb9b004fbe82c87292cac4a2ef8d18cd423c18 
[.] We were unable to parse [TC] Linux64 PGO Opt/9743c69f21f052afc4d3e74efb3e9f62563e2184 
[.] We were unable to parse [TC] Linux64 Artifact Opt/061104ffcc2d90c7bbdb7f6ef1060f4420f5a709 
[.] We were unable to parse [TC] Linux64 Opt/e8278ad2dc4c9b5385b7a2a82d1b9eb08fc6a57a 
[.] We were unable to parse [TC] Linux64 Opt/a3e0854cee62aabda126361c9f342664f1c0b242 
[.] We were unable to parse [TC] Linux64 Valgrind Opt/ce94918823dc6d59bff679830ed9cc45fe39fc30 
[.] We were unable to parse [TC] Linux64 Valgrind Opt/58b385f39a0c5ca351e88f5d9d62117e95154a67 
[.] We were unable to parse [TC] Linux64 PGO Opt/20f2af7d2e4a286789a63b8746716f7c65ad3403 
[.] We were unable to parse [TC] Linux64 PGO Opt/9743c69f21f052afc4d3e74efb3e9f62563e2184 
[.] We were unable to parse [TC] Linux64 web-platform-tests-e10s-4/0ef7eeb97fddb23eef5b9d13f2e3c9dc9b4046bd 
[.] We were unable to parse [TC] Linux64 web-platform-tests-e10s-4/f90145ba20373709f27cbc9ecaaad48a54241caa 
[.] We were unable to parse [TC] Linux64 web-platform-tests-e10s-4/f7a61589f6bf5975916f0b4b4e6ba335396632f6 
[.] We were unable to parse [TC] Linux64 web-platform-tests-4/3aa548f8b7cf8cd5fb52c373bc0cfb77986d46ad 
[.] We were unable to parse [TC] Linux64 web-platform-tests-4/dd1475331e9b30ea000f858e05c66f6ca4309998 
[.] We were unable to parse [TC] Linux64 web-platform-tests-4/6e6c839d33dd3d31172edaa45cb890e79d2777ab 

[.] We were unable to parse Linux 32 (opt)/c7209f6ed9a96ce9e3c42f88159ffd911aed6dce 
[.] We were unable to parse Linux 64 (opt)/73f2ca46c099e2558b753d0318e452690bfacf88 
[.] We were unable to parse Linux 64 (opt) w/ gcc-4.8/cad6fd7a8f250f7be5cc9fa099155608b2a0c64a 
[.] We were unable to parse Linux 32 (opt) w/ gcc-6.1/8cd4fb3ba029fee400d158b1bf682b1134367365 
[.] We were unable to parse Linux 64 (opt) w/ gcc-6.1/b1bab18cae1110bb0f799ccc95b145fd98e7a435 
[.] We were unable to parse Linux 32 (opt) w/ gcc-4.8/1b5cbf92a855e0cd52b96e4837abf849bd663826 

[.] We were unable to parse Linux 64 (debug)/d07d40d73dcc53c9565bd80fa5873bcb844dfea1 
[.] We were unable to parse Linux 32 (debug)/16955f62a8373cc1f17ccbb6d68cb664233b7e04 
[.] We were unable to parse Linux 64 (debug) w/ NSS_DISABLE_LIBPKIX=1/ee7554d0fa06d01f502e63dc3a93ff57eb494ae5 
[.] We were unable to parse Linux 32 (debug) w/ gcc-6.1/4c6fac4540867c829160fc2081ba80ec538aef80 
[.] We were unable to parse Linux 64 (debug) w/ gcc-6.1/72aba356204415f654e7bc3489451ad1f6f039ee 
[.] We were unable to parse Linux 32 (debug) w/ gcc-4.8/4b168e1bc8986a625b25bcd5efe27f40bbf75db2 
[.] We were unable to parse Linux 64 (debug) w/ gcc-4.8/c050d64b8bcd43f07d8c1401dabdc4edcc4b6510 
[.] We were unable to parse scan-build-3.9/d02d8932874b247eea2ef249866d2cb970439124 

[.] We were unable to parse [TC] Linux32 Opt/c7ecd19691607d503a36f37d0c06380c41340b60 
[.] We were unable to parse [TC] Linux32 Debug/322b11ab228b5b3057e503dc8fc622091a8fe57b 
[.] We were unable to parse [TC] Linux32 Debug/d530095b720d6e085aebc871807bdc78a951132b 
[.] We were unable to parse [TC] Linux32 Opt/45a9097b8948861850d83c3247a8a15dca308bc3 
[.] We were unable to parse [TC] Browser Hazard Linux/8d0657454790f9c10dab834bbe033025b058025f 
[.] We were unable to parse [TC] Browser Hazard Linux/4c8294c820958546c727152226aa6d79a7dfc556 
[.] We were unable to parse [TC] JS Shell Hazard Linux/dc49f629c2c2bd780cb4d4d3540ee760aa048886 
[.] We were unable to parse [TC] Android armv7 API 15+ Dbg/2a617bce9c6131a157331b7866f9b83080124078 
[.] We were unable to parse [TC] Android armv7 API 15+/2289a93b05a8f3be29cbbb720dd0d6007dcd34e4 
[.] We were unable to parse [TC] Android x86/b09d7240f41b7b3a83a4184ce7fc79b0edfb3a68 
[.] We were unable to parse [TC] - ESLint/cd5c5e1e5f737fa99c0a7f0e1b6f6a664c2ac1a3 
[.] We were unable to parse [TC] Spidermonkey Plain Debug/b692a574b452fae523e96c288e863f59b8581e46 
[.] We were unable to parse [TC] Spidermonkey Plain/b3687a4f904739969623524377b08e4f8224a286 
[.] We were unable to parse [TC] Spidermonkey Root Analysis/a5c608980b7698b759e934ad4ffb5c7f376f4423 
[.] We were unable to parse [TC] Spidermonkey Compactng/eb3a597213c005a2ef5f53f8cc2e87baab4b3eea 
[.] We were unable to parse [TC] Spidermonkey Package/7fcda99c2cd14fede92a80efa3d7358badd89223 
[.] We were unable to parse [TC] Spidermonkey Non-Unified Debug/681d9afed4366b75d88f26b33d7c8875aaa87e7a 
[.] We were unable to parse [TC] Spidermonkey ARM64 simulator/834fc095e4ed25a1f1d06c35bc50f6f4894e647d 
[.] We were unable to parse [TC] Spidermonkey ARM sim/ccf0bfc5f59adb2afcfb92098fc30ef36bf143d1 


If we could reduce these known warnings, then what we print in the papertrail will be very beneficial.

Comment 2

[Robert Wood [:rwood]]

Another place where more job dropping is done, is here (jobs are filtered out when the JobPriority table is updated, via analyze_failures:

https://github.com/mozilla/treeherder/blob/master/treeherder/etl/seta.py#L142

Comment 3

[Robert Wood [:rwood]]

So I believe the best route is to have a blacklist of tasks/jobs that we want to exclude from SETA. Should be a blacklist because if it's a whitelist we couldn't automatically have new job types included without adding them to the whitelist first which would be a pain.

I propose:

- storing the blacklist in .json in tree /taskcluster/taskgraph/util/seta-blacklist.json

In tree /taskcluster/taskgraph/util/seta.py:is_low_value_task, where the gecko decision task calls SETA, first check if the task is in the blacklist. If it is then don't even call SETA (and return false i.e. it's not a low value task, since we don't want SETA to optimize it out).

Then have treeherder SETA also check the same blacklist here (in _ignore):
https://github.com/mozilla/treeherder/blob/master/treeherder/etl/seta.py#L53

If the job is in the blacklist then it will be ignored. The _ignore function is already called by both the SETA job_types API and the job-priority API. Also I would propose that when updating the job priority table in the first place (seta/update_job_priority.py: update_job_priority_table) instead of calling valid_platform (comment 2) we call the same _ignore function and have the invalid platforms included in the same blacklist.

I will rename _ignore to something like 'is_job_blacklisted'.

:jmaher, what do you think?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I like the proposal for moving this in tree.  Please add comments, etc to the file so we can understand how it is used and how to add things to it.

Comment 5

[Robert Wood [:rwood]]

Actually, just realized this won't work, as how will treeherder SETA access the blacklist. We don't want to pass it in every time the gecko decision task calls SETA, and also will need to access it elsewhere in treeherder seta besides the entrypoint that the decision task uses. I don't think it can be in tree, I think it will need to be in treeherder/seta.

Comment 6

[GitHub Autolander Bot]

Attachment: [treeherder] rwood-moz:bug1339141 > mozilla:master

Comment 7

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/945277f092e52103600de7e22cbc02dc7bc79432
Bug 1339141 - SETA: Add a settings file to filter out platforms/jobs/testtypes (#2181)