Closed Bug 1608427 Opened 2 years ago Closed 1 year ago

Schedule some builds only on full autoland pushes

Categories

(Testing :: General, enhancement, P3)

Version 3
enhancement

Tracking

(firefox76 fixed)

RESOLVED FIXED
Tracking Status
firefox76 --- fixed

People

(Reporter: marco, Assigned: bc)

References

(Blocks 1 open bug)

Details

(Whiteboard: [ci-costs-2020:done])

Attachments

(14 files, 5 obsolete files)

1.89 KB, text/plain
Details
47 bytes, text/x-github-pull-request
Details | Review
47 bytes, text/x-github-pull-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-github-pull-request
Details | Review
1.38 KB, application/json
Details
47 bytes, text/x-github-pull-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-github-pull-request
Details | Review
8.94 KB, patch
Details | Diff | Splinter Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-github-pull-request
Details | Review

We currently schedule all builds on all autoland pushes because we need a build readily available in case we need to backfill to find a patch causing a regression.

There are some builds which have no tests though, e.g. the fuzzing builds. We could consider these builds as being similar to test jobs: we only want to make sure they pass, but we don't have to use them to run tests. Since they are not needed for backfilling, we could avoid scheduling them on all autoland pushes, but just schedule them on the full ones.

This way we still guarantee the builds are green on mozilla-central.

Even if we apply this to fuzzing builds alone, we could save around 3-4 hours of running time per push.

there is bug 1598744 which is intended to reduce tier2/3 jobs as much as possibly on autoland; I suspect many of these build jobs fall into the tier-2 category :)

:bc, how is bug 1598744 coming along- maybe solving this first would reduce the list :)

Flags: needinfo?(bob)

It fell off my radar. On it now.

Flags: needinfo?(bob)

Note this would be a bit lighter than bug 1598744, as we'd still be running these tasks on autoland, just not as frequently as now. They'd be basically handled as test tasks.

Priority: -- → P3

I'm not clear as to how we would restrict these builds to "full" autoland pushes. Does anyone have a suggestion?

I think full = every 10th push; although if we are fine with noopt builds on m-c and backfill as needed, I don't know why we cannot do the same and have the sheriffs manually add the jobs and backfill them if there is an issue.

I see 2 possible options:

  1. have SETA ignore certain builds (would take some hacking)
  2. only run those builds on m-c and try --full;

I would vote for #2 for the fuzzing builds and call it a day.

we have win/aarch64 as well. I suspect one or two others could be quick wins, but reducing the fuzzing, noopt, etc. first would allow for seeing it easier.

I think with regard to win aarch64, I think tests are restricted to mozilla-central and/or try. We build everywhere though but I'm not sure that is appropriate to change.

so fuzzing and noopt had no tests at all on m-c, therefore the risk of not having the builds available for backfilling when we see test failures is minimal. In 2019 there were 4 regressions that were unique to win-aarch64, the resolution of those regressions were 3 test fixes and 1 infrastructure fix- we would still have a need for backfilling, but the only risk is time.

Assignee: nobody → bob
Status: NEW → ASSIGNED
Attachment #9122507 - Attachment description: Bug 1608427 - Only run fuzzing builds on mozilla-central and try with --full, r=decoder,tsmith → Bug 1608427 - Only run fuzzing builds on mozilla-central and try with --full, r=decoder,tsmith,ahal

leave open for win aarch64 work.

Keywords: leave-open

This way we still guarantee the builds are green on mozilla-central.

With the planned changes (comment 5), does this statement from comment 0 still hold? We need things to be backed out if the fuzzing build process breaks, so central remains green.

Flags: needinfo?(jmaher)
Flags: needinfo?(bob)

decoder: You are asking if it will be possible to backfill the fuzzing builds if something breaks? I think so yes. These are tier 1 (though I can't for the life of me find where that is specified) so the sheriff's should be able to backfill on autoland to find the failing push and back it out. Aryx: Is that correct?

Flags: needinfo?(bob) → needinfo?(aryx.bugmail)

this is done for other tier-2 jobs (windows/aarch64), we don't backout, but typically within 24 hours the root cause is found and proper people are notified.

Flags: needinfo?(jmaher)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #14)

this is done for other tier-2 jobs (windows/aarch64), we don't backout, but typically within 24 hours the root cause is found and proper people are notified.

Fuzzing builds are Tier1 and if they break, the offending commit needs to be backed out, notifying us is not sufficient if the builds fail. mozilla-central needs to be green.

every build needs to be green? currently there is a chance that a build will fail on mozilla-central, then there would be a fix up. In the new model if it fails it would be determined the root cause before the next build, I assume we could back that out.

How often has a fuzzzy build failed in the last 6 months? I am not finding evidence of it failing and being marked as fixed_by_commit in 2019, so the risk is low based on past history.

:bc, could you figure out the cost of fuzzy builds on autoland via a raw query?

Flags: needinfo?(bob)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #16)

every build needs to be green? currently there is a chance that a build will fail on mozilla-central, then there would be a fix up. In the new model if it fails it would be determined the root cause before the next build, I assume we could back that out.

I think it would be okay if the build failure would automatically be resolved in about 24 hours. But anything that takes longer or even manual intervention (no backout) is not going to be okay.

How often has a fuzzzy build failed in the last 6 months? I am not finding evidence of it failing and being marked as fixed_by_commit in 2019, so the risk is low based on past history.

The risk is not purely how often it failed in the last 6 months. Fuzzing builds can break e.g. with any Clang update, any build system or linker change or even if people land their own fuzzing targets that don't build properly for some reason. That said, I am pretty certain the builds broke at least once (on Mac) in 2019. Is "fixed_by_commit" including fixed by backout?

In any case, what was discussed with Marco (in comment 0) is perfectly acceptable to us. It would also be acceptable if we have guaranteed green m-c builds in a reasonable time window (e.g. 24 hours seems fine to me). What is not acceptable is build regression where no backouts occur and therefore we fuzz an older revision (which causes us a lot of problems) or even have to stop fuzzing completely.

Here's another wild idea to solve this and save money:

How about we introduce a set of builds that sheriffs trigger manually on autoland before a merge and only merge if they are green? Builds don't take long anymore and if the build set only includes builds with low failure probability, then this should be a minimum amount of work. If one of the builds fails (and it is tier1), it blocks the merge until the backfilling process has identified the regressor and it can be backed out.

(In reply to Bob Clary [:bc:] from comment #13)

decoder: You are asking if it will be possible to backfill the fuzzing builds if something breaks? I think so yes.

These are tier 1 (though I can't for the life of me find where that is specified) so the sheriff's should be able to backfill on autoland to find the failing push and back it out.

The definition of Tier 1 and its requirements requires a tier 1 task to also run on integration branches. If it only runs on central, it shall be set as tier 2. Practically, a new tier system would make sense

  • Tier 1: current Tier 1, runs on autoland and central
  • Tier 2: only runs on central, failure will trigger backout on autoland which will get merged to central 6-12h later. Backouts always have the risk to break something, that's why backouts on central shall be avoided. They would affect e.g. developers pushing to Try or creating artifact builds.
  • Tier 3: only runs on central, failure will be handled in a bug with the suite owner and the person regressing it needinfoed. It's the suite owners duty to drive the fix, else the test or task will eventually get disabled for permafailing.
  • Tier 4: current Tier 3, experimental stuff or new platforms and suits which are getting set up and only looked at by their developers.

Changing the tiers will require treeherder changes. At least short term, sheriffs could handle it if the follow-up rule is simple like "If it is ccov failure file a bug, else back it out on autoland."

The offending bug gets either found by searching for affected build or test files on hg.mozilla.org or by backfilling it the former is insufficient.

Flags: needinfo?(aryx.bugmail)

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #19)

The definition of Tier 1 and its requirements requires a tier 1 task to also run on integration branches.

We could also change this definition to allow for running certain Tier1 builds on integration branches only once prior to merging, rather than on every push. This would also avoid the problem of having to back out things from central.

If these builds get manually requested on autoland (after waiting for the other builds and tests to be eligible for a merge) and fail, that would require for the check-in causing the regression to be identified until the Nightly starts, including the merge from autoland to central and the backout from central (= 2 pushes, else tools like mozregression break). It will also require a merge back. Having the builds run as part of full set of tasks with every 10th push or failure being managed like outline in comment 19 are better defined processes with less assumptions and overhead.

Attachment #9122507 - Attachment description: Bug 1608427 - Only run fuzzing builds on mozilla-central and try with --full, r=decoder,tsmith,ahal → Bug 1608427 - Only run fuzzing builds on mozilla-central and try with --full, r=decoder,tsmith
Attachment #9122507 - Attachment description: Bug 1608427 - Only run fuzzing builds on mozilla-central and try with --full, r=decoder,tsmith → Bug 1608427 - Only run fuzzing builds on mozilla-central and try with --full, r=decoder,tsmith,ahal.

Updated again with autoland and mozilla-central and with only fuzzing and not asan builds.

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #16)

:bc, could you figure out the cost of fuzzy builds on autoland via a raw query?

I had a bug in my costs script which ignored run_on_projects ["all"]. I've fixed that and here are the results for asan builds (Bof, Bf) on autoland all tiers for the firefox 72 release cycle.

| provisionerId | workerType | project         | tier | suite | groupSymbol | symbol | collection | cost    | label                                                                                                  |
|---------------|------------|-----------------|------|-------|-------------|--------|------------|---------|--------------------------------------------------------------------------------------------------------|
| gecko-3       | b-linux    | autoland        | 1    |       | nil         | Bf     | debug      | 1024.16 | build-android-x86-fuzzing/debug, build-linux64-fuzzing/debug, build-macosx64-fuzzing/debug             |
| gecko-3       | b-win2012  | autoland        | 1    |       | nil         | Bf     | debug      | 627.79  | build-win64-fuzzing/debug                                                                              |
| gecko-3       | b-linux    | autoland        | 1    |       | nil         | Bof    | asan       | 963.31  | build-android-x86_64-asan-fuzzing/opt, build-linux64-asan-fuzzing/opt, build-macosx64-asan-fuzzing/opt |
| gecko-3       | b-win2012  | autoland        | 1    |       | nil         | Bof    | asan       | 678.72  | build-win64-asan-fuzzing/opt                                                                           |
| gecko-3       | b-linux    | autoland        | 1    |       | nil         | Bof    | tsan       | 247.52  | build-linux64-tsan-fuzzing/opt                                                                         |
| gecko-3       | b-linux    | autoland        | 1    |       | SM          | f      | opt        | 66.27   | spidermonkey-sm-fuzzing-linux64/opt                                                                    |
|               |            | autoland        |      |       |             |        |            | 3607.77 |                                                                                                        |
| gecko-3       | b-linux    | mozilla-central | 1    |       | nil         | Bf     | debug      | 44.14   | build-android-x86-fuzzing/debug, build-linux64-fuzzing/debug, build-macosx64-fuzzing/debug             |
| gecko-3       | b-win2012  | mozilla-central | 1    |       | nil         | Bf     | debug      | 30.23   | build-win64-fuzzing/debug                                                                              |
| gecko-3       | b-linux    | mozilla-central | 1    |       | nil         | Bocf   | asan       | 31.08   | build-linux64-asan-fuzzing-ccov/opt                                                                    |
| gecko-3       | b-linux    | mozilla-central | 1    |       | nil         | Bocf   | opt        | 27.16   | build-linux64-fuzzing-ccov/opt                                                                         |
| gecko-3       | b-linux    | mozilla-central | 1    |       | nil         | Bof    | asan       | 42.96   | build-android-x86_64-asan-fuzzing/opt, build-linux64-asan-fuzzing/opt, build-macosx64-asan-fuzzing/opt |
| gecko-3       | b-win2012  | mozilla-central | 1    |       | nil         | Bof    | asan       | 31.50   | build-win64-asan-fuzzing/opt                                                                           |
| gecko-3       | b-linux    | mozilla-central | 1    |       | nil         | Bof    | tsan       | 9.63    | build-linux64-tsan-fuzzing/opt                                                                         |
| gecko-3       | b-linux    | mozilla-central | 1    |       | SM          | f      | opt        | 10.53   | spidermonkey-sm-fuzzing-linux64/opt                                                                    |
|               |            | mozilla-central |      |       |             |        |            | 227.23  |                                                                                                        |
|               |            |                 |      |       |             |        |            | 3835.00 |                                                                                                        |

Note this cost is for the full Dec 2 - Jan 6 release cycle.
PS. I'm going to redo my earlier cost analysis with the fixed script and review my previous results.

Flags: needinfo?(bob)

jmaher: It seems to me that the way forward is to hack seta to support fuzzing builds and to keep running the fuzzing builds on autoland with a schedule. Ok?

Flags: needinfo?(jmaher)

hacking SETA could be tricky, but it would open the door to more jobs to fall into this category, so if you can make this change work for SETA, I am game for that. I think there will only be a few places to fix this up, the challenge here is testing the fix.

Alternatively if we can find the changes that are most likely to trigger this to break (i.e. certain toolchains, specific build files changed, etc.) we could build the fuzzing builds then via a SCHEDULES rule (like we do for jsreftests/jittests). Maybe :decoder has a list of conditions and we could start there.

Flags: needinfo?(jmaher)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #24)

Alternatively if we can find the changes that are most likely to trigger this to break (i.e. certain toolchains, specific build files changed, etc.) we could build the fuzzing builds then via a SCHEDULES rule (like we do for jsreftests/jittests). Maybe :decoder has a list of conditions and we could start there.

Unfortunately this is really tricky. First of all, any code change within #ifdef FUZZING can cause build failures and this code can appear virtually everywhere in the codebase. Sometimes, this guard is in C/C++ code, sometimes it is in moz.build files. Reliably identifying this is probably going to be hard on its own.

In addition any toolchain change (esp. compiler and linker upgrades) can break the fuzzing builds. The same holds for changes in the build system itself (order of linking libraries or how things are linked together in general). So overall, this would be pretty hard to identify, definitely not possible with a simple file list.

Edit: Oh and I forgot to mention: Fuzzing builds can also break when APIs are changed that are used in compiled targets. You might not see these in your statistics as fixed by a commit because people do try runs and might have included this build type in their run already (and made the change).

I think :bc is looking at modifying SETA, lets see where we get with that in the next week or two and revisit if there are big roadblocks.

After finally getting docker and docker-compose to work on Fedora 31 and after some brain farts, I think this might do the job. It is much simpler than I thought.

The following diff shows the differences in the seta job priorities before the treeherder patch and after.

$ diff -u before.json after.json 
--- before.json	2020-01-29 12:05:46.039902623 -0800
+++ after.json	2020-01-29 12:05:55.862935891 -0800
@@ -1,6 +1,16 @@
 {
    "jobtypes" : {
       "2020-01-29" : [
+         "build-android-x86_64-asan-fuzzing/opt",
+         "build-linux64-asan-fuzzing-ccov/opt",
+         "build-linux64-asan-fuzzing/opt",
+         "build-linux64-fuzzing-ccov/opt",
+         "build-linux64-fuzzing/debug",
+         "build-linux64-tsan-fuzzing/opt",
+         "build-macosx64-asan-fuzzing/opt",
+         "build-macosx64-fuzzing/debug",
+         "build-win64-asan-fuzzing/opt",
+         "build-win64-fuzzing/debug",
          "test-android-em-7.0-x86_64-qr/debug-geckoview-crashtest-e10s",
          "test-android-em-7.0-x86_64-qr/debug-geckoview-crashtest-fis-e10s",
          "test-android-em-7.0-x86_64-qr/debug-geckoview-reftest-e10s-1",

Joel, Armen: What do you think?

Flags: needinfo?(jmaher)
Flags: needinfo?(armenzg)

This looks reasonable. I'm glad to hear that it was not too bad to set it up!

Flags: needinfo?(armenzg)

Let me know if you need me to deploy a full Treeherder instance to test any try pushes against (it is just few clicks). I'm sure you can change the gecko decision task to use an alternative Treeherder deployment to test your SETA changes.

I'm not sure try will help since all the action will be on autoland but I would love to be able to check that this doesn't do anything untoward on autoland. I just don't know how we would test that. Maybe we can just land the changes on both autoland and treeherder and see how it goes. As the creator of SETA, maybe Joel has more ideas.

my primary test will be looking at the .json data returned from the API. Given the diff in comment 27, I am happy.

We can test on try (thanks to recent changes to TH API and seta.py) to ensure that the decision task has no issues. I would modify seta.py:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/optimize/seta.py#25

and point it to a custom location or get your patch on staging and point it to staging.

Flags: needinfo?(jmaher)

Once you have a PR and you're ready to test against me let me know and we will get it deployed.
I don't think landing this would cause havoc but I'm not sure for sure. It will be a good exercise to try to test just for the sake of proving that it can be somehow tested.

See Also: → 1612539

I wonder if we have other builds with the same characteristics, we should look into that (I've filed bug 1612539).

(In reply to Bob Clary [:bc:] from comment #22)

Note this cost is for the full Dec 2 - Jan 6 release cycle.
PS. I'm going to redo my earlier cost analysis with the fixed script and review my previous results.

What about the cost in a normal month? (that is, a month without the holidays)

Could you publish the script on a GitHub repo so I can also use it sometimes?

Armen: I think we are ready to merge https://github.com/mozilla/treeherder/pull/5898 . If you don't mind, please let me know when it's deployed.

Flags: needinfo?(armenzg)

decoder : Once Armen merges the Treeherder changes, I plan to land this. Do you have any objections? If so, just r- the phabricator request and we'll proceed from there. I'll wait for your response before moving forward.

Flags: needinfo?(choller)
Attachment #9122507 - Attachment is obsolete: true

That patch was obsolete. I'll push the seta version now.

Flags: needinfo?(choller)
Attachment #9124570 - Attachment description: Bug 1608427 - SETA: Treat fuzzing build tasks as possible low value tasks,?. → Bug 1608427 - SETA: Treat fuzzing build tasks as possible low value tasks, r=jmaher!,decoder!

(Keeping NI)

I will deploy it to production tomorrow.

(In reply to Bob Clary [:bc:] from comment #39)

decoder : Once Armen merges the Treeherder changes, I plan to land this. Do you have any objections? If so, just r- the phabricator request and we'll proceed from there. I'll wait for your response before moving forward.

If this works as suggested in comment 0, then that seems fine to me! And I think in the future we will want to make more use of this "Tier 1.5".

The staging API has the fuzzing builds:

            "build-android-x86_64-asan-fuzzing/opt",
            "build-linux64-asan-fuzzing-ccov/opt",
            "build-linux64-asan-fuzzing/opt",
            "build-linux64-fuzzing-ccov/opt",
            "build-linux64-fuzzing/debug",
            "build-linux64-tsan-fuzzing/opt",
            "build-macosx64-asan-fuzzing/opt",
            "build-macosx64-fuzzing/debug",
            "build-win64-asan-fuzzing/opt",
            "build-win64-fuzzing/debug",

I will let you know when production is deployed and updated.

decoder: While looking at jmaher's review comment, I realized I'm missing spidermonkey-sm-fuzzing-linux64/opt. I should include that as well?

Armen: Wait a bit before we go to production. We might a slight update.

Flags: needinfo?(choller)

(In reply to Bob Clary [:bc:] from comment #45)

decoder: While looking at jmaher's review comment, I realized I'm missing spidermonkey-sm-fuzzing-linux64/opt. I should include that as well?

Armen: Wait a bit before we go to production. We might a slight update.

Yes, we can include that too.

Flags: needinfo?(choller)

bc: I included the last PR and I did a deployment. I can do more deployments as you need

Flags: needinfo?(armenzg)
Pushed by bclary@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9c90cbf62eee
SETA: Treat fuzzing build tasks as possible low value tasks, r=jmaher,decoder
      "build-android-x86_64-asan-fuzzing/opt",
      "build-linux64-asan-fuzzing/opt",
      "build-linux64-fuzzing-ccov/opt",
      "build-linux64-fuzzing/debug",
      "build-macosx64-asan-fuzzing/opt",
      "build-macosx64-fuzzing/debug",
      "build-win64-asan-fuzzing/opt",
      "build-win64-fuzzing/debug",
      "spidermonkey-sm-fuzzing-linux64/opt",

are still listed as priority 1 tasks. Armen, Camd: Will it take two weeks for the normal processing for these to be listed a priority 5 (low value) tasks? If so, could you use preseed.json to force them to priority 5 low value?

Flags: needinfo?(cdawson)
Flags: needinfo?(armenzg)

If we want to have a fixed priority (rather than a fluctuating one) then a PR with the values in preseed.json will guarantee that.
I can review the PR.

Flags: needinfo?(cdawson)
Flags: needinfo?(armenzg)

I don't think we want it fixed at priority 5 for all time. Could we update the DB as outlined in seta maintenance. jmaher: What do you think?

Flags: needinfo?(jmaher)

updating the db will not help because now that we are looking at builds it will find that there are regressions the builds will find and we are only considering removing fuzzing (vs all other build types), so in order to catch all the regressions found by builds, we cannot remove fuzzing builds. In truth if we were willing to put all builds up, then some if not all of the fuzzing builds would be optimized out.

We did agree that we wanted the fuzzing builds to be run every 10th push, to me that sounds like a forced priority=5.

Flags: needinfo?(jmaher)
Attached file preseed.json

Here is what I would put in the PR. Let me know if this is ok and I'll create the PR.

I'm a little wary about setting some of the platforms since it seems possible to have a mismatch in case things change. For example, I set platform to * for build-android-x86_64-asan-fuzzing/opt, build-macosx64-asan-fuzzing/opt, build-macosx64-fuzzing/debug. The windows2012-64 platforms seem like good candidates for a * platform as well.

Armen: What do you think?

Flags: needinfo?(armenzg)

a quick glance looks good from my perspective

It seems alright to me. Once the PR is up I can help test it.

Flags: needinfo?(armenzg)

Part 3 is now live.
Ping me if it does not show up updated.

Attachment #9125137 - Attachment is obsolete: true
Attachment #9125177 - Attachment is obsolete: true
Attachment #9126216 - Attachment is obsolete: true
Attachment #9126767 - Attachment is obsolete: true

Armen: Thank you!

I'm going to call this bug resolved. If we need more work to be done, let's do it in a new bug.

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED

looking at the tip of autoland, I see the full set of fuzzing builds running. Possibly there are more tweaks to make in the taskcluster/.../seta.py or related in-tree code?

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Armen says to look at the decision task. On it next.

Probably the caching https://github.com/mozilla/treeherder/blob/master/treeherder/seta/settings.py#L81
I'll check back in an hour or so.

The failure to prevent fuzzing builds from being created for every push is not related to caching. I have been trying to debug this on try using logging statements but the code in seta.py is never called and yes I have modifed SETA_PROJECTS to include try.

ahal you have been most active in this area and tomprince you have been the main reviewer. Can you tell me what I have done wrong or if what we want to do is even possible?

Flags: needinfo?(mozilla)
Flags: needinfo?(ahal)
Whiteboard: [ci-costs-2020:todo]

I wonder if SETA optimizations can be run locally at |./mach try fuzzy| stage? if so, then it would be easier to debug.

./mach taskgraph optimized helps a bit especially when run in the debugger.

I think the build optimization needs to be changed

I have a couple of other changes I'd like to make.

Note we store the low value tests as a list then look them up using the list which is O(N^2)

I can see the seta output when running my patches locally though they mark the fuzz builds as not low value.

ahal, tomprince: thoughts?

Depends on: 1618622

(In reply to Bob Clary [:bc:] from comment #72)

I think the build optimization needs to be changed

I think the change should be more targeted. From skimming the bug, it looks like the intent is to change the behavior only for fuzzing builds. Given that, optimization should only be changed for those tasks. I think probably inclined to have a new optional value in the build job definition, that the build_attr transform can look at (rather than just setting optimization on those tasks, to handle the logic for finding the schedules to pass to the optimization).

Additionally, it should not use the test optimization strategy alias, but a new one (or possibly two, one for build and one for fuzzing). The alias are designed to (among other things) allow experimenting with strategies, and decoupling the transforms from the specifics of the optimization chosen. Having tasks with different requirements using the same alias undoes some of that decoupling.

Flags: needinfo?(mozilla)

(In reply to Tom Prince [:tomprince] from comment #76)

I partially addressed this in https://phabricator.services.mozilla.com/D64628 but did not change the job definition.

Flags: needinfo?(ahal)
Pushed by bclary@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/968394a5f161
remove mozilla-inbound from SETA_PROJECTS, r=tomprince.
https://hg.mozilla.org/integration/autoland/rev/dcb423f0b0e6
Eliminate O(N^2) behavior with lists and sets in SETA.query_low_value_tasks, r=ahal.
https://hg.mozilla.org/integration/autoland/rev/80962be24a6c
Simplify SkipLowValue.should_remove_task, r=tomprince.

I'm still seeing fuzzing builds on autoland. I'm at a loss.

Keywords: leave-open

We're eliminating 3 jobs. I don't know why the others are still running. jmaher: Any thoughts from Mount Olympus?

Flags: needinfo?(jmaher)

one step at a time.

We have asan/tsan fuzzing builds still running, and android 4.2 debug.

Looking at SETA, asan/tsan are marked as P1:
https://treeherder.mozilla.org/api/project/autoland/seta/job-priorities/?buildsystem=taskcluster&priority=1&format=json

and looking at the priority=5 version, I see a many fuzzing jobs as P5, but android 4.2 debug is not listed there.

Flags: needinfo?(jmaher)

job_priority=1 has asan/tsan:

        "build-linux64-asan-fuzzing/opt",
        "build-linux64-tsan-fuzzing/opt",
        "build-macosx64-asan-fuzzing/opt",
        "build-win64-asan-fuzzing/opt",

preseed.json has

$ egrep 'build-linux64-asan-fuzzing/opt|build-linux64-tsan-fuzzing/opt|build-macosx64-asan-fuzzing/opt|build-win64-asan-fuzzing/opt'  $(find . -name preseed.json)
    "testtype": "build-linux64-asan-fuzzing/opt",
    "testtype": "build-macosx64-asan-fuzzing/opt",
    "testtype": "build-win64-asan-fuzzing/opt",

So we're missing build-linux64-tsan-fuzzing/opt and build-android-x86-fuzzing/debug

decoder: Ok?

Flags: needinfo?(choller)

(In reply to Bob Clary [:bc:] from comment #83)

job_priority=1 has asan/tsan:

        "build-linux64-asan-fuzzing/opt",
        "build-linux64-tsan-fuzzing/opt",
        "build-macosx64-asan-fuzzing/opt",
        "build-win64-asan-fuzzing/opt",

preseed.json has

$ egrep 'build-linux64-asan-fuzzing/opt|build-linux64-tsan-fuzzing/opt|build-macosx64-asan-fuzzing/opt|build-win64-asan-fuzzing/opt'  $(find . -name preseed.json)
    "testtype": "build-linux64-asan-fuzzing/opt",
    "testtype": "build-macosx64-asan-fuzzing/opt",
    "testtype": "build-win64-asan-fuzzing/opt",

So we're missing build-linux64-tsan-fuzzing/opt and build-android-x86-fuzzing/debug

decoder: Ok?

lgtm :)

Flags: needinfo?(choller)
WARNING [treeherder.seta.job_priorities:45] 
Job priority (taskcluster,build-android-x86_64-asan-fuzzing/opt,asan,android-5-0-x86_64) not found in accepted jobs list
Job priority (taskcluster,build-linux64-asan-fuzzing/opt,asan,linux64) not found in accepted jobs list
Job priority (taskcluster,build-macosx64-asan-fuzzing/opt,asan,osx-cross) not found in accepted jobs list
Job priority (taskcluster,build-win64-asan-fuzzing/opt,asan,windows2012-64) not found in accepted jobs list
Job priority (taskcluster,build-android-x86-fuzzing/debug,debug,android-4-2-x86) not found in accepted jobs list

The latest attempt has restricted the fuzzing builds. Linux 64 tsan is the only remaining build running on every push.

is tsan applicable?

It wasn't in our list to be low valued in preseed so it is expected that it is continuing to run. decoder might know if it is something that needs to run per commit. Easy to add now if we want to do so.

I would like the regular tsan build to run per push until we have data how often it is actually regressed. We've only been running this for a short while now.

Sounds good. And.... we're done here.

Status: REOPENED → RESOLVED
Closed: 1 year ago1 year ago
Resolution: --- → FIXED
Regressions: 1621472

Reopening since the PR 6118 has been reverted and deployed. Until I know how to test this properly I am going to put it on hold.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

From looking at this last night it is possible that most of the over scheduling was due to what was removed in seta.py in tree, not what changes were made to treeherder. When you have a fresh mind, lets get back to this.

I looked over the logs and runnable-jobs from the decision tasks before and after the change yesterday and didn't see anything outside of the android raptor jobs that stood out. We'll get the android raptor low value hack back into autoland then attempt to do the Treeherder PR again.

Pushed by bclary@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/abae2f3bd8eb
revert changes in Bug 1618953 that removed logic which forced android raptor tests to be low value, r=jmaher.
Status: REOPENED → RESOLVED
Closed: 1 year ago1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla76
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: mozilla76 → ---

I think this is going to stick. Resolving again. Thank you to everyone involved! It was quite a journey!

Status: REOPENED → RESOLVED
Closed: 1 year ago1 year ago
Resolution: --- → FIXED
Whiteboard: [ci-costs-2020:todo] → [ci-costs-2020:done]
You need to log in before you can comment on or make changes to this bug.