1294200 - Clobberer doesn't work at all for taskcluster builds

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

If I load https://api.pub.build.mozilla.org/clobberer/ the "Taskcluster" section takes forever to load, and when it does, only mozilla-release shows any worker types.

The request that takes forever is to https://api.pub.build.mozilla.org/clobberer/tc/branches and the response is:
{
  "request_id": "415afd4e-a410-4e34-9332-c47030aee2c2",
  "result": [
    {
      "name": "jamun",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "b2g-ota",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "mozilla-release",
      "provisionerId": "aws-provisioner-v1",
      "workerTypes": {
        "android-api-15": {
          "caches": [
            "level-3-mozilla-release-tc-vcs",
            "level-3-mozilla-release-build-android-api-15-frontend-workspace"
          ],
          "name": "android-api-15"
        },
        "b2gtest": {
          "caches": [
            "level-3-mozilla-release-tc-vcs"
          ],
          "name": "b2gtest"
        },
        "dbg-linux32": {
          "caches": [
            "level-3-mozilla-release-tc-vcs",
            "level-3-mozilla-release-build-linux32-workspace"
          ],
          "name": "dbg-linux32"
        },
        "dbg-linux64": {
          "caches": [
            "level-3-mozilla-release-tc-vcs",
            "level-3-mozilla-release-build-linux64-workspace"
          ],
          "name": "dbg-linux64"
        },
        "dbg-macosx64": {
          "caches": [
            "level-3-mozilla-release-build-macosx64-workspace",
            "level-3-mozilla-release-tc-vcs"
          ],
          "name": "dbg-macosx64"
        },
        "desktop-test": {
          "caches": [
            "level-3-mozilla-release-test-workspace"
          ],
          "name": "desktop-test"
        },
        "desktop-test-xlarge": {
          "caches": [
            "level-3-mozilla-release-test-workspace"
          ],
          "name": "desktop-test-xlarge"
        },
        "opt-linux32": {
          "caches": [
            "level-3-mozilla-release-tc-vcs",
            "level-3-mozilla-release-build-linux32-workspace"
          ],
          "name": "opt-linux32"
        },
        "opt-linux64": {
          "caches": [
            "level-3-mozilla-release-tc-vcs",
            "level-3-mozilla-release-build-linux64-artifact-workspace",
            "level-3-mozilla-release-build-linux64-workspace",
            "level-3-mozilla-release-build-linux64-st-an-workspace"
          ],
          "name": "opt-linux64"
        },
        "opt-macosx64": {
          "caches": [
            "level-3-mozilla-release-build-macosx64-workspace",
            "level-3-mozilla-release-tc-vcs",
            "level-3-mozilla-release-build-macosx64-st-an-workspace"
          ],
          "name": "opt-macosx64"
        }
      }
    },
    {
      "name": "oak",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "hey",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "pine",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "mozilla-central",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "mozilla-inbound",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "mozilla-b2g44_v2_5",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "mozilla-beta",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "alder",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "mozilla-aurora",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "b2g-inbound",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "cedar",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "mozilla-esr45",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "date",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "ash",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "autoland",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "elm",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "try",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "larch",
      "provisionerId": null,
      "workerTypes": {}
    },
    {
      "name": "fx-team",
      "provisionerId": null,
      "workerTypes": {}
    }
  ]
}

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(Not sure if this belongs on the Taskcluster side or the RelengAPI side.)

Comment 2

[Greg Arndt [:garndt]]

I believe this should be moved under a releng related component for managing the clobberer tool.  I'm not sure what that component is though.

Comment 3

[Alin Selagea [:aselagea]]

I haven't touched Clobberrer in the past, so maybe it would be faster if someone else could take a look/come up with a suggestion.

Comment 4

[Rok Garbas [:garbas]]

:KWierso hey! i took over clobberer tool. new clobberer tool version will be released in ~week time and should address that slow initial loading of branches.

about the correctness of it i will look at this tomorrow and report back. i might need some help what the correct result would be since i'm new to all taskcluster topic.

Comment 5

[Rok Garbas [:garbas]]

I found out the problem. it looks like the location&structure of decision task artifact `public/graph.json` changed and is now at located at `public/task-graph.json`.

i will prepare PR to fix the above issue, but for performance issue you will need to wait a week more to switch clobberer to new setup.

Comment 6

[Intermittent Failures Robot]

33 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 33

Platform breakdown:
* linux64: 33

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1294200&startday=2016-08-26&endday=2016-08-26&tree=all

Comment 7

[Intermittent Failures Robot]

33 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 33

Platform breakdown:
* linux64: 33

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1294200&startday=2016-08-22&endday=2016-08-28&tree=all

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

New clobberer is in place for almost a year if I recally correctly and I haven't seen any new oranges tagged with this bug so I think it's safe to close this for now.  Feel free to reopen if I missed anything.

Comment 9

[Phil Ringnalda (:philor)]

This wasn't an intermittent-failure bug, the starring was just when someone got annoyed by a needs-clobber failure that they could fix for the bulk of the jobs, the buildbot jobs, but couldn't fix for the rare taskcluster jobs.

Clobberer doesn't work at all for taskcluster builds, which is pretty much everything outside of esr52. That gives you a couple choices: leave this bug open and ignored until esr52 and buildbot builds are gone, and then file "Decomm clobberer" and "Remove references to clobberer" (probably that's just a mozharness bug, I don't think taskcluster pays the slightest attention to its existence, though I could be wrong), or, WONTFIX this and file those bugs as "... once esr52 and buildbot builds die".

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Thanks for the elaborated answer :philor. 
I'd go with the latter, let's WONTFIX this and deal with it once esr52 and BB are history!
I'll file a bug to reference in the meantime.

Thanks.