1401604 - Hook for Heavy Profile

Status: RESOLVED FIXED

(Taskcluster :: Services, enhancement)

Description

[Tarek Ziadé (:tarek)]

I am working on a "heavy profile" generator for talos.

The first version is functional, and located at https://github.com/tarekziade/heavy-profile triggered by the taskcluster github bot and is doing the expected work. It spits out public artifacts I can grab from the TC indexed artifacts.

What I need now is a hook with the right permissions, I've created a first version at https://tools.taskcluster.net/hooks/garbage/heavyprofile

it requires more permissions so I guess I need a new project in the hooks service to add it.

Note that the cron is set to run every day at 01:00 UTC but my real goal is to have it running sometimes after firefox nightly was built and made available on our ftp servers. A v2 will have a cleaner design, e.g. added in a graph so it's all automated, but for v1 this is good enough for us to start playing with the generated profiles.

Thanks a lot!

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

@tarek,

if the long term goal is to add it in-tree and have it in the task-graph. Then maybe we should keep v1 and quick and dirty
as possible :)

One option is to just use the garbage hook and set:
  provisionerId: aws-provisioner-v1
  workerType: tutorial

if you look at the poorly formatted error code you'll see that it says the hook has scope:
  queue:create-task:aws-provisioner-v1/tutorial
So we could just use that.

Obviously, garbage hooks aren't good for anything long-term; as they can be modified by anyone in team_moco.
But I don't think it's particularly hard to add the task to the task-graph in-tree.
I bet you can test it wit with "mach try fuzzy --full" too...

I suspect one might call it a build and add a heavy-profile.yml to: taskcluster/ci/build/
Probably something like:
  heavy-profile:  # TRY: |mach try fuzzy| and this label will hopefully show up :)
    description: "...heavy-profile.."
    attributes:
        nightly: true  # only run on nightly
    index:
        product: firefox
        job-name: heavy-profile
        type: nightly
    treeherder:
        platform: heavy-profile
        symbol: tc(HP)
    worker-type: aws-provisioner-v1/gecko-{level}-b-linux
    worker:
        max-run-time: 36000
        docker-image: {in-tree: heavy-profile}  # TODO: add taskcluster/docker/heavy-profile/Dockerfile
    run:
        using: mach
        mach: 'something heavy-profile'  # TODO: add |mach something heavy-profile|

I could be very wrong about this ^ approach... Maybe it needs a dedicated kind, maybe it doesn't need
it's own docker image but can reuse an existing in-tree docker image. Or maybe you just reference an
external docker image like: tarekziade/heavyprofile@sha256:<hash> so that the image hash is locked in-tree.

Certainly, we should look at the task definition before we run this, to check out what the index namespaces
generated looks like. Some of the |mach taskgraph| commands support outputting all tasks, then we can easily see
if it's sane (we really don't want to pollute the index too much).

Comment 2

[Tarek Ziadé (:tarek)]

Thanks for your feedback, I'll mark this bug as fixe since the garbage hook can index in index.garbage now - I can revisit once we want to move away from the garbage area