taskgraph ignores run-on-project for fennec-nightlies

RESOLVED FIXED

Status

defect
P1
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: jlorenzo, Assigned: jlorenzo)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox55 fixed)

Details

Attachments

(1 attachment)

Bug 1317783 showed that even though push-apk specifies "run-on-project" to exclude mozilla-central, it does get included in the central nightly graph.
Comment on attachment 8853461 [details]
Bug 1352477 - taskgraph ignores run-on-project for fennec-nightlies

Clearing review flag for now... I provided comments in IRC.
Attachment #8853461 - Flags: review?(aki)
Blocks: 1317783
No longer depends on: 1317783
Priority: -- → P1
This patch makes sure there is no new job created/deleted, with the exception of jamun, where:

> build-linux64-nightly/opt-upload-symbols
> build-linux-nightly/opt-upload-symbols

are now part of the regular jamun decision task. I didn't added them on purpose, initially. However as jamun should be par with beta, I believe we can keep them.

TESTING PROTOCOL

1. Download all the parameters.yml from either treeherder (like [1]), or from the hooks[2]. In my case, I named them parameters-branch-nightly-platform.yml
2. Before the patch is applied, run: 
> for name in $(find . -name 'parameters-*.yml' | cut -d'-' -f'2-' | cut -d'.' -f1); do echo "$name" ; ./mach taskgraph target -p parameters-$name.yml > $name-before.list ; done
This generates the task lists for every parameters file. You may need to tweak the files from beta/release/jamun because they are outdated. You may also need:
> export BUILD_NUMBER=1

3. Apply the patch
4. Run:
> for name in $(find . -name 'parameters-*.yml' | cut -d'-' -f'2-' | cut -d'.' -f1); do echo "$name" ; ./mach taskgraph target -p parameters-$name.yml > $name-after.list ; done
5. Concatenate the differences between before and after:
> rm -f concatenated.diff && for name in $(find . -name 'parameters-*.yml' | cut -d'-' -f'2-' | cut -d'.' -f1); do echo $name >> meta.diff && diff <(sort $name-before.list) <(sort $name-after.list) >> concatenated.diff ; done
6. Read concatenated.diff.

In my case, I have 16 different parameters files, which are:
* aurora
* aurora-nightly-desktop
* aurora-nightly-fennec
* beta
* beta-candidate-fennec
* central
* central-nightly-desktop
* central-nightly-fennec
* date
* date-nightly-desktop
* date-nightly-fennec
* inbound
* jamun
* jamun-candidate-fennec
* release
* try

Like said at the beginning, the only difference spotted was in jamun. All the other remained intact. 

As a follow up, we need to hardcode the list of expected tasks in an integration test. I haven't looked into it yet.

[1] https://treeherder.mozilla.org/#/jobs?repo=date
[2] https://tools.taskcluster.net/hooks/#project-releng
(In reply to Johan Lorenzo [:jlorenzo] from comment #4)
I also tested this patch on top of bug 1317783, where I could see the push-apk tasks (and only these ones) being filtered out of date-nightly, jamun-candidate, and more importantly: central-nightly.
Comment on attachment 8853461 [details]
Bug 1352477 - taskgraph ignores run-on-project for fennec-nightlies

https://reviewboard.mozilla.org/r/125554/#review128638

r+ with nits.

::: taskcluster/ci/nightly-l10n/kind.yml:36
(Diff revision 2)
> +   run-on-projects:
> +     - mozilla-central
> +     - mozilla-aurora
> +     - mozilla-beta
> +     - date
> +     - jamun

As per an earlier review, I think it's cleaner having the real set of branches land, and then update date and jamun to add them.  (It may be messy when we land and back out patches, but I think it'll be cleaner overall.)

If we plan on having a post-migration sweep to clean up all the jamun and date references once we're done with them, that works.

::: taskcluster/ci/upload-symbols/job-template.yml:10
(Diff revision 2)
>  deadline-after: 24 hours
>  run-on-projects:
>      - try
> -    - mozilla-beta
> -    - mozilla-release
> +    - release
> +    - date
> +    - jamun

same here.

::: taskcluster/taskgraph/target_tasks.py:36
(Diff revision 2)
> -        return False
>      run_on_projects = set(task.attributes.get('run_on_projects', []))
>      return match_run_on_projects(parameters['project'], run_on_projects)
>  
>  
> +def filter_upload_symbols(task, parameters):

We really should add the 'nightly' parameter to these rather than have this function.  Maybe as a followup?
Attachment #8853461 - Flags: review?(aki) → review+
Comment on attachment 8853461 [details]
Bug 1352477 - taskgraph ignores run-on-project for fennec-nightlies

https://reviewboard.mozilla.org/r/125554/#review128638

> We really should add the 'nightly' parameter to these rather than have this function.  Maybe as a followup?

I filed bug 1353296
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5533bb014569
taskgraph ignores run-on-project for fennec-nightlies r=aki
https://hg.mozilla.org/mozilla-central/rev/5533bb014569
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
No longer blocks: 1386995
Depends on: 1386995
You need to log in before you can comment on or make changes to this bug.