Closed Bug 1352477 Opened 7 years ago Closed 7 years ago

taskgraph ignores run-on-project for fennec-nightlies

Categories

(Release Engineering :: Release Automation: Other, defect, P1)

Product:
Release Engineering
Component:
Release Automation: Other
Type:
defect

Tracking

(firefox55 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox55 --- fixed

People

(Reporter: jlorenzo, Assigned: jlorenzo)

References

Details

Attachments

(1 file)

Bug 1352477 - taskgraph ignores run-on-project for fennec-nightlies
59 bytes, text/x-review-board-request
mozilla
: review+
Details 
Bug 1317783 showed that even though push-apk specifies "run-on-project" to exclude mozilla-central, it does get included in the central nightly graph.
Comment on attachment 8853461 [details]
Bug 1352477 - taskgraph ignores run-on-project for fennec-nightlies

Clearing review flag for now... I provided comments in IRC.
Attachment #8853461 - Flags: review?(aki)
Blocks: 1317783
No longer depends on: 1317783
Priority: -- → P1
This patch makes sure there is no new job created/deleted, with the exception of jamun, where:

> build-linux64-nightly/opt-upload-symbols
> build-linux-nightly/opt-upload-symbols

are now part of the regular jamun decision task. I didn't added them on purpose, initially. However as jamun should be par with beta, I believe we can keep them.

TESTING PROTOCOL

1. Download all the parameters.yml from either treeherder (like [1]), or from the hooks[2]. In my case, I named them parameters-branch-nightly-platform.yml
2. Before the patch is applied, run: 
> for name in $(find . -name 'parameters-*.yml' | cut -d'-' -f'2-' | cut -d'.' -f1); do echo "$name" ; ./mach taskgraph target -p parameters-$name.yml > $name-before.list ; done
This generates the task lists for every parameters file. You may need to tweak the files from beta/release/jamun because they are outdated. You may also need:
> export BUILD_NUMBER=1

3. Apply the patch
4. Run:
> for name in $(find . -name 'parameters-*.yml' | cut -d'-' -f'2-' | cut -d'.' -f1); do echo "$name" ; ./mach taskgraph target -p parameters-$name.yml > $name-after.list ; done
5. Concatenate the differences between before and after:
> rm -f concatenated.diff && for name in $(find . -name 'parameters-*.yml' | cut -d'-' -f'2-' | cut -d'.' -f1); do echo $name >> meta.diff && diff <(sort $name-before.list) <(sort $name-after.list) >> concatenated.diff ; done
6. Read concatenated.diff.

In my case, I have 16 different parameters files, which are:
* aurora
* aurora-nightly-desktop
* aurora-nightly-fennec
* beta
* beta-candidate-fennec
* central
* central-nightly-desktop
* central-nightly-fennec
* date
* date-nightly-desktop
* date-nightly-fennec
* inbound
* jamun
* jamun-candidate-fennec
* release
* try

Like said at the beginning, the only difference spotted was in jamun. All the other remained intact. 

As a follow up, we need to hardcode the list of expected tasks in an integration test. I haven't looked into it yet.

[1] https://treeherder.mozilla.org/#/jobs?repo=date
[2] https://tools.taskcluster.net/hooks/#project-releng
(In reply to Johan Lorenzo [:jlorenzo] from comment #4)
I also tested this patch on top of bug 1317783, where I could see the push-apk tasks (and only these ones) being filtered out of date-nightly, jamun-candidate, and more importantly: central-nightly.
Comment on attachment 8853461 [details]
Bug 1352477 - taskgraph ignores run-on-project for fennec-nightlies

https://reviewboard.mozilla.org/r/125554/#review128638

r+ with nits.

::: taskcluster/ci/nightly-l10n/kind.yml:36
(Diff revision 2)
> +   run-on-projects:
> +     - mozilla-central
> +     - mozilla-aurora
> +     - mozilla-beta
> +     - date
> +     - jamun

As per an earlier review, I think it's cleaner having the real set of branches land, and then update date and jamun to add them.  (It may be messy when we land and back out patches, but I think it'll be cleaner overall.)

If we plan on having a post-migration sweep to clean up all the jamun and date references once we're done with them, that works.

::: taskcluster/ci/upload-symbols/job-template.yml:10
(Diff revision 2)
>  deadline-after: 24 hours
>  run-on-projects:
>      - try
> -    - mozilla-beta
> -    - mozilla-release
> +    - release
> +    - date
> +    - jamun

same here.

::: taskcluster/taskgraph/target_tasks.py:36
(Diff revision 2)
> -        return False
>      run_on_projects = set(task.attributes.get('run_on_projects', []))
>      return match_run_on_projects(parameters['project'], run_on_projects)
>  
>  
> +def filter_upload_symbols(task, parameters):

We really should add the 'nightly' parameter to these rather than have this function.  Maybe as a followup?
Attachment #8853461 - Flags: review?(aki) → review+
Blocks: 1353296
Comment on attachment 8853461 [details]
Bug 1352477 - taskgraph ignores run-on-project for fennec-nightlies

https://reviewboard.mozilla.org/r/125554/#review128638

> We really should add the 'nightly' parameter to these rather than have this function.  Maybe as a followup?

I filed bug 1353296
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5533bb014569
taskgraph ignores run-on-project for fennec-nightlies r=aki
https://hg.mozilla.org/mozilla-central/rev/5533bb014569
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
I had to uplift this patch to aurora, in order to make bug 1357409 happen: https://hg.mozilla.org/releases/mozilla-aurora/rev/4dae386f2c331851b192a3ef493cc139fd562a44
Blocks: 1357409
See Also: → 1386995
Blocks: 1386995
See Also: 1386995 →
No longer blocks: 1386995
Depends on: 1386995
See Also: → 1386995
No longer blocks: 1428444
You need to log in before you can comment on or make changes to this bug.