Closed Bug 1407299 Opened 7 years ago Closed 7 years ago

fennec mozilla-beta version bump in-tree task

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: garbas, Assigned: garbas)

Details

User Story

just for my record here is what i did so far:

1. find a a fennec graph to have something to compare to

- select `task group 1` from `Build 1` from `fennec-beta-57.0b5.md`[1] release notes
- select `fennec mozilla-beta version bump` task
- find `Task Definition` in `Task Details` and download the json
- in my case this is::

  $ curl https://queue.taskcluster.net/v1/task/NZbWWhuhSaK1IyIGzPI6Zw > current_task.json

2. get a maple working copy our of your mozilla-unified

  $ cd path/to/mozilla-unified
  $ hg share ../mozilla-maple
  $ cd ../mozilla-maple
  $ hg pull maple
  $ hg up maple

3. use nixpkgs-mozilla to get development tools for gecko

  $ git clone git://github.com/mozilla/nixpkgs-mozilla
  $ cd path/to/mozilla-maple
  $ nix-shell path/to/nixpkgs-mozilla/release.nix -A gecko.x86_64-linux.gcc
  (nix-shell) $ ./mach taskgraph --help
  ...

4. create `publish_fennec.yml` with the following content

```
base_repository: https://hg.mozilla.org/mozilla-central
build_date: 1507178755
do_not_optimize: []
existing_tasks: {}
filters:
- check_servo
- target_tasks_method
head_ref: a76bd64bcdd3b360918936a3cfbc5e3e604b1d1c
head_repository: https://hg.mozilla.org/projects/maple
head_rev: a76bd64bcdd3b360918936a3cfbc5e3e604b1d1c
include_nightly: true
level: '3'
message: ' '
moz_build_date: '20171005044555'
optimize_target_tasks: true
owner: asasaki@mozilla.com
project: maple
pushdate: 1507178755
pushlog_id: '13'
release_history: {}
target_tasks_method: publish_fennec
try_mode: null
try_options: null
try_task_config: null
``` 

Above content was profided by :aki.

5. Created folder `taskcluster/ci/release-version-bump` with `kind.yml` in it. Becuase this is going to be a buildbot task i copied what :aki did `taskcluster/ci/release-bouncer-sub/kind.yml`.

6. Generating the taskcluster task using taskgraph

$ cd path/to/mozilla-maple
$ ./mach taskgraph target-graph  -p path/to/publish_fennec.yml --json | jq '.["release-version-bump-fennec"] | .task' > new_task.json


7. Cleaning up not needed data from new_task.json and current_task.json

$ cat new_task.json | jq -S 'del(.created,.deadline,.expires,.priority)' > new_task_clean.json

$ cat current_task.json | jq -S 'del(.created,.deadline,.expires,.priority)' > current_task_clean.json

8. Comparing new_task_clean.json and current_task_clean.json

$ diff current_task_clean.json new_task_clean.json

9. change/edit/update `taskcluster/ci/release-version-bump/kind.yml` and `taskcluster/taskgraph/target_tasks.py` then rerun 6., 7. and 8.



[1] https://github.com/mozilla/releasewarrior/blob/master/releases/fennec-beta-57.0b5.md

Attachments

(1 file, 3 obsolete files)

      No description provided.
Assignee: nobody → rgarbas
Rok:

I landed my latest changes on maple for bouncer-sub.

- we no longer have to specify BUILD_NUMBER=1 on the commandline. It'll default to 1; we need to specify BUILD_NUMBER if we want something other than 1.
- the product is changing from 'mobile' to 'fennec'. I adjusted the candidates_fennec target_tasks_method.
- we now have release_promotion: true and routes in the kind.yml: https://hg.mozilla.org/projects/maple/rev/632d3fbe9697#l1.23
- we auto-populate scopes https://hg.mozilla.org/projects/maple/rev/632d3fbe9697#l3.54

I'm also merging in m-c into maple so we don't diverge too far from central. If you're working on a bookmark, you can

    hg rebase -b BOOKMARK -d DESTINATION

where DESTINATION is probably `maple`. If you're looking at `git rebase -i` functionality on a bookmark, you probably want `hg histedit`. https://hg.mozilla.org/projects/maple/pushloghtml may be useful.
User Story: (updated)
User Story: (updated)
adding `.priority` to the list of things to ignore since its is set based on branch it is running on[1]

[1] https://hg.mozilla.org/projects/maple/file/632d3fbe9697/taskcluster/taskgraph/transforms/task.py#l1191
User Story: (updated)
Attached patch version_bump_v1.patch (obsolete) — Splinter Review
:rail let me know what do you think? is adding next_version parameter the way to go? or is there better way?
Attachment #8918217 - Flags: review?(rail)
Comment on attachment 8918217 [details] [diff] [review]
version_bump_v1.patch

Review of attachment 8918217 [details] [diff] [review]:
-----------------------------------------------------------------

I think this should just work. Probably you want to rebase your work against maple and push it there.
Attachment #8918217 - Flags: review?(rail) → feedback+
Attached patch fennec-version-bump-task.patch (obsolete) — Splinter Review
:rail looks like I don't have level_2 as it is needed to push to maple repo[1].

I rebase and exported the commit and asked for another review by you.

As I'm not 100% sure what I'm doing (learning and I'm getting there :P) I'd ask you to double check that everything is correct. As I'm trying to understand how everything works I'll write down what I checked. If there is something else I should check for please write in reply

I checked that:

1. version bump task is not included in candidates-fennec target or when .

```
./mach taskgraph target-graph -p promote_fennec.yml --json | grep version-bump
```
Question1: is it ok that next_version parameter is required by candidate-fennec target even if it is not used by any tasks?

2. that task generated matched what is already in taskcluster (ignoring .created,.deadline,.expires,.priority,.extra,.payload.sourcestamp,.requires,.retries,.schedulerId fields)

```
$ rm -f maple-publish-fennec-dirty.json && ./mach taskgraph target-graph -p publish-fennec.yml --json > maple-publish-fennec-dirty.json && cat maple-publish-fennec-dirty.json | jq -S '."release-version-bump-fennec".task | del(.created,.deadline,.expires,.priority,.extra,.payload.sourcestamp,.requires,.retries,.schedulerId)' | less
```

and then comparing the output with 

```
$ curl https://queue.taskcluster.net/v1/task/NZbWWhuhSaK1IyIGzPI6Zw | jq -S 'del(.created,.deadline,.expires,.priority,.extra,.payload.sourcestamp,.requires,.retries,.schedulerId)' | less  
```


[1] https://wiki.mozilla.org/ReleaseEngineering/DisposableProjectRepositories#What_is_a_disposable_project_branch.3F
Attachment #8918217 - Attachment is obsolete: true
Attachment #8918822 - Flags: review?(rail)
Comment on attachment 8918822 [details] [diff] [review]
fennec-version-bump-task.patch

(In reply to Rok Garbas [:garbas] from comment #5)
> :rail looks like I don't have level_2 as it is needed to push to maple
> repo[1].

I'll push it for you. Meanwhile we should file a bug to get you at least Level 2 access.
 

> Question1: is it ok that next_version parameter is required by
> candidate-fennec target even if it is not used by any tasks?

I think this is OK. At least it's not a regression (we pre-set this value in the current implementation). We can probably optimize the schema to be per target task method.

> 2. that task generated matched what is already in taskcluster (ignoring
> .created,.deadline,.expires,.priority,.extra,.payload.sourcestamp,.requires,.
> retries,.schedulerId fields)

If taks.payload is the same, we should be ok

I'll land this a bit. Can you update the google doc with the corresponding commit please.
Attachment #8918822 - Flags: review?(rail) → review+
bah, had to backout the push in https://hg.mozilla.org/projects/maple/rev/5cda86ee0df2cd410ed8c77817db7cce381825a2 because it broke the decision task:
https://tools.taskcluster.net/groups/IyuAbbPvTG6gTz91_RwYng/tasks/IyuAbbPvTG6gTz91_RwYng/runs/0/logs/public%2Flogs%2Flive.log

[task 2017-10-16T13:46:43.035273Z] Generating tasks for release-bouncer-aliases fennec
[task 2017-10-16T13:46:43.036142Z] Traceback (most recent call last):
[task 2017-10-16T13:46:43.036250Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 165, in taskgraph_decision
[task 2017-10-16T13:46:43.036328Z]     return taskgraph.decision.taskgraph_decision(options)
[task 2017-10-16T13:46:43.036422Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 119, in taskgraph_decision
[task 2017-10-16T13:46:43.036677Z]     full_task_json = tgg.full_task_graph.to_json()
[task 2017-10-16T13:46:43.036960Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 130, in full_task_graph
[task 2017-10-16T13:46:43.037217Z]     return self._run_until('full_task_graph')
[task 2017-10-16T13:46:43.037493Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 287, in _run_until
[task 2017-10-16T13:46:43.037562Z]     k, v = self._run.next()
[task 2017-10-16T13:46:43.037656Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 217, in _run
[task 2017-10-16T13:46:43.037710Z]     new_tasks = kind.load_tasks(self.parameters, list(all_tasks.values()))
[task 2017-10-16T13:46:43.038187Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 65, in load_tasks
[task 2017-10-16T13:46:43.038239Z]     for task_dict in transforms(trans_config, inputs)]
[task 2017-10-16T13:46:43.038307Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1303, in check_run_task_caches
[task 2017-10-16T13:46:43.038347Z]     for task in tasks:
[task 2017-10-16T13:46:43.038406Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1142, in build_task
[task 2017-10-16T13:46:43.038438Z]     for task in tasks:
[task 2017-10-16T13:46:43.038494Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1112, in add_index_routes
[task 2017-10-16T13:46:43.038524Z]     for task in tasks:
[task 2017-10-16T13:46:43.038578Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1002, in validate
[task 2017-10-16T13:46:43.038767Z]     for task in tasks:
[task 2017-10-16T13:46:43.038845Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/__init__.py", line 165, in make_task_description
[task 2017-10-16T13:46:43.038960Z]     configure_taskdesc_for_run(config, job, taskdesc, impl)
[task 2017-10-16T13:46:43.039199Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/__init__.py", line 217, in configure_taskdesc_for_run
[task 2017-10-16T13:46:43.039242Z]     func(config, job, taskdesc)
[task 2017-10-16T13:46:43.039454Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/buildbot.py", line 110, in mozharness_on_buildbot_bridge
[task 2017-10-16T13:46:43.039526Z]     bb_release_worker(config, worker, run)
[task 2017-10-16T13:46:43.039595Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/buildbot.py", line 50, in bb_release_worker
[task 2017-10-16T13:46:43.039637Z]     'next_version': config.params['next_version'],
[task 2017-10-16T13:46:43.039691Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/parameters.py", line 119, in __getitem__
[task 2017-10-16T13:46:43.039933Z]     raise KeyError("taskgraph parameter {!r} not found".format(k))
[task 2017-10-16T13:46:43.040005Z] KeyError: u"taskgraph parameter u'next_version' not found"
:rail Ah yes, that what was my question about. for some reason ``next_version`` parameter is not set. But I create assigned it to `None`. Any idea how to handle this properly?
maybe doing ``'next_version': config.params.get('next_version')`` in ``taskcluster/taskgraph/parameters.py``
(In reply to Rok Garbas [:garbas] from comment #10)
> maybe doing ``'next_version': config.params.get('next_version')`` in
> ``taskcluster/taskgraph/parameters.py``

That should work, but I'm not sure if this is the right approach. Let's ask Aki.
Flags: needinfo?(aki)
For the full graph, having a default next_version (None is fine, as is some logic that increments the current version 58.0b1 to next_version 58.0b2). For the action, we can require the parameter for certain release_promotion_flavors.
Flags: needinfo?(aki)
Attached patch fennec-version-bump-task.patch (obsolete) — Splinter Review
As we discussed in the meeting I'm handling the NEXT_VERSION variable similar as BUILD_NUMBER.
Attachment #8918822 - Attachment is obsolete: true
Attachment #8919064 - Flags: review?(aki)
Comment on attachment 8919064 [details] [diff] [review]
fennec-version-bump-task.patch

This looks good!
The only comment I have is this makes `next_version` a required field for all release promotion flavors; I think we only need `next_version` for certain flavors of release promotion, when we're going to bump the version.

>diff --git a/taskcluster/taskgraph/actions/release_promotion.py b/taskcluster/taskgraph/actions/release_promotion.py
>--- a/taskcluster/taskgraph/actions/release_promotion.py
>+++ b/taskcluster/taskgraph/actions/release_promotion.py
>@@ -112,22 +112,27 @@ def is_release_promotion_available(param
>                 'type': 'array',
>                 'description': ('Optional: an array of taskIds of decision or action '
>                                 'tasks from the previous graph(s) to use to populate '
>                                 'our `previous_graph_kinds`.'),
>                 'items': {
>                     'type': 'string',
>                 },
>             },
>+            'next_version': {
>+                'type': 'string',
>+                'description': 'Next version.',
>+            },
>         },
>-        "required": ['release_promotion_flavor', 'build_number'],
>+        "required": ['release_promotion_flavor', 'build_number', 'next_version'],

Right now that's `publish_fennec`; later I think it'll include `publish_firefox` and possibly `publish_devedition`.

Let's maybe remove `next_version` from the "required" list, and

```
VERSION_BUMP_FLAVORS = (
    'promote_fennec', 'promote_firefox', 'promote_devedition'
)

...

if release_promotion_flavor in VERSION_BUMP_FLAVORS:
    os.environ['NEXT_VERSION'] = str(input['next_version'])
```

which would make those release promotion flavors die with a `KeyError`. If you want a more explicit error, you could check for `input.get('next_version')` and throw a more helpful error message.
Attachment #8919064 - Flags: review?(aki) → feedback+
(In reply to Aki Sasaki [:aki] from comment #14)
> ```
> VERSION_BUMP_FLAVORS = (
>     'promote_fennec', 'promote_firefox', 'promote_devedition'
> )

These should read `publish_fennec`, `publish_firefox`, `publish_devedition` :)
:aki tnx, i updated the patch and made `next_version` property optional, while raising an exceptions when ran with publish_* targets
Attachment #8919064 - Attachment is obsolete: true
Attachment #8919244 - Flags: review?(aki)
Comment on attachment 8919244 [details] [diff] [review]
fennec-version-bump-task.patch

Thanks! I'll try landing this.
Attachment #8919244 - Flags: review?(aki) → review+
:aki some questions so I can do better job next time :)

1. Is there a way to be run target actions with some custom input? so I can avoid the above typos.
2. Which ./mach command do you use to flake8 check?
Status: NEW → RESOLVED
Closed: 7 years ago
Flags: needinfo?(aki)
Resolution: --- → FIXED
(In reply to Rok Garbas [:garbas] from comment #21)
> :aki some questions so I can do better job next time :)
> 
> 1. Is there a way to be run target actions with some custom input? so I can
> avoid the above typos.

I think the typo broke the decision task. Otherwise you can follow https://github.com/mozilla/releasewarrior/blob/master/how-tos/maple.md#debug-release-promotion-action .

> 2. Which ./mach command do you use to flake8 check?

It looks like you can `./mach lint -l flake8 taskcluster` . I installed a virtualenv with flake8, added whatever missing modules, and just ran `flake8 taskcluster`. Either way works; the mach command may be easier.
Flags: needinfo?(aki)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: