Closed
Bug 1407299
Opened 7 years ago
Closed 7 years ago
fennec mozilla-beta version bump in-tree task
Categories
(Release Engineering :: Release Automation: Other, defect)
Release Engineering
Release Automation: Other
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: garbas, Assigned: garbas)
Details
User Story
just for my record here is what i did so far: 1. find a a fennec graph to have something to compare to - select `task group 1` from `Build 1` from `fennec-beta-57.0b5.md`[1] release notes - select `fennec mozilla-beta version bump` task - find `Task Definition` in `Task Details` and download the json - in my case this is:: $ curl https://queue.taskcluster.net/v1/task/NZbWWhuhSaK1IyIGzPI6Zw > current_task.json 2. get a maple working copy our of your mozilla-unified $ cd path/to/mozilla-unified $ hg share ../mozilla-maple $ cd ../mozilla-maple $ hg pull maple $ hg up maple 3. use nixpkgs-mozilla to get development tools for gecko $ git clone git://github.com/mozilla/nixpkgs-mozilla $ cd path/to/mozilla-maple $ nix-shell path/to/nixpkgs-mozilla/release.nix -A gecko.x86_64-linux.gcc (nix-shell) $ ./mach taskgraph --help ... 4. create `publish_fennec.yml` with the following content ``` base_repository: https://hg.mozilla.org/mozilla-central build_date: 1507178755 do_not_optimize: [] existing_tasks: {} filters: - check_servo - target_tasks_method head_ref: a76bd64bcdd3b360918936a3cfbc5e3e604b1d1c head_repository: https://hg.mozilla.org/projects/maple head_rev: a76bd64bcdd3b360918936a3cfbc5e3e604b1d1c include_nightly: true level: '3' message: ' ' moz_build_date: '20171005044555' optimize_target_tasks: true owner: asasaki@mozilla.com project: maple pushdate: 1507178755 pushlog_id: '13' release_history: {} target_tasks_method: publish_fennec try_mode: null try_options: null try_task_config: null ``` Above content was profided by :aki. 5. Created folder `taskcluster/ci/release-version-bump` with `kind.yml` in it. Becuase this is going to be a buildbot task i copied what :aki did `taskcluster/ci/release-bouncer-sub/kind.yml`. 6. Generating the taskcluster task using taskgraph $ cd path/to/mozilla-maple $ ./mach taskgraph target-graph -p path/to/publish_fennec.yml --json | jq '.["release-version-bump-fennec"] | .task' > new_task.json 7. Cleaning up not needed data from new_task.json and current_task.json $ cat new_task.json | jq -S 'del(.created,.deadline,.expires,.priority)' > new_task_clean.json $ cat current_task.json | jq -S 'del(.created,.deadline,.expires,.priority)' > current_task_clean.json 8. Comparing new_task_clean.json and current_task_clean.json $ diff current_task_clean.json new_task_clean.json 9. change/edit/update `taskcluster/ci/release-version-bump/kind.yml` and `taskcluster/taskgraph/target_tasks.py` then rerun 6., 7. and 8. [1] https://github.com/mozilla/releasewarrior/blob/master/releases/fennec-beta-57.0b5.md
Attachments
(1 file, 3 obsolete files)
7.01 KB,
patch
|
mozilla
:
review+
|
Details | Diff | Splinter Review |
No description provided.
Assignee | ||
Updated•7 years ago
|
Assignee: nobody → rgarbas
Comment 1•7 years ago
|
||
Rok: I landed my latest changes on maple for bouncer-sub. - we no longer have to specify BUILD_NUMBER=1 on the commandline. It'll default to 1; we need to specify BUILD_NUMBER if we want something other than 1. - the product is changing from 'mobile' to 'fennec'. I adjusted the candidates_fennec target_tasks_method. - we now have release_promotion: true and routes in the kind.yml: https://hg.mozilla.org/projects/maple/rev/632d3fbe9697#l1.23 - we auto-populate scopes https://hg.mozilla.org/projects/maple/rev/632d3fbe9697#l3.54 I'm also merging in m-c into maple so we don't diverge too far from central. If you're working on a bookmark, you can hg rebase -b BOOKMARK -d DESTINATION where DESTINATION is probably `maple`. If you're looking at `git rebase -i` functionality on a bookmark, you probably want `hg histedit`. https://hg.mozilla.org/projects/maple/pushloghtml may be useful.
Assignee | ||
Updated•7 years ago
|
User Story: (updated)
Assignee | ||
Updated•7 years ago
|
User Story: (updated)
Assignee | ||
Comment 2•7 years ago
|
||
adding `.priority` to the list of things to ignore since its is set based on branch it is running on[1] [1] https://hg.mozilla.org/projects/maple/file/632d3fbe9697/taskcluster/taskgraph/transforms/task.py#l1191
User Story: (updated)
Assignee | ||
Comment 3•7 years ago
|
||
:rail let me know what do you think? is adding next_version parameter the way to go? or is there better way?
Attachment #8918217 -
Flags: review?(rail)
Comment 4•7 years ago
|
||
Comment on attachment 8918217 [details] [diff] [review] version_bump_v1.patch Review of attachment 8918217 [details] [diff] [review]: ----------------------------------------------------------------- I think this should just work. Probably you want to rebase your work against maple and push it there.
Attachment #8918217 -
Flags: review?(rail) → feedback+
Assignee | ||
Comment 5•7 years ago
|
||
:rail looks like I don't have level_2 as it is needed to push to maple repo[1]. I rebase and exported the commit and asked for another review by you. As I'm not 100% sure what I'm doing (learning and I'm getting there :P) I'd ask you to double check that everything is correct. As I'm trying to understand how everything works I'll write down what I checked. If there is something else I should check for please write in reply I checked that: 1. version bump task is not included in candidates-fennec target or when . ``` ./mach taskgraph target-graph -p promote_fennec.yml --json | grep version-bump ``` Question1: is it ok that next_version parameter is required by candidate-fennec target even if it is not used by any tasks? 2. that task generated matched what is already in taskcluster (ignoring .created,.deadline,.expires,.priority,.extra,.payload.sourcestamp,.requires,.retries,.schedulerId fields) ``` $ rm -f maple-publish-fennec-dirty.json && ./mach taskgraph target-graph -p publish-fennec.yml --json > maple-publish-fennec-dirty.json && cat maple-publish-fennec-dirty.json | jq -S '."release-version-bump-fennec".task | del(.created,.deadline,.expires,.priority,.extra,.payload.sourcestamp,.requires,.retries,.schedulerId)' | less ``` and then comparing the output with ``` $ curl https://queue.taskcluster.net/v1/task/NZbWWhuhSaK1IyIGzPI6Zw | jq -S 'del(.created,.deadline,.expires,.priority,.extra,.payload.sourcestamp,.requires,.retries,.schedulerId)' | less ``` [1] https://wiki.mozilla.org/ReleaseEngineering/DisposableProjectRepositories#What_is_a_disposable_project_branch.3F
Attachment #8918217 -
Attachment is obsolete: true
Attachment #8918822 -
Flags: review?(rail)
Comment 6•7 years ago
|
||
Comment on attachment 8918822 [details] [diff] [review] fennec-version-bump-task.patch (In reply to Rok Garbas [:garbas] from comment #5) > :rail looks like I don't have level_2 as it is needed to push to maple > repo[1]. I'll push it for you. Meanwhile we should file a bug to get you at least Level 2 access. > Question1: is it ok that next_version parameter is required by > candidate-fennec target even if it is not used by any tasks? I think this is OK. At least it's not a regression (we pre-set this value in the current implementation). We can probably optimize the schema to be per target task method. > 2. that task generated matched what is already in taskcluster (ignoring > .created,.deadline,.expires,.priority,.extra,.payload.sourcestamp,.requires,. > retries,.schedulerId fields) If taks.payload is the same, we should be ok I'll land this a bit. Can you update the google doc with the corresponding commit please.
Attachment #8918822 -
Flags: review?(rail) → review+
Comment 7•7 years ago
|
||
Comment on attachment 8918822 [details] [diff] [review] fennec-version-bump-task.patch https://hg.mozilla.org/projects/maple/rev/f6280d70de3b0d87f884a1387e69731fb9643f47
Comment 8•7 years ago
|
||
bah, had to backout the push in https://hg.mozilla.org/projects/maple/rev/5cda86ee0df2cd410ed8c77817db7cce381825a2 because it broke the decision task: https://tools.taskcluster.net/groups/IyuAbbPvTG6gTz91_RwYng/tasks/IyuAbbPvTG6gTz91_RwYng/runs/0/logs/public%2Flogs%2Flive.log [task 2017-10-16T13:46:43.035273Z] Generating tasks for release-bouncer-aliases fennec [task 2017-10-16T13:46:43.036142Z] Traceback (most recent call last): [task 2017-10-16T13:46:43.036250Z] File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 165, in taskgraph_decision [task 2017-10-16T13:46:43.036328Z] return taskgraph.decision.taskgraph_decision(options) [task 2017-10-16T13:46:43.036422Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 119, in taskgraph_decision [task 2017-10-16T13:46:43.036677Z] full_task_json = tgg.full_task_graph.to_json() [task 2017-10-16T13:46:43.036960Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 130, in full_task_graph [task 2017-10-16T13:46:43.037217Z] return self._run_until('full_task_graph') [task 2017-10-16T13:46:43.037493Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 287, in _run_until [task 2017-10-16T13:46:43.037562Z] k, v = self._run.next() [task 2017-10-16T13:46:43.037656Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 217, in _run [task 2017-10-16T13:46:43.037710Z] new_tasks = kind.load_tasks(self.parameters, list(all_tasks.values())) [task 2017-10-16T13:46:43.038187Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 65, in load_tasks [task 2017-10-16T13:46:43.038239Z] for task_dict in transforms(trans_config, inputs)] [task 2017-10-16T13:46:43.038307Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1303, in check_run_task_caches [task 2017-10-16T13:46:43.038347Z] for task in tasks: [task 2017-10-16T13:46:43.038406Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1142, in build_task [task 2017-10-16T13:46:43.038438Z] for task in tasks: [task 2017-10-16T13:46:43.038494Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1112, in add_index_routes [task 2017-10-16T13:46:43.038524Z] for task in tasks: [task 2017-10-16T13:46:43.038578Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1002, in validate [task 2017-10-16T13:46:43.038767Z] for task in tasks: [task 2017-10-16T13:46:43.038845Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/__init__.py", line 165, in make_task_description [task 2017-10-16T13:46:43.038960Z] configure_taskdesc_for_run(config, job, taskdesc, impl) [task 2017-10-16T13:46:43.039199Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/__init__.py", line 217, in configure_taskdesc_for_run [task 2017-10-16T13:46:43.039242Z] func(config, job, taskdesc) [task 2017-10-16T13:46:43.039454Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/buildbot.py", line 110, in mozharness_on_buildbot_bridge [task 2017-10-16T13:46:43.039526Z] bb_release_worker(config, worker, run) [task 2017-10-16T13:46:43.039595Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/job/buildbot.py", line 50, in bb_release_worker [task 2017-10-16T13:46:43.039637Z] 'next_version': config.params['next_version'], [task 2017-10-16T13:46:43.039691Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/parameters.py", line 119, in __getitem__ [task 2017-10-16T13:46:43.039933Z] raise KeyError("taskgraph parameter {!r} not found".format(k)) [task 2017-10-16T13:46:43.040005Z] KeyError: u"taskgraph parameter u'next_version' not found"
Assignee | ||
Comment 9•7 years ago
|
||
:rail Ah yes, that what was my question about. for some reason ``next_version`` parameter is not set. But I create assigned it to `None`. Any idea how to handle this properly?
Assignee | ||
Comment 10•7 years ago
|
||
maybe doing ``'next_version': config.params.get('next_version')`` in ``taskcluster/taskgraph/parameters.py``
Comment 11•7 years ago
|
||
(In reply to Rok Garbas [:garbas] from comment #10) > maybe doing ``'next_version': config.params.get('next_version')`` in > ``taskcluster/taskgraph/parameters.py`` That should work, but I'm not sure if this is the right approach. Let's ask Aki.
Flags: needinfo?(aki)
Comment 12•7 years ago
|
||
For the full graph, having a default next_version (None is fine, as is some logic that increments the current version 58.0b1 to next_version 58.0b2). For the action, we can require the parameter for certain release_promotion_flavors.
Flags: needinfo?(aki)
Assignee | ||
Comment 13•7 years ago
|
||
As we discussed in the meeting I'm handling the NEXT_VERSION variable similar as BUILD_NUMBER.
Attachment #8918822 -
Attachment is obsolete: true
Attachment #8919064 -
Flags: review?(aki)
Comment 14•7 years ago
|
||
Comment on attachment 8919064 [details] [diff] [review] fennec-version-bump-task.patch This looks good! The only comment I have is this makes `next_version` a required field for all release promotion flavors; I think we only need `next_version` for certain flavors of release promotion, when we're going to bump the version. >diff --git a/taskcluster/taskgraph/actions/release_promotion.py b/taskcluster/taskgraph/actions/release_promotion.py >--- a/taskcluster/taskgraph/actions/release_promotion.py >+++ b/taskcluster/taskgraph/actions/release_promotion.py >@@ -112,22 +112,27 @@ def is_release_promotion_available(param > 'type': 'array', > 'description': ('Optional: an array of taskIds of decision or action ' > 'tasks from the previous graph(s) to use to populate ' > 'our `previous_graph_kinds`.'), > 'items': { > 'type': 'string', > }, > }, >+ 'next_version': { >+ 'type': 'string', >+ 'description': 'Next version.', >+ }, > }, >- "required": ['release_promotion_flavor', 'build_number'], >+ "required": ['release_promotion_flavor', 'build_number', 'next_version'], Right now that's `publish_fennec`; later I think it'll include `publish_firefox` and possibly `publish_devedition`. Let's maybe remove `next_version` from the "required" list, and ``` VERSION_BUMP_FLAVORS = ( 'promote_fennec', 'promote_firefox', 'promote_devedition' ) ... if release_promotion_flavor in VERSION_BUMP_FLAVORS: os.environ['NEXT_VERSION'] = str(input['next_version']) ``` which would make those release promotion flavors die with a `KeyError`. If you want a more explicit error, you could check for `input.get('next_version')` and throw a more helpful error message.
Attachment #8919064 -
Flags: review?(aki) → feedback+
Comment 15•7 years ago
|
||
(In reply to Aki Sasaki [:aki] from comment #14) > ``` > VERSION_BUMP_FLAVORS = ( > 'promote_fennec', 'promote_firefox', 'promote_devedition' > ) These should read `publish_fennec`, `publish_firefox`, `publish_devedition` :)
Assignee | ||
Comment 16•7 years ago
|
||
:aki tnx, i updated the patch and made `next_version` property optional, while raising an exceptions when ran with publish_* targets
Attachment #8919064 -
Attachment is obsolete: true
Attachment #8919244 -
Flags: review?(aki)
Comment 17•7 years ago
|
||
Comment on attachment 8919244 [details] [diff] [review] fennec-version-bump-task.patch Thanks! I'll try landing this.
Attachment #8919244 -
Flags: review?(aki) → review+
Comment 18•7 years ago
|
||
https://hg.mozilla.org/projects/maple/rev/f72f105218c59019f85cbc70b18cac8f97a9cc3e https://treeherder.mozilla.org/#/jobs?repo=maple&revision=f72f105218c59019f85cbc70b18cac8f97a9cc3e
Comment 19•7 years ago
|
||
Typo fix https://hg.mozilla.org/projects/maple/rev/57a92aaf9e0fca804d5b7f75d9e0489cea8b91cf
Comment 20•7 years ago
|
||
Flake8 fix :) https://hg.mozilla.org/projects/maple/rev/2544b5aba5d6e52bf37779cf1798d8617c0f46ed
Assignee | ||
Comment 21•7 years ago
|
||
:aki some questions so I can do better job next time :) 1. Is there a way to be run target actions with some custom input? so I can avoid the above typos. 2. Which ./mach command do you use to flake8 check?
Status: NEW → RESOLVED
Closed: 7 years ago
Flags: needinfo?(aki)
Resolution: --- → FIXED
Comment 22•7 years ago
|
||
(In reply to Rok Garbas [:garbas] from comment #21) > :aki some questions so I can do better job next time :) > > 1. Is there a way to be run target actions with some custom input? so I can > avoid the above typos. I think the typo broke the decision task. Otherwise you can follow https://github.com/mozilla/releasewarrior/blob/master/how-tos/maple.md#debug-release-promotion-action . > 2. Which ./mach command do you use to flake8 check? It looks like you can `./mach lint -l flake8 taskcluster` . I installed a virtualenv with flake8, added whatever missing modules, and just ran `flake8 taskcluster`. Either way works; the mach command may be easier.
Flags: needinfo?(aki)
You need to log in
before you can comment on or make changes to this bug.
Description
•