1343393 - fennec staging candidates 53b1 on jamun

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P1)

Comment 1

[Aki Sasaki (not active)]

Attachment: beetmover.diff

Staging creds for release bucket for now

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8842220 [details] [diff] [review]
beetmover.diff

Review of attachment 8842220 [details] [diff] [review]:
-----------------------------------------------------------------

Bleah, I meant to split them nicely before but

Comment 3

[Aki Sasaki (not active)]

I've pinned beetmoverworker-{1,2} to my puppet env for now.

Comment 4

[Aki Sasaki (not active)]

https://hg.mozilla.org/projects/jamun/rev/374872d263535faf0e0024f622d112f578756175
bug 1343393 - enable staging fennec via candidates_fennec target_tasks_method

Comment 5

[Aki Sasaki (not active)]

Attachment: candidates_fennec.diff

This patch:

* adds a candidates_fennec target_task_method, which I am currently calling from https://tools.taskcluster.net/hooks/#project-releng/candidates-fennec-dev
    * this function disables balrog, since a) I think we might be pushing to the wrong channel atm, and b) we don't have release update channel support yet
* adds a `build_number` to the task.payload in beetmover if a) this is a candidates_fennec graph, and b) BUILD_NUMBER is defined in the environment.
    * a candidates_fennec graph without BUILD_NUMBER in the env will throw a ValueError.
* Removes jamun from the valid branches.  We can patch jamun for testing, but let's not have it ride the trains
* Changes the beetmover bucket logic!  Since we have both 'push-to-nightly' and 'push-to-candidates' live on jamun, I needed something other than the project to define the bucket scopes.  I'm using the new `get_scope_from_target_method_and_project` function, which looks at both target_task_method and project, and only gives you privileged scopes if both the project and target_task_method align.

What do you think?

(Go ahead and land your other util.scriptworker changes when you're ready; I can deal with bitrot.)

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8842289 [details] [diff] [review]
candidates_fennec.diff

Review of attachment 8842289 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, none of the comments are blocking! Thanks for verbossed comments!

::: taskcluster/taskgraph/target_tasks.py
@@ +228,5 @@
> +    """Select the set of tasks required for a candidates build of fennec. The
> +    nightly build process involves a pipeline of builds, signing,
> +    and, eventually, uploading the tasks to balrog."""
> +    filtered_for_project = target_tasks_nightly(full_task_graph, parameters)
> +    def filter(task):

I think the nightly check doesn't hurt but is useless here as it will always return True, because filtered_for_project is already and instance of `target_tasks_nightly` which is at its turn a filtering in the full task graph retaining *just* the android-based for which the task.attributes returns *nightly*. So I think it's safe to simplify this to:

`def filter(task):
    return task.kind not in ['balrog']
`

::: taskcluster/taskgraph/transforms/task.py
@@ +15,5 @@
>  
>  from taskgraph.util.treeherder import split_symbol
>  from taskgraph.transforms.base import TransformSequence
>  from taskgraph.util.schema import validate_schema
> +from taskgraph.util.scriptworker import get_release_build_number

In bug 1343349 I'm adding a new ~/taskcluster/taskgraph/util/release.py in order to add some stuff that's not related with the scopes. Initially I was gonna suggest we move stuff like this beetmover-buildnumber thing + my routes from bug 1343349 there to avoid polluting the scriptworker.py scopes script but I changed my mind. I think your approach is better as we might reuse some of those scopes dicts in the util/helper functions as you did with buildnumber ++

::: taskcluster/taskgraph/util/scriptworker.py
@@ -52,5 @@
>  BEETMOVER_SCOPE_ALIAS_TO_PROJECT = [[
>      'all-nightly-branches', set([
>          'mozilla-central',
>          'mozilla-aurora',
> -        'jamun',

Good call here!

@@ +57,1 @@
>          'mozilla-beta',

Before Monday, aren't we supposed to delete the "mozilla-beta" and "mozilla-release" from here to make sure the release scopes are pointed at?

@@ +79,5 @@
>  BEETMOVER_SCOPE_ALIAS_TO_TARGET_TASK = [[
>      'all-nightly-tasks', set([
>          'nightly_fennec',
>          'nightly_linux',
>          'mozilla_beta_tasks',

Normally, I'd say we need to delete the "mozilla-beta" and "mozilla-release" from here and add them in the "all-release-tasks" section but I guess we'd be calling the "fennec_candidates" target task method anyway for the real beta 1 on Monday, right?

@@ +176,5 @@
>              return alias_to_scope_map[alias]
>      return alias_to_scope_map['default']
>  
>  
> +def get_scope_from_target_method_and_project(alias_to_tasks_map, alias_to_project_map,

Good job on refactoring this to take both into account!

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #6)
> @@ +57,1 @@
> >          'mozilla-beta',
> 
> Before Monday, aren't we supposed to delete the "mozilla-beta" and
> "mozilla-release" from here to make sure the release scopes are pointed at?
> 
> @@ +79,5 @@
> >  BEETMOVER_SCOPE_ALIAS_TO_TARGET_TASK = [[
> >      'all-nightly-tasks', set([
> >          'nightly_fennec',
> >          'nightly_linux',
> >          'mozilla_beta_tasks',
> 
> Normally, I'd say we need to delete the "mozilla-beta" and "mozilla-release"
> from here and add them in the "all-release-tasks" section but I guess we'd
> be calling the "fennec_candidates" target task method anyway for the real
> beta 1 on Monday, right?
> 

Bah, nevermind these two questions from above comment. I just reread https://bugzilla.mozilla.org/show_bug.cgi?id=1343144#c3 :) Forgot about the push-to-nightly for CI on beta and release.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/232a19e40a52
add graph logic to handle a fennec beta build. r=mtabara DONTBUILD

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

@aki: hope you don't mind, I took the liberty to land the patch for you on inbound, to save some time.

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Comparing http://archive.mozilla.org/pub/mobile/releases/52.0b8/ with the staging fennec build "53.0" out of jamun, my observations are these:

1. At the root folder level we're missing the following
Dir 	logs/ 		
Dir 	source/ 		
File 	KEY 	4K 	20-Feb-2017 22:15
File 	SHA512SUMS 	14K 	20-Feb-2017 22:15
File 	SHA512SUMS.asc 	836 	20-Feb-2017 22:15

The Key and SHA512SUMS are coming from BBB jobs if I'm not wrong. Not sure what we're gonna do with the logs since we don't have a centralized BB way to do that. But I'm sure there are workarounds (to add them in beetmover or so).

2. At the single locale level, we seem to be having an extra "ur" locale but I guess that's fine. Most likelt that's been added recently in aurora and has been riding the trains.

3. At the en-US locale, for both Fennecx86 and Fennec:

a) we seem to be missing:

File 	fennec-52.0b8.en-US.android-arm.crashreporter-symbols-full.zip 	453M 	20-Feb-2017 21:16
File 	fennec-52.0b8.en-US.android-arm.crashreporter-symbols.zip 	97M 	20-Feb-2017 21:17

b) we're missing the /host/bin/{mar,mbsdiff}. I see they're present in beetmover templates but they're not being specified in the upstreamArtifacts, see last fennec staging task[2]

c) checksums look good, except that we're short on the {mar,mbsdiff} artifacts

4. At the "multi" locale we're 100% accurate for both Fennecx86 and Fennec.

[1]: https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/templates/fennec_candidates.yml#L102
[2]: https://tools.taskcluster.net/task-group-inspector/#/QQjEh3q1TOu2liv57iHawg/OzMj6pdsQJSM6H9oKIu7yg?_k=mlp0pv

Comment 11

[Aki Sasaki (not active)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ec8ac10801d6c054a3e74f89fd1857c0bdac638b
bug 1343393 - fix flake8 linting. r=bustage

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1343393 - add missing script_config in beetmoverworker.

Review commit: https://reviewboard.mozilla.org/r/116396/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/116396/

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 8842625 [details]
Bug 1343393 - add missing script_config in beetmoverworker.

https://reviewboard.mozilla.org/r/116396/#review118026

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8842625 [details]
Bug 1343393 - add missing script_config in beetmoverworker.

https://hg.mozilla.org/build/puppet/rev/acca7d430371

Comment 15

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/232a19e40a52
https://hg.mozilla.org/mozilla-central/rev/ec8ac10801d6

Comment 16

[Aki Sasaki (not active)]

Rail, it looks like we might have a good candidates build for Fennec now.
I could use a hand running the candidates->releases portion Thursday if you have cycles? or I can pester someone else :)

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Wes Kocher (:KWierso) from comment #15)
> https://hg.mozilla.org/mozilla-central/rev/232a19e40a52
> https://hg.mozilla.org/mozilla-central/rev/ec8ac10801d6

:aki melded these two and uplifted them to aurora:
https://hg.mozilla.org/releases/mozilla-aurora/rev/15eece89ef86

Comment 18

[Rail Aliiev [:rail]]

(In reply to Aki Sasaki [:aki] from comment #16)
> Rail, it looks like we might have a good candidates build for Fennec now.
> I could use a hand running the candidates->releases portion Thursday if you
> have cycles? or I can pester someone else :)

Sure!

Comment 19

[Aki Sasaki (not active)]

Thanks! To be clear, I or someone else can do it, but will probably need pointers or help.

Comment 20

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Aki Sasaki [:aki] from comment #19)
> Thanks! To be clear, I or someone else can do it, but will probably need
> pointers or help.

I can help / volunteer to help here with some pointers. I'm looking forward for a break from {beetmover,balrog} work.

Comment 21

[Aki Sasaki (not active)]

I think we're done with the staging candidates build portion.  :kmoir is working on the releasetasks graph1 triggering the candidates_fennec graph in bug 1343649, and :mtabara is working on the candidates->release portion.

We can keep using this as a tracking bug, though.

Comment 22

[Aki Sasaki (not active)]

The releasetasks graph work are both in progress and tracked in the phase2 relpro trello board.  I'm going to resolve this bug.

Comment 23

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.