1289980 - Add new routes for releases

Status: RESOLVED DUPLICATE of bug 1425571

(Release Engineering :: Release Automation: Other, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

In Taskcluster we have various routes for nightly builds including
  gecko.v2.<branch>.nightly.<yyyy>.<mm>.<dd>.revision.<revision>.<app>.<platform>
while for releases we have only
  releases.v1.<branch>.<revision>.<app>.<version>.build<build#>

To enable Socorro to query for new releases I'd like to add a date-based route. Here are some guesses to start discussion
  releases.v1.<branch>.<yyyy>.<mm>.<dd>.<version>.build<build#>
  releases.v1.<yyyy>.<mm>.<dd>.<branch>.<version>.build<build#>
or even
  gecko.v2.<branch>.releases.<yyyy>.<mm>.<dd>.<version>.build<build#>
(A new job will have to put something in these places for Socorro to find.)

catlee also suggested a version-based route, maybe one of
  releases.v1.<branch>.version.<version>.build<build#>
  releases.v1.<app>.version.<version>.build<build#>
  releases.v1.version.<version>.<app>.build<build#>

rail, mshal - what would you prefer ?

Comment 1

[Rail Aliiev [:rail]]

(In reply to Nick Thomas [:nthomas] from comment #0)

> To enable Socorro to query for new releases I'd like to add a date-based
> route. Here are some guesses to start discussion
>   releases.v1.<branch>.<yyyy>.<mm>.<dd>.<version>.build<build#>
>   releases.v1.<yyyy>.<mm>.<dd>.<branch>.<version>.build<build#>
> or even
>   gecko.v2.<branch>.releases.<yyyy>.<mm>.<dd>.<version>.build<build#>
> (A new job will have to put something in these places for Socorro to find.)

Hmm, the routes do not contain <app>. Is this on purpose?
 
> catlee also suggested a version-based route, maybe one of
>   releases.v1.<branch>.version.<version>.build<build#>
>   releases.v1.<app>.version.<version>.build<build#>
>   releases.v1.version.<version>.<app>.build<build#>

or releases.v1.<app>.<branch>.version.<version>.build<build#> to make sure we isolate branches from each other.

Comment 2

[Michael Shal [:mshal]]

(In reply to Rail Aliiev [:rail] from comment #1)
> > catlee also suggested a version-based route, maybe one of
> >   releases.v1.<branch>.version.<version>.build<build#>
> >   releases.v1.<app>.version.<version>.build<build#>
> >   releases.v1.version.<version>.<app>.build<build#>
> 
> or releases.v1.<app>.<branch>.version.<version>.build<build#> to make sure
> we isolate branches from each other.

Is it beneficial to isolate the branches? It looks like on FTP all releases are dropped in the same directory, but you can tell them apart based on the version string (betas have a 'b', etc). Is "version" the same thing here? Like "47.0", "47.0b9", etc.

Comment 3

[Rail Aliiev [:rail]]

Hmm, probably it shouldn't be an issue, because our versions are unique, right. I worried about ESR for a moment (the first cycle has the same app versions with release), but we do not use app version, we always add "esr".

releases.v1.<app>.version.<version>.build<build#> SGTM. Maybe we can even drop "version" after <app>?

Comment 4

[Michael Shal [:mshal]]

(In reply to Nick Thomas [:nthomas] from comment #0)
> To enable Socorro to query for new releases I'd like to add a date-based
> route. Here are some guesses to start discussion
>   releases.v1.<branch>.<yyyy>.<mm>.<dd>.<version>.build<build#>

Since releases.v1.<branch> already contains a list of revisions, I'm not sure it makes sense to also add <yyyy> at that level. For example, if you look at releases.v1.mozilla-release, you'd get a mix of revisions and years. I think it helps to have each level the same type of information (only revisions, or only years). To separate the two you can insert keywords like "revision" or "pushdate" (maybe "builddate" makes more sense for releases? I'm not sure which date we'd use here). So maybe something like:

releases.v2.<branch>.pushdate.<yyyy>...
releases.v2.<branch>.revision.<sha1>...

I didn't quite manage to do this everywhere in gecko.v2 unfortunately - eg: gecko.v2.mozilla-central.nightly has both dates and keywords ("latest" and "revision").

>   releases.v1.<yyyy>.<mm>.<dd>.<branch>.<version>.build<build#>

Similarly here, with releases.v1 as it currently is, browsing there would show both branches and years in the same group.

> or even
>   gecko.v2.<branch>.releases.<yyyy>.<mm>.<dd>.<version>.build<build#>
> (A new job will have to put something in these places for Socorro to find.)

I don't have a preference either way of whether or not it goes in gecko.v2 or releases.*. I guess we already have a top-level releases namespace, though. Would just using gecko.v2 simplify things?

Comment 5

[Michael Shal [:mshal]]

(In reply to Rail Aliiev [:rail] from comment #3)
> releases.v1.<app>.version.<version>.build<build#> SGTM. Maybe we can even
> drop "version" after <app>?

I think "version" would make sense if that's where we want to split between version / pushdate / revision. eg:

releases.v2.<app>.version.<version>.build<build#>
releases.v2.<app>.revision.<sha1>.<version>.build<build#>
releases.v2.<app>.pushdate.<yyyy>.<mm>.<dd>.<version>.build<build#>

Though do we need a branch separator when looking at revisions?

Comment 6

[Connor Sheehan [:sheehan]]

Attachment: Move common routes and routing key prefixes to a single file

Rail told me that many of the routing keys in the release tasks had common prefixes or that they were the same across the whole task graph. I have defined these prefixes and routes in a single file that is included at the top of release_graph.yml so they can be changed from a single location. I also included a temporary first-draft of the version key and date key mentioned above.

Once we decide on the format of these new keys I can commit this change and make a PR.

Comment 7

[Jordan Lund (:jlund)]

(In reply to Michael Shal [:mshal] from comment #5)
> (In reply to Rail Aliiev [:rail] from comment #3)

> I think "version" would make sense if that's where we want to split between
> version / pushdate / revision. eg:
> 
> releases.v2.<app>.version.<version>.build<build#>
> releases.v2.<app>.revision.<sha1>.<version>.build<build#>
> releases.v2.<app>.pushdate.<yyyy>.<mm>.<dd>.<version>.build<build#>
> 
> Though do we need a branch separator when looking at revisions?

this makes sense to me and was the last comment without objection. I am going to unbitrot connor's work and then apply the following template:

releases.v2.<app>.version.<version>.build<build#>
releases.v2.<app>.revision.<branch>.<sha1>.<version>.build<build#>
releases.v2.<app>.pushdate.<yyyy>.<mm>.<dd>.<version>.build<build#>

speak now or forever hold your peace! just kidding, thanks to connor's routes.yml globals, further changes will be easy :)

Comment 8

[Jordan Lund (:jlund)]

oh, and there is the latest route as well. I think that's not really useful for version or date based routes as their route value is chronological. So I'll just keep the latest route for revision.

iow -

before:
index.releases.v1.{{ branch }}.latest.{{ product }}.latest

after:
releases.v2.{{ app }}.revision.{{ branch }}.latest.{{ version }}.build{{ build_num }}

This will leave us with 4 routes:

- version
- pushdate
- revision
- latest_revision

each of them starts with: `releases.v2.{{ app }}`

and each of them finishes with: `{{ version }}.build{{ build_num }}`

Comment 9

[Jordan Lund (:jlund)]

Comment on attachment 8775747 [details] [diff] [review]
Move common routes and routing key prefixes to a single file

Review of attachment 8775747 [details] [diff] [review]:
-----------------------------------------------------------------

::: releasetasks/templates/firefox/bb_update_verify.yml.tmpl
@@ +22,5 @@
>          scopes:
>              - project:releng:buildbot-bridge:builder-name:{{ uv_buildername }}
>          routes:
> +            - {{ dated_routing_key }}
> +            - {{ version_routing_key }}

I'm making the assumption that we want to have the same suffix to the version and dated routes as we do for the revision based ones.

i.e. add '.{{ channel }}.{{ platform }}.{{ chunk }}' here

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8775747 [details] [diff] [review]
Move common routes and routing key prefixes to a single file

Review of attachment 8775747 [details] [diff] [review]:
-----------------------------------------------------------------

::: releasetasks/templates/firefox/l10n.yml.tmpl
@@ -68,5 @@
>          scopes:
>              - project:releng:buildbot-bridge:builder-name:{{ buildername }}
>          routes:
> -            - index.releases.v1.{{ branch }}.{{ revision }}.{{ product }}.{{ version | replace(".", "_") }}.build{{ buildNumber }}.l10n_artifacts.{{ platform }}.{{ chunk }}
> -            - index.releases.v1.{{ branch }}.latest.{{ product }}.latest.l10n_artifacts.{{ platform }}.{{ chunk }}

hm, why do we sometimes just have index.releases.v1.* in the routes, and other times (see below)..

@@ -100,5 @@
>          priority: "high"
>          retries: 5
>          routes:
> -            - tc-treeherder-stage.v2.{{ branch }}.{{ revision }}.{{ pushlog_id }}
> -            - tc-treeherder.v2.{{ branch }}.{{ revision }}.{{ pushlog_id }}

(continued) and other times we also have tc-treeherder* routes


when should we *not* have tc-treeherder* routes?

Comment 11

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #10)
> Comment on attachment 8775747 [details] [diff] [review]

> 
> when should we *not* have tc-treeherder* routes?

ignore this. I found all the answers here: https://github.com/mozilla/releasetasks/pull/184

Comment 12

[Jordan Lund (:jlund)]

Attachment: github pr

mshal r? just on: routes.yml.tmpl https://github.com/mozilla/releasetasks/pull/196/commits/270214743ff3a59972651faf3b381e46c9e03d10#diff-2fb360990671f23e992b547e58727b28

rail r? whole patch :)

disclaimer: this only adds prefixed routes. it does not change the treeherder ones or the individual task suffix routes. maybe we could do a follow up if you would like to change those?

Comment 13

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #12)
> Created attachment 8790565 [details] [review]
> github pr
> 
> mshal r? just on: routes.yml.tmpl
> https://github.com/mozilla/releasetasks/pull/196/commits/
> 270214743ff3a59972651faf3b381e46c9e03d10#diff-
> 2fb360990671f23e992b547e58727b28

mshal: sorry, --amend + push forced new shas. so correct url is: https://github.com/mozilla/releasetasks/pull/196/files#diff-2fb360990671f23e992b547e58727b28

Comment 14

[Michael Shal [:mshal]]

Comment on attachment 8790565 [details] [review]
github pr

Added a bunch of comments / questions to the PR. Looks pretty good overall! I just want to make sure we get it the way we want from the start. If my comments are confusing we can try to sort it out on vidyo :)

Comment 15

[Rail Aliiev [:rail]]

Mihai, you wanted to poke something. ;) Can you look at this bug?

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

@rail: Sure! today is full of meetings and Italian food in the office :p If tomorrow/friday sounds good enough, feel free to assign it to me and I'll have a look ;)

I'll leave the NI for reference for now.

Comment 17

[Rail Aliiev [:rail]]

No rush, at all.

Comment 18

[Jordan Lund (:jlund)]

looks like I let this slip. Now that I am back, I can relook at the review comments and let mihai finish nightly stuff :)

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

FTR, the plan for Socorro to start polling the TC index for nightlies and releases got put on hold when I discovered that we are beetmoving TC nightly builds to archive.m.o. ie Socorro keeps on looking where it always has.

Comment 20

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #18)
> looks like I let this slip. Now that I am back, I can relook at the review
> comments and let mihai finish nightly stuff :)

more slipping. This isn't in my near radar. Perhaps it is worth thinking about in-tree rather than fixing releasetasks with bugs like this.

Comment 21

[Aki Sasaki (not active)]

I think we should dup against bug 1425571.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

FWIW, I think I read a blog post that Socorro are using https://mozilla-services.github.io/buildhub/ now.

Comment 23

[Justin Wood (:Callek)]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #22)
> FWIW, I think I read a blog post that Socorro are using
> https://mozilla-services.github.io/buildhub/ now.

Not yet, but likely soon (I bet blog post was http://bluesock.org/~willkg/blog/mozilla/socorro_2017.html which points at https://bugzilla.mozilla.org/show_bug.cgi?id=1366301)