Closed Bug 1289980 Opened 4 years ago Closed 3 years ago

Add new routes for releases

Categories

(Release Engineering :: Release Automation: Other, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1425571

People

(Reporter: nthomas, Unassigned)

References

Details

Attachments

(1 file, 1 obsolete file)

In Taskcluster we have various routes for nightly builds including
  gecko.v2.<branch>.nightly.<yyyy>.<mm>.<dd>.revision.<revision>.<app>.<platform>
while for releases we have only
  releases.v1.<branch>.<revision>.<app>.<version>.build<build#>

To enable Socorro to query for new releases I'd like to add a date-based route. Here are some guesses to start discussion
  releases.v1.<branch>.<yyyy>.<mm>.<dd>.<version>.build<build#>
  releases.v1.<yyyy>.<mm>.<dd>.<branch>.<version>.build<build#>
or even
  gecko.v2.<branch>.releases.<yyyy>.<mm>.<dd>.<version>.build<build#>
(A new job will have to put something in these places for Socorro to find.)

catlee also suggested a version-based route, maybe one of
  releases.v1.<branch>.version.<version>.build<build#>
  releases.v1.<app>.version.<version>.build<build#>
  releases.v1.version.<version>.<app>.build<build#>

rail, mshal - what would you prefer ?
(In reply to Nick Thomas [:nthomas] from comment #0)

> To enable Socorro to query for new releases I'd like to add a date-based
> route. Here are some guesses to start discussion
>   releases.v1.<branch>.<yyyy>.<mm>.<dd>.<version>.build<build#>
>   releases.v1.<yyyy>.<mm>.<dd>.<branch>.<version>.build<build#>
> or even
>   gecko.v2.<branch>.releases.<yyyy>.<mm>.<dd>.<version>.build<build#>
> (A new job will have to put something in these places for Socorro to find.)

Hmm, the routes do not contain <app>. Is this on purpose?
 
> catlee also suggested a version-based route, maybe one of
>   releases.v1.<branch>.version.<version>.build<build#>
>   releases.v1.<app>.version.<version>.build<build#>
>   releases.v1.version.<version>.<app>.build<build#>

or releases.v1.<app>.<branch>.version.<version>.build<build#> to make sure we isolate branches from each other.
(In reply to Rail Aliiev [:rail] from comment #1)
> > catlee also suggested a version-based route, maybe one of
> >   releases.v1.<branch>.version.<version>.build<build#>
> >   releases.v1.<app>.version.<version>.build<build#>
> >   releases.v1.version.<version>.<app>.build<build#>
> 
> or releases.v1.<app>.<branch>.version.<version>.build<build#> to make sure
> we isolate branches from each other.

Is it beneficial to isolate the branches? It looks like on FTP all releases are dropped in the same directory, but you can tell them apart based on the version string (betas have a 'b', etc). Is "version" the same thing here? Like "47.0", "47.0b9", etc.
Hmm, probably it shouldn't be an issue, because our versions are unique, right. I worried about ESR for a moment (the first cycle has the same app versions with release), but we do not use app version, we always add "esr".

releases.v1.<app>.version.<version>.build<build#> SGTM. Maybe we can even drop "version" after <app>?
(In reply to Nick Thomas [:nthomas] from comment #0)
> To enable Socorro to query for new releases I'd like to add a date-based
> route. Here are some guesses to start discussion
>   releases.v1.<branch>.<yyyy>.<mm>.<dd>.<version>.build<build#>

Since releases.v1.<branch> already contains a list of revisions, I'm not sure it makes sense to also add <yyyy> at that level. For example, if you look at releases.v1.mozilla-release, you'd get a mix of revisions and years. I think it helps to have each level the same type of information (only revisions, or only years). To separate the two you can insert keywords like "revision" or "pushdate" (maybe "builddate" makes more sense for releases? I'm not sure which date we'd use here). So maybe something like:

releases.v2.<branch>.pushdate.<yyyy>...
releases.v2.<branch>.revision.<sha1>...

I didn't quite manage to do this everywhere in gecko.v2 unfortunately - eg: gecko.v2.mozilla-central.nightly has both dates and keywords ("latest" and "revision").

>   releases.v1.<yyyy>.<mm>.<dd>.<branch>.<version>.build<build#>

Similarly here, with releases.v1 as it currently is, browsing there would show both branches and years in the same group.

> or even
>   gecko.v2.<branch>.releases.<yyyy>.<mm>.<dd>.<version>.build<build#>
> (A new job will have to put something in these places for Socorro to find.)

I don't have a preference either way of whether or not it goes in gecko.v2 or releases.*. I guess we already have a top-level releases namespace, though. Would just using gecko.v2 simplify things?
(In reply to Rail Aliiev [:rail] from comment #3)
> releases.v1.<app>.version.<version>.build<build#> SGTM. Maybe we can even
> drop "version" after <app>?

I think "version" would make sense if that's where we want to split between version / pushdate / revision. eg:

releases.v2.<app>.version.<version>.build<build#>
releases.v2.<app>.revision.<sha1>.<version>.build<build#>
releases.v2.<app>.pushdate.<yyyy>.<mm>.<dd>.<version>.build<build#>

Though do we need a branch separator when looking at revisions?
Rail told me that many of the routing keys in the release tasks had common prefixes or that they were the same across the whole task graph. I have defined these prefixes and routes in a single file that is included at the top of release_graph.yml so they can be changed from a single location. I also included a temporary first-draft of the version key and date key mentioned above.

Once we decide on the format of these new keys I can commit this change and make a PR.
Assignee: nobody → jlund
(In reply to Michael Shal [:mshal] from comment #5)
> (In reply to Rail Aliiev [:rail] from comment #3)

> I think "version" would make sense if that's where we want to split between
> version / pushdate / revision. eg:
> 
> releases.v2.<app>.version.<version>.build<build#>
> releases.v2.<app>.revision.<sha1>.<version>.build<build#>
> releases.v2.<app>.pushdate.<yyyy>.<mm>.<dd>.<version>.build<build#>
> 
> Though do we need a branch separator when looking at revisions?

this makes sense to me and was the last comment without objection. I am going to unbitrot connor's work and then apply the following template:

releases.v2.<app>.version.<version>.build<build#>
releases.v2.<app>.revision.<branch>.<sha1>.<version>.build<build#>
releases.v2.<app>.pushdate.<yyyy>.<mm>.<dd>.<version>.build<build#>

speak now or forever hold your peace! just kidding, thanks to connor's routes.yml globals, further changes will be easy :)
oh, and there is the latest route as well. I think that's not really useful for version or date based routes as their route value is chronological. So I'll just keep the latest route for revision.

iow -

before:
index.releases.v1.{{ branch }}.latest.{{ product }}.latest

after:
releases.v2.{{ app }}.revision.{{ branch }}.latest.{{ version }}.build{{ build_num }}

This will leave us with 4 routes:

- version
- pushdate
- revision
- latest_revision

each of them starts with: `releases.v2.{{ app }}`

and each of them finishes with: `{{ version }}.build{{ build_num }}`
Comment on attachment 8775747 [details] [diff] [review]
Move common routes and routing key prefixes to a single file

Review of attachment 8775747 [details] [diff] [review]:
-----------------------------------------------------------------

::: releasetasks/templates/firefox/bb_update_verify.yml.tmpl
@@ +22,5 @@
>          scopes:
>              - project:releng:buildbot-bridge:builder-name:{{ uv_buildername }}
>          routes:
> +            - {{ dated_routing_key }}
> +            - {{ version_routing_key }}

I'm making the assumption that we want to have the same suffix to the version and dated routes as we do for the revision based ones.

i.e. add '.{{ channel }}.{{ platform }}.{{ chunk }}' here
Comment on attachment 8775747 [details] [diff] [review]
Move common routes and routing key prefixes to a single file

Review of attachment 8775747 [details] [diff] [review]:
-----------------------------------------------------------------

::: releasetasks/templates/firefox/l10n.yml.tmpl
@@ -68,5 @@
>          scopes:
>              - project:releng:buildbot-bridge:builder-name:{{ buildername }}
>          routes:
> -            - index.releases.v1.{{ branch }}.{{ revision }}.{{ product }}.{{ version | replace(".", "_") }}.build{{ buildNumber }}.l10n_artifacts.{{ platform }}.{{ chunk }}
> -            - index.releases.v1.{{ branch }}.latest.{{ product }}.latest.l10n_artifacts.{{ platform }}.{{ chunk }}

hm, why do we sometimes just have index.releases.v1.* in the routes, and other times (see below)..

@@ -100,5 @@
>          priority: "high"
>          retries: 5
>          routes:
> -            - tc-treeherder-stage.v2.{{ branch }}.{{ revision }}.{{ pushlog_id }}
> -            - tc-treeherder.v2.{{ branch }}.{{ revision }}.{{ pushlog_id }}

(continued) and other times we also have tc-treeherder* routes


when should we *not* have tc-treeherder* routes?
(In reply to Jordan Lund (:jlund) from comment #10)
> Comment on attachment 8775747 [details] [diff] [review]

> 
> when should we *not* have tc-treeherder* routes?

ignore this. I found all the answers here: https://github.com/mozilla/releasetasks/pull/184
Attached file github pr
mshal r? just on: routes.yml.tmpl https://github.com/mozilla/releasetasks/pull/196/commits/270214743ff3a59972651faf3b381e46c9e03d10#diff-2fb360990671f23e992b547e58727b28

rail r? whole patch :)

disclaimer: this only adds prefixed routes. it does not change the treeherder ones or the individual task suffix routes. maybe we could do a follow up if you would like to change those?
Attachment #8775747 - Attachment is obsolete: true
Attachment #8790565 - Flags: review?(rail)
Attachment #8790565 - Flags: review?(mshal)
Attachment #8790565 - Flags: review?(rail) → review+
(In reply to Jordan Lund (:jlund) from comment #12)
> Created attachment 8790565 [details] [review]
> github pr
> 
> mshal r? just on: routes.yml.tmpl
> https://github.com/mozilla/releasetasks/pull/196/commits/
> 270214743ff3a59972651faf3b381e46c9e03d10#diff-
> 2fb360990671f23e992b547e58727b28

mshal: sorry, --amend + push forced new shas. so correct url is: https://github.com/mozilla/releasetasks/pull/196/files#diff-2fb360990671f23e992b547e58727b28
Comment on attachment 8790565 [details] [review]
github pr

Added a bunch of comments / questions to the PR. Looks pretty good overall! I just want to make sure we get it the way we want from the start. If my comments are confusing we can try to sort it out on vidyo :)
Attachment #8790565 - Flags: review?(mshal)
Mihai, you wanted to poke something. ;) Can you look at this bug?
Flags: needinfo?(mtabara)
@rail: Sure! today is full of meetings and Italian food in the office :p If tomorrow/friday sounds good enough, feel free to assign it to me and I'll have a look ;)

I'll leave the NI for reference for now.
No rush, at all.
looks like I let this slip. Now that I am back, I can relook at the review comments and let mihai finish nightly stuff :)
Flags: needinfo?(mtabara)
FTR, the plan for Socorro to start polling the TC index for nightlies and releases got put on hold when I discovered that we are beetmoving TC nightly builds to archive.m.o. ie Socorro keeps on looking where it always has.
Priority: -- → P2
(In reply to Jordan Lund (:jlund) from comment #18)
> looks like I let this slip. Now that I am back, I can relook at the review
> comments and let mihai finish nightly stuff :)

more slipping. This isn't in my near radar. Perhaps it is worth thinking about in-tree rather than fixing releasetasks with bugs like this.
Assignee: jlund → nobody
I think we should dup against bug 1425571.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1425571
FWIW, I think I read a blog post that Socorro are using https://mozilla-services.github.io/buildhub/ now.
(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #22)
> FWIW, I think I read a blog post that Socorro are using
> https://mozilla-services.github.io/buildhub/ now.

Not yet, but likely soon (I bet blog post was http://bluesock.org/~willkg/blog/mozilla/socorro_2017.html which points at https://bugzilla.mozilla.org/show_bug.cgi?id=1366301)
You need to log in before you can comment on or make changes to this bug.