677496 - introduce release candidates

Status: RESOLVED FIXED

(Cloud Services :: Server: Core, defect)

Comment 1

[Tarek Ziadé (:tarek)]

Attachment: patch to the doc that explains the rc process

Comment 2

[Tarek Ziadé (:tarek)]

Attachment: patch to the doc that explains the rc process

Comment 3

[Pete Fritchman [:petef]]

Comment on attachment 551725 [details] [diff] [review]
patch to the doc that explains the rc process

Review of attachment 551725 [details] [diff] [review]:
-----------------------------------------------------------------

::: source/server-devguide/release.rst
@@ -36,0 +36,27 @@
> > +Release candidates
> > +==================
> > +
> > +When a project is about to hit production, it has to pass tests and QA on the
NaN more ...

should be rc1, right?

@@ -36,0 +36,33 @@
> > +Release candidates
> > +==================
> > +
> > +When a project is about to hit production, it has to pass tests and QA on the
NaN more ...

this happens to work well with RPM's version compare algorithm, too.

1.0rc1-1 < 1.0-1 < 1.1rc1-1 < 1.1rc2-1 < 1.1-1

@@ -36,0 +36,54 @@
> > +Release candidates
> > +==================
> > +
> > +When a project is about to hit production, it has to pass tests and QA on the
NaN more ...

maybe "tip" should versions should be "1.5dev-1" when building an RPM, because in RPM 1.5dev-1 < 1.5rc1-1 < 1.5-1

Comment 4

[Toby Elliott [:telliott]]

I'm fine with this so long as tagging can be done to older revs without branching.

I also question a little bit whether this solves much of a problem, because 99% of folks out there are just pulling from tip :P

Comment 5

[Tarek Ziadé (:tarek)]

(In reply to Toby Elliott [:telliott] from comment #4)
> I'm fine with this so long as tagging can be done to older revs without
> branching.

older revs ? what's the use case ?

 
> I also question a little bit whether this solves much of a problem, because
> 99% of folks out there are just pulling from tip :P

They won't anymore with the channels. They will be on the stable versions for all apps by default, because we will be able to ensure that the latest final tag, as opposed to a stage tag or tip, is what we have in prod.

Comment 6

[Toby Elliott [:telliott]]

(In reply to Tarek Ziadé (:tarek) from comment #5)
> (In reply to Toby Elliott [:telliott] from comment #4)
> > I'm fine with this so long as tagging can be done to older revs without
> > branching.
> 
> older revs ? what's the use case ?
> 

I tag something 1.1rc. It gets pushed up to stage for testing.

You check in some code (for 1.2)

1.1rc clears testing. We now tag that point as 1.1. However, hg is annoying and decides that that means we're creating a new branch. My understanding is that it doesn't do this, but I want to verify rather than have to deal with that.


> > I also question a little bit whether this solves much of a problem, because
> > 99% of folks out there are just pulling from tip :P
> 
> They won't anymore with the channels. They will be on the stable versions
> for all apps by default, because we will be able to ensure that the latest
> final tag, as opposed to a stage tag or tip, is what we have in prod.

I don't follow this. I'm saying that the majority of people out there run hg clone http://myproject, then just do pull -u on a regular basis. Tags don't enter into their thinking.

Comment 7

[Tarek Ziadé (:tarek)]

(In reply to Pete Fritchman [:petef] from comment #3)
> Comment on attachment 551725 [details] [diff] [review] [diff] [details] [review]
> patch to the doc that explains the rc process
> 
> Review of attachment 551725 [details] [diff] [review] [diff] [details] [review]:
> -----------------------------------------------------------------
> 
> ::: source/server-devguide/release.rst
> @@ -36,0 +36,27 @@
> > > +Release candidates
> > > +==================
> > > +
> > > +When a project is about to hit production, it has to pass tests and QA on the
> NaN more ...
> 
> should be rc1, right?

c1 or rc1, bith work. In PEP 386 I've defined c1 so we have (a)lpha, (b)eta and (c)andidate - http://www.python.org/dev/peps/pep-0386/

But rc1 works for me as well

> 
> @@ -36,0 +36,33 @@
> > > +Release candidates
> > > +==================
> > > +
> > > +When a project is about to hit production, it has to pass tests and QA on the
> NaN more ...
> 
> this happens to work well with RPM's version compare algorithm, too.
> 
> 1.0rc1-1 < 1.0-1 < 1.1rc1-1 < 1.1rc2-1 < 1.1-1

1.0c1 does not work ? IIRC It does but I might be wrong. The scheme I picked should work on most packaging systems as long as we don't introduce ~

> 
> @@ -36,0 +36,54 @@
> > > +Release candidates
> > > +==================
> > > +
> > > +When a project is about to hit production, it has to pass tests and QA on the
> NaN more ...
> 
> maybe "tip" should versions should be "1.5dev-1" when building an RPM,
> because in RPM 1.5dev-1 < 1.5rc1-1 < 1.5-1

Yeah good point, I did not mention this, will add it

Comment 8

[Tarek Ziadé (:tarek)]

(In reply to Toby Elliott [:telliott] from comment #6)
...
> I tag something 1.1rc. It gets pushed up to stage for testing.
> 
> You check in some code (for 1.2)
> 
> 1.1rc clears testing. We now tag that point as 1.1. However, hg is annoying
> and decides that that means we're creating a new branch. My understanding is
> that it doesn't do this, but I want to verify rather than have to deal with
> that.

It will create an unnamed branch and you will need to force the tagging with -f, merge and probably resolve a conflict on the .hgtags file. But at the end the tags are just a list of hashes in .hgtags, hg uses when you do "hg up TAG"

But yeah, that a little bit twisted.


> > > I also question a little bit whether this solves much of a problem, because
> > > 99% of folks out there are just pulling from tip :P
> > 
> > They won't anymore with the channels. They will be on the stable versions
> > for all apps by default, because we will be able to ensure that the latest
> > final tag, as opposed to a stage tag or tip, is what we have in prod.
> 
> I don't follow this. I'm saying that the majority of people out there run hg
> clone http://myproject, then just do pull -u on a regular basis. Tags don't
> enter into their thinking.

Yeah but once they have an up-to-date clone, they do "make build" with our stuff. In server-full that means they'll get the latest stable. 

In apps like server-reg, which are not meta packages, that means "make build" might update to the latest stable tag. that's not an issue unless you change some files in the code -- in that case you might get some conflict but that's part of the game if you develop/debug in there

Comment 9

[Toby Elliott [:telliott]]

(In reply to Tarek Ziadé (:tarek) from comment #8)

> Yeah but once they have an up-to-date clone, they do "make build" with our
> stuff. In server-full that means they'll get the latest stable. 
> 
> In apps like server-reg, which are not meta packages, that means "make
> build" might update to the latest stable tag. that's not an issue unless you
> change some files in the code -- in that case you might get some conflict
> but that's part of the game if you develop/debug in there

Definitely cleaner with the metapackages.

hmm, maybe we could have a set of metarepos for each channel. So you clone hg://metarepo.beta, then do make sreg, or make sreg_rpm and it builds you the betachannel sreg. That repo is just pointers to our others. Point endusers at those, and we don't need to worry about them pulling tip.

Comment 10

[Tarek Ziadé (:tarek)]

(In reply to Toby Elliott [:telliott] from comment #9)
> (In reply to Tarek Ziadé (:tarek) from comment #8)
> 
> > Yeah but once they have an up-to-date clone, they do "make build" with our
> > stuff. In server-full that means they'll get the latest stable. 
> > 
> > In apps like server-reg, which are not meta packages, that means "make
> > build" might update to the latest stable tag. that's not an issue unless you
> > change some files in the code -- in that case you might get some conflict
> > but that's part of the game if you develop/debug in there
> 
> Definitely cleaner with the metapackages.
> 
> hmm, maybe we could have a set of metarepos for each channel. So you clone
> hg://metarepo.beta, then do make sreg, or make sreg_rpm and it builds you
> the betachannel sreg. That repo is just pointers to our others. Point
> endusers at those, and we don't need to worry about them pulling tip.

Mmm.. this is starting to be overkill imo.

I think if we're starting to add metapackages for all projects so people simply get the right build without worrying about hg, I'd be inclined not to do this but simply to provide source releases in form of non-RPM archives at some place, and even push the stable versions at PyPI.

Then, to install or update to the latest stable:

$ easy_install -U server-reg... FTW

Jenkins can feed a directory with all those tarballs continuously for every channel, so we're sure people get the right version depending on the channel they want.

So at the end it boils down to:

what people do with our stuff ? if they need a stable version, they could simply get the stable tarball we continuously provide.

if they are developers, they need to go back-and-forth between the prod version (with the right versions for its deps) the stage version and the dev version.
I include in developers Ops and QA as we'll all need to update some of our clone to a specific channel.

Comment 11

[Toby Elliott [:telliott]]

(In reply to Tarek Ziadé (:tarek) from comment #10)
> Mmm.. this is starting to be overkill imo.
> 
> I think if we're starting to add metapackages for all projects so people
> simply get the right build without worrying about hg, I'd be inclined not to
> do this but simply to provide source releases in form of non-RPM archives at
> some place, and even push the stable versions at PyPI.

I was thinking more in terms of a single repo that had a makefile in it that let you build the other projects individually, but yeah, providing prebuilt packages somewhere would also be a solution.

Right now, the vast majority of people using our code are not developers (in the sense that they want it to work, not to hack on). It would be nice to encourage more ecosystem there, but making the process clean and safe for someone who just wants to run a sync server has to be our second priority (priority #1 is, of course, making it work well for us :P ).

Comment 12

[Tarek Ziadé (:tarek)]

(In reply to Toby Elliott [:telliott] from comment #11)
..
> I was thinking more in terms of a single repo that had a makefile in it that
> let you build the other projects individually, but yeah, providing prebuilt
> packages somewhere would also be a solution.
> 
> Right now, the vast majority of people using our code are not developers (in
> the sense that they want it to work, not to hack on). It would be nice to
> encourage more ecosystem there, but making the process clean and safe for
> someone who just wants to run a sync server has to be our second priority
> (priority #1 is, of course, making it work well for us :P ).

yeah, we need a good story for both crowds. 

Thinking harder about source releases I think repos are superior here for a non-agnostic installations. The biggest advantage of telling people to "hg pull" our stuff is that they get a self-contained directory that contains, besides the code, all the config files and such things. 

So.. I like the single repo idea, we could call it server-projects and 'register' all our projects there for people to get.

note that this does not change the channels stories and the need to differenciate stage and prod versions : the makefile needs a way to know which tag to pick for the project. Then the requirements file for the channel do the rest.

Comment 13

[Toby Elliott [:telliott]]

I was thinking one repo per channel.

Comment 14

[Tarek Ziadé (:tarek)]

what does several repos bring ? if you have one repo for prod, stage and dev, you still need to decide which tag you will check out for the target project.

or is this so people don't specify which channel to get once they've cloned the repo ? 

can you elaborate with a full example ?

Comment 15

[Toby Elliott [:telliott]]

(In reply to Tarek Ziadé (:tarek) from comment #14)

> or is this so people don't specify which channel to get once they've cloned
> the repo ? 


This, pretty much. If I decide I want to be working in the beta channel, I check out the beta repo, and build from there and continue to build from there until I decide I don't want to be in beta channel.

If there's a use case for being in multiple channels for different projects, we should be able to support that by building from the different repos, but, realistically, I don't know that that's going to happen. Really, we're likely to have 3 classes of people: those who just want the stable versions of everything, those who want to live on the beta channel, and those who are just checking out tip for everything anyway.

Comment 16

[Tarek Ziadé (:tarek)]

I can see the (In reply to Toby Elliott [:telliott] from comment #15)
> (In reply to Tarek Ziadé (:tarek) from comment #14)
> 
> > or is this so people don't specify which channel to get once they've cloned
> > the repo ? 
> 
> 
> This, pretty much. If I decide I want to be working in the beta channel, I
> check out the beta repo, and build from there and continue to build from
> there until I decide I don't want to be in beta channel.
> 
> If there's a use case for being in multiple channels for different projects,
> we should be able to support that by building from the different repos, but,
> realistically, I don't know that that's going to happen. Really, we're
> likely to have 3 classes of people: those who just want the stable versions
> of everything, those who want to live on the beta channel, and those who are
> just checking out tip for everything anyway.

In practice, I find maintaining 3 repos that are basically duplicates (besides one single value, the channel) quite a burden -- unless they use the same build script. So maybe it's simpler to add in MopyTools a script that can build Services apps, like this:

$ easy_install MopyTools
$ services-build-app server-reg --channel=stage /its/here

and

$ services-update-app /its/here --channel=prod
 

But let's focus on the priority first:

We agreed that we need Jenkins to be able to build apps for each channel, and that can happen in the initial project repo. So I am going to go ahead and make this work.

In a second phase, we can work on the higher level to provide the simplest UI for people to build a project on a given channel and stick with it. And maybe make Jenkins use this UI at the end.

Comment 17

[Tarek Ziadé (:tarek)]

Toby: let's talk in the other bug.

I've pushed a branch with two refactored commands

see: https://bugzilla.mozilla.org/show_bug.cgi?id=677344#c8

These commands can work on any "services-like" repo, as long as we have a few things around:
- the reqs files
- a setup.py
- the spec file when it's not a meta-project

The Makefile can still be used, but moving most things into a python command that can be installed as a standalone app, makes it easier to play around in projects. 

Also, one more command we can add is fetchapp, that can fetch an app in our repos (hg clone), and then build it.

Comment 18

[:Atoll]

Comment on attachment 551725 [details] [diff] [review]
patch to the doc that explains the rc process

no useful feedback this time.  seems like the tag names being provided to ops will get more complex, but they're just arbitrary strings to me anyways, so that's still fine.

Comment 19

[James Bonacci [:jbonacci]]

I think this change should be great for QA. Knowing that there is an RC release for Stage is a very good idea. Knowing exactly which RC release (by explicit tagging) will allow us to better track what goes in to weekly trains.

If we are going to be deploying builds tagged like the following to Stage:
rpm-W.XrcY-Z
then I think QA will be most interested in the current rcY that is going to Stage.

We will need to rely on Dev and OPs to advertise (early) the RC release (the explicit rpm tag) that is part of the current train to Stage.
We could do this in each deployment ticket of the current train AND include it in the deployment email that Pete sends out to QA before staging starts.

Thanks.

Comment 20

[Tarek Ziadé (:tarek)]

ok thanks all for the feedback.

I am finishing the release doc and will get back to you for a final review

Comment 21

[Tarek Ziadé (:tarek)]

Attachment: New release process

New version -- still looking at a simpler way to tag rcs, but the document is almost in the final form

Comment 22

[Tarek Ziadé (:tarek)]

Attachment: The releas process in HTML

It's simpler to read the full render than the patch I guess

Comment 23

[Toby Elliott [:telliott]]

Comment on attachment 553797 [details] [diff] [review]
New release process

I like most of this, but think we're missing an opportunity in the release numbers. The most common use of the release bump is that we have a train, and find something trivially wrong with the train (there's a missing item in setup.py, or a bug in .spec, or we find a small logic error, etc). Since we work iteratively to get these up on stage, it seems likely that we'll want to tag those with increased release numbers rather than version numbers. Or are you thinking those are rc bumps? (in which case, why bother with release numbers at all?)

Comment 24

[Tarek Ziadé (:tarek)]

feedback from Atoll om irc:

18:43 <atoll> there's a problem with the Full example
18:44 <atoll> * devs fix the problems and tag 1.4rc1
18:44 <atoll> * another dev resumes work on 1.5.dev1 in tip
18:44 <atoll> which more accurately reflects reality
18:44 <atoll> 1.3 is pushed on stage, it's working
18:44 <atoll> and then someone commits a build system change to the product
18:45 <atoll> and meanwhile we release 1.3 as tagged
18:45 <atoll> and then an issue comes up - we have to fork 1.3 to 1.3.1, because tip is not safe
18:45 <atoll> it's ahead, and contains unrelated changes
18:46 <atoll> that's not reflected here, and since no version number in this document reflects the forked 1.3.1 type 
              versions, it cannot reflect reality

Comment 25

[Tarek Ziadé (:tarek)]

(In reply to Toby Elliott [:telliott] from comment #23)
> Comment on attachment 553797 [details] [diff] [review]
> New release process
> 
> I like most of this, but think we're missing an opportunity in the release
> numbers. The most common use of the release bump is that we have a train,
> and find something trivially wrong with the train (there's a missing item in
> setup.py, or a bug in .spec, or we find a small logic error, etc). Since we
> work iteratively to get these up on stage, it seems likely that we'll want
> to tag those with increased release numbers rather than version numbers. Or
> are you thinking those are rc bumps? (in which case, why bother with release
> numbers at all?)

We can increment the minor version for this, I think we should not worry about the rpm release. From my point of view, it's incremented only when the released is patched and we don't want the patch to make it upstream.

That "extra" number has no existence on the python side.

Comment 26

[JR Conlin [:jrconlin,:jconlin]]

Comment on attachment 553797 [details] [diff] [review]
New release process

For Version Numbers: 

The method I've used in the past is:

Major - Introduces backwards breaking feature/effect. Code using this library/app will need to be examined for damage before upgrading.

Minor - Introduces non-backwards breaking feature/effect. Code using this library/app will NOT need to be examined for damage. 

Release - Indicates minor fix (e.g. resolves bug with existing interfaces.)

E.g. 1.0.0 - Initial library release
1.1.0 introduces new "Bar" interface/function.
1.1.1 resolves bug with Bar (it only Gorped every other time.)
2.0.0 removes Bar interface (since folks had no idea what Gorping meant.)

Comment 27

[Tarek Ziadé (:tarek)]

reworking the doc with all the feedback so far, thanks

Comment 28

[Tarek Ziadé (:tarek)]

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #26)
> Comment on attachment 553797 [details] [diff] [review]
> New release process
> 
> For Version Numbers: 
> 
> The method I've used in the past is:
> 
> Major - Introduces backwards breaking feature/effect. Code using this
> library/app will need to be examined for damage before upgrading.
> 
> Minor - Introduces non-backwards breaking feature/effect. Code using this
> library/app will NOT need to be examined for damage. 
> 
> Release - Indicates minor fix (e.g. resolves bug with existing interfaces.)
> 
> E.g. 1.0.0 - Initial library release
> 1.1.0 introduces new "Bar" interface/function.
> 1.1.1 resolves bug with Bar (it only Gorped every other time.)
> 2.0.0 removes Bar interface (since folks had no idea what Gorpin meant.)

yes that what I use to do too (besides the naming s/release/micro).

Comment 29

[Tarek Ziadé (:tarek)]

Attachment: New version

Comment 30

[Tarek Ziadé (:tarek)]

Attachment: New version (HTML)

Comment 31

[Rob Miller [:rmiller]]

Comment on attachment 553862 [details]
New version (HTML)

First, thanks for putting so much thought into this process, and for such complete documentation.  Clearly defined process FTW!  Overall I think it's coming together very nicely.  Here are my thoughts from perusing the docs, in no particular order:

- Seems a little weird that we have to hand maintain separate <FLAVOR>-reqs.txt files, esp when prod and stage will almost always be identical.  Might be worth thinking about streamlining this a bit, at the very least having stage use the prod reqs file by default but merging in overrides from its own file.

- It's not yet clear to me exactly how channels will be specified and used in all cases.  Do devs have to update the channels in the Makefile when release branches are cut?  I like Toby's suggestion that downstream devs be able to clone a repo url specific to a channel and then never have to worry about it again, are we planning on providing that?  If not, how exactly do I track a channel w/o having to specify the channel on every 'make build' command?

- This doesn't negate any of the process specified in the document (read: this isn't a reason to not roll with this), but there are a couple more places that could be automated.  In particular, all of section 1 ("Start a release branch") could probably be done w/ a single command, possibly even including generation of the version files as specified in step 2, inserting skeleton text in the RELEASE.txt (step 3).

- (tongue-in-cheek, but not really) We should explicitly specify that "MozSvc" is pronounced "Mozz-Vik" so people know how to speak our package names out loud.

- Is the RPM "release" number meant to correspond to the MICRO portion of the deployed version?  If so, can we just change "release" to "micro" to make this more clear.  If not, it's not quite clear from the docs how each of these is to be used; they both seem to be more on the ops side, but I'm unclear on them.

- Similarly, is there any relationship btn the N in "devN" and a MICRO portion of the deployed version number?  If not, it seems weird that "1.7.dev3" might correspond to "1.7.1" while "1.7.dev24" might then correspond to "1.7.3".

That's it for now.  None of these are a big enough deal to not roll with what's specified here, IMO, it's more just ideas for how it could be further refined.

Comment 32

[Tarek Ziadé (:tarek)]

(In reply to Rob Miller [:RaFromBRC] from comment #31)
> - Seems a little weird that we have to hand maintain separate
> <FLAVOR>-reqs.txt files, esp when prod and stage will almost always be
> identical.  Might be worth thinking about streamlining this a bit, at the
> very least having stage use the prod reqs file by default but merging in
> overrides from its own file.

Yeah, fair enough. Maybe stage-reqs.txt can be optional, and if it existing just contains overrides. The tool can then build a list using prod-reqs.txt + stage override.

> 
> - It's not yet clear to me exactly how channels will be specified and used
> in all cases.  Do devs have to update the channels in the Makefile when
> release branches are cut? 

Not really because the channel is 'dev' by defaut when you 'make build' and 'prod' by default when you 'make build_rpms' since we never by default build rpm released for the dev versions,

> I like Toby's suggestion that downstream devs be
> able to clone a repo url specific to a channel and then never have to worry
> about it again, are we planning on providing that?

the only case were we want downstream devs to be on the prod channel in in server-full, otherwise it's always 'dev'. but you can force it otherwise if you want.

>  If not, how exactly do I
> track a channel w/o having to specify the channel on every 'make build'
> command?

what do you mean ? like keeping the last used channel ?

while Makefile forces the option with a variable, you can do:

  $ bin/buildapp -c channel

then the channel becomes sticky, it's written in .channel and the next time you use the command its using that channel again

is that what you want ?



> 
> - This doesn't negate any of the process specified in the document (read:
> this isn't a reason to not roll with this), but there are a couple more
> places that could be automated.  In particular, all of section 1 ("Start a
> release branch") could probably be done w/ a single command, possibly even
> including generation of the version files as specified in step 2, inserting
> skeleton text in the RELEASE.txt (step 3).

I am not a big fan of automation for release notes (people tend to not edit them then), for the branching, yeah why not

> 
> - (tongue-in-cheek, but not really) We should explicitly specify that
> "MozSvc" is pronounced "Mozz-Vik" so people know how to speak our package
> names out loud.

Ok :D also nice t-shirt theme. 


> 
> - Is the RPM "release" number meant to correspond to the MICRO portion of
> the deployed version?  If so, can we just change "release" to "micro" to
> make this more clear.  If not, it's not quite clear from the docs how each
> of these is to be used; they both seem to be more on the ops side, but I'm
> unclear on them.

the rpm release should be completely forgot about. it should just stay to "1". 
I'll try to make it clearer. It used out there to patch the code downstream, before upstream gets fixed (or not)

> 
> - Similarly, is there any relationship btn the N in "devN" and a MICRO
> portion of the deployed version number?  

No 

> If not, it seems weird that
> "1.7.dev3" might correspond to "1.7.1" while "1.7.dev24" might then
> correspond to "1.7.3".

1.7.dev3 does not correspond to 1.7.1. and 1.7.dev24 not to 1.7.3.

the ordering is:

1.7.dev3 < 1.7.dev24 < 1.7.1 < 1.7.3.


> 
> That's it for now.  None of these are a big enough deal to not roll with
> what's specified here, IMO, it's more just ideas for how it could be further
> refined.

thanks for the feedback, I'll apply what I can now, push a doc and add relevant bugs for improvements when it makes sense,

Comment 33

[Tarek Ziadé (:tarek)]

Pushed at https://hg.mozilla.org/services/docs/rev/c2fef7f3c2ad

Thanks all !

Created Bug 680360 for improvements

Comment 34

[JR Conlin [:jrconlin,:jconlin]]

Comment on attachment 553862 [details]
New version (HTML)

clearing old request