801711 - [OTA update] Set up 'promoted_to_stable' folder at https://releases.mozilla.com/b2g/

Status: VERIFIED FIXED

(Firefox OS Graveyard :: General, defect, P1)

Description

[Lukas Blakk [:lsblakk] use ?needinfo]

Until we get onto Releng automation and are able to do builds based on TBPL results & green checkins here's my proposal for what our Stable channel should be:

* Nightly & Stable-Test and Stable builds are built nightly from mozilla-aurora
* All builds run through smoke tests to see if results are 100% 'green' (all pass or whatever measurement we're using)
* If pass:
	Push Stable-Test updates
	QA tests update on Stable-Test channel, confirms no regressions in updating between Stable-Test builds (check basic functionality?)
        If QA are OK with the Stable-Test builds:
	          Next Day: Stable channel updates published to Stable update channel
* If fail:
	No Stable build today, try again tomorrow

This gives a 24 hour window between when Nightly builds are pushed out to developers & QA as well to buy time in case there is a busted build we want to spare our test users from.

Comment 1

[Jonathan Griffin (:jgriffin)]

In thinking about this more, I'm not sure there's a need for a Stable-Test channel/build.  The only difference in this build from nightly would be the name of the update channel ('stable-test' vs 'nightly').

If we don't publish a stable update for X days, QA can still test the ability to update from an older nightly to the current one, which I believe is the intention behind the Stable-Test idea.

E.g., there's no Stable update on Nov 1 or 2 because of bustage.
On Nov 3, QA could test an Oct 31 nightly with the Nov 3 nightly update to verify that the update path between builds of those dates works. If it does, then we could promote the Nov 3 nightly to the stable update channel.

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

Good point. I guess my only hesitation here is that by not publishing the updates to an intermediate location (Stable-Test) then we have to keep track by some other means what the most up to date Stable channel build is.  Do you know of another easy way to do this?

Comment 3

[Jonathan Griffin (:jgriffin)]

The Jenkins instance we use for making these builds keeps track of which nightly builds were promoted to stable, so I can see for instance, that the most recent unagi_stable build was created from this nightly build using this manifest:

http://builder1.ateam.phx1.mozilla.com/job/build-unagi/lastSuccessfulBuild/artifact/b2g_update_source_2012-10-14_074710.xml (requires MPT)

We can use this information to help QA determine which nightly they need to use for update testing, in the cases where we skip days between stable updates, or in cases where we spin multiple nightlies in a single day.

I'm not sure that having a Stable-Test channel would improve this any.  There is always the possibility that we would not promote a Stable-Test to Stable, due to bugs found late in the cycle, and so we still couldn't use Stable-Test as a means of identifying what the most recent Stable channel build is.

Or am I missing something?

Comment 4

[Andreas Gal :gal]

What is the status here. We need a decision on this end of business today.

Comment 5

[Jonathan Griffin (:jgriffin)]

I'd suggest the following:

We'll create a folder at https://releases.mozilla.com/b2g/ named 'promoted_to_stable'.  In this folder, we'll copy each nightly build that is promoted to a stable build, as it is promoted.

When QA needs to test update tests, they should use the most recent build in this folder.  This will allow them to test the exact same update path that dogfood testers will be using, and avoids any confusion about which build they should use.

:lsblakk, does this sound reasonable?

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

Sounds good - and gives us a nice history of every build that gets promoted in a public-ish place.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Any way to make copying the promoted builds to that dir something QA can do with a push-button interface would probably help out process here too.

Comment 8

[Jonathan Griffin (:jgriffin)]

This is done.  Each time a build is promoted from nightly to stable, the nightly build is copied to https://releases.mozilla.com/b2g/promoted_to_stable/.

(In reply to Lukas Blakk [:lsblakk] from comment #7)
> Any way to make copying the promoted builds to that dir something QA can do
> with a push-button interface would probably help out process here too.

The process is done via a simple web UI, but it requires access to MPT VPN.  I'll document this for QA.

Comment 9

[Tony Chung [:tchung]]

Verified fix and location.