904868 - run a staging release off mozilla-aurora ~two weeks previous to the 25.0b1 merge

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Comment 1

[Aki Sasaki (not active)]

We will probably have to create a user repo that makes the changes that relman would usually make on merge day.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

cc-ing akeybl/bhavana/lsblakk. Specifically, any changes you approve for landing on aurora *after* this work starts, and before migration starts, should be cross-checked to make sure it does not risk impacting beta builds (cue: flashbacks to FF24.0b1).

From postmortem meeting, we selected ~2weeks before migration because:
* it was close enough to migration to be useful
* it was far enough from migration that there'd be time to find/backout/fix any breakages discovered in this staging build.

Possible extra goodness: after migration and before go-to-build-for-FF25.0beta1, it might be possible to compare this staging environment with the post-migration live environment as a sanity check for any changes getting missed in migration. Details TBD.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

We probably want to track this, but flagging for our relman triage so we can discuss.

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

Just for peace of mind - would we then consider the build configs 'frozen' from the time of this staging build until the completed builds of the 25.0b1 release?

Comment #2 makes this sounds like the staging run is to prevent errors from changes landing in the firefox repo but the failures for 24.0b1 were from build config changes to the repack process - 2 weeks before migration will result in a lot of changes to mozilla-aurora repo so this might need to be done within a day or two of the real migration.

Comment 5

[Aki Sasaki (not active)]

(In reply to [e-mail] [:lsblakk] from comment #4)
> Just for peace of mind - would we then consider the build configs 'frozen'
> from the time of this staging build until the completed builds of the 25.0b1
> release?

Since the build configs in question were in-tree, run by :gps' team, these changes would be subject to relman aurora landing approval.

So yes, we can consider them frozen.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to [e-mail] [:lsblakk] from comment #4)
> Just for peace of mind - would we then consider the build configs 'frozen'
> from the time of this staging build until the completed builds of the 25.0b1
> release?
> 
> Comment #2 makes this sounds like the staging run is to prevent errors from
> changes landing in the firefox repo but the failures for 24.0b1 were from
> build config changes to the repack process - 2 weeks before migration will
> result in a lot of changes to mozilla-aurora repo so this might need to be
> done within a day or two of the real migration.

My understanding of this is that we wanted build configs frozen from 2 weeks before until we ship 25.0b1 -- that is to say, that build config changes should only land prior to that. Landing build config changes after the freeze date invalidates our staging run. If 2 weeks is too far in advance to freeze we could probably shorten that a bit (1.5w?).

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Ben Hearsum [:bhearsum] from comment #6)
> (In reply to [e-mail] [:lsblakk] from comment #4)
> > Just for peace of mind - would we then consider the build configs 'frozen'
> > from the time of this staging build until the completed builds of the 25.0b1
> > release?
> > 
> > Comment #2 makes this sounds like the staging run is to prevent errors from
> > changes landing in the firefox repo but the failures for 24.0b1 were from
> > build config changes to the repack process - 2 weeks before migration will
> > result in a lot of changes to mozilla-aurora repo so this might need to be
> > done within a day or two of the real migration.
> 
> My understanding of this is that we wanted build configs frozen from 2 weeks
> before until we ship 25.0b1 -- that is to say, that build config changes
> should only land prior to that. Landing build config changes after the
> freeze date invalidates our staging run. If 2 weeks is too far in advance to
> freeze we could probably shorten that a bit (1.5w?).

Thanks Aki & Ben, that matches my assumption - and helps confirm the other issue as well.  What RelMan can do then to help out is make sure to not approve any in-tree build config changes after the 'freeze' on that area.  I'll add a reminder of this to our release calendar & notes on releasing.

Comment 8

[Massimo Gervasini [:massimo]]

It's running now, but there is an error in repack 6/6: it fails on language ta-LK: bug 913470.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

What's the latest status ?

Comment 10

[Massimo Gervasini [:massimo]]

After re triggering the failed repacks, everything went smoothly except:

* win32 update_verify because of signatures:

  ERROR: Error verifying signature. 
  ERROR: Not all signatures were verified.

but this was expected in staging

* some linux64 verify_updates failed too because
  http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/firefox/nightly/25.0b1-candidates 
  got deleted from dev-stage01 while the job was running. Again I think this is related to the staging environment and it's not a real issue.

Comment 11

[Armen [:armenzg]]

This is not specifically related to the merge day.