1330008 - Manually verify update scenarios for switching nightlies to TaskCluster

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris Cooper [:coop] (he/him)]

We're ready to replace the buildbot-generated nightlies with those generated in TaskCluster (TC) for Linux and Android. Before we do so, we should manually verify a few important update scenarios to make sure we don't strand anyone by accident:

* buildbot nightly -> tc nightly
* TC nightly -> buildbot nightly (in the event of a backout)
* partial updates (for Linux)

Comment 1

[Chris Cooper [:coop] (he/him)]

We're still gated on CoT and switching to the real nightly signing keys, but I'm cc-ing some more people to help flesh out how, exactly, we plan to test this.

Will we need special balrog rules for this?
Can we use existing channels and do manual manipulation of the browser config files, i.e. changing the update channel by hand on disk?

I'm lining up SV testing support, but we need a complete story here if we can expect them to help.

Comment 2

[Aki Sasaki (not active)]

(In reply to Chris Cooper [:coop] from comment #1)
> We're still gated on CoT and switching to the real nightly signing keys,

The latter landed in bug 1330087.

Comment 3

[[e-mail] (:bhearsum)]

I'm told that by next week, we should be in a place where Linux Date nightlies post update information to Balrog prod, and the binaries produced check Balrog prod for updates, so I'll be working on that assumption here.

Given that, testing these scenarios should be quite easy. The mozilla-central nightlies are shipped to the Firefox nightly channel from Balrog prod, and the Date nightlies should be available from the "nightly-date" channel. So this means that:

(In reply to Chris Cooper [:coop] from comment #0)
> * buildbot nightly -> tc nightly

You should be able to test this by installing a mozilla-central Nightly, switching to the "nightly-date" channel, and checking for updates.

> * TC nightly -> buildbot nightly (in the event of a backout)

You should be able to test this by installing a Date Nightly, switching to the "nightly" channel, and checking for updates.

In both scenarios, you'll still need to make sure that the buildid you're checking from is lower than the buildid of the latest nightly from the channel you've swictched to. The easiest way to do this is to install a Nightly from a previous day.

> * partial updates (for Linux)

This might be trickier, and it's a bit out of my wheelhouse. It sounds like you'll want to somehow make Funsize generate extra partials, but against previous builds from a different channel. Eg: tell it to make new partials for date, but use mozilla-central builds as the "from" side. Simon or Rail are probably the best resources for this.

Comment 4

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Adding Ioana and Andrei to make sure this gets tested this week on Linux and Android.

Comment 5

[Andrei Vaida [:avaida]]

(In reply to Ben Hearsum (:bhearsum) from comment #3)
> > * partial updates (for Linux)
> 
> This might be trickier, and it's a bit out of my wheelhouse. It sounds like
> you'll want to somehow make Funsize generate extra partials, but against
> previous builds from a different channel. Eg: tell it to make new partials
> for date, but use mozilla-central builds as the "from" side. Simon or Rail
> are probably the best resources for this.

Thank you for detailing all these steps, Ben! My understanding is that the first two scenarios from Comment 0 / Comment 3 require testing on Android, while the last one is for Desktop. Did I get that right?

Rail, could you help us out please with the last part? The one referring to partial updates on Linux.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Andrei Vaida, QA [:avaida] – please ni? me from comment #5)
> (In reply to Ben Hearsum (:bhearsum) from comment #3)
> > > * partial updates (for Linux)
> > 
> > This might be trickier, and it's a bit out of my wheelhouse. It sounds like
> > you'll want to somehow make Funsize generate extra partials, but against
> > previous builds from a different channel. Eg: tell it to make new partials
> > for date, but use mozilla-central builds as the "from" side. Simon or Rail
> > are probably the best resources for this.
> 
> Thank you for detailing all these steps, Ben! My understanding is that the
> first two scenarios from Comment 0 / Comment 3 require testing on Android,
> while the last one is for Desktop. Did I get that right?

I was only speaking to Desktop in comment #3. I'll defer to others on Android.

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to Andrei Vaida, QA [:avaida] – please ni? me from comment #5) 
> Thank you for detailing all these steps, Ben! My understanding is that the
> first two scenarios from Comment 0 / Comment 3 require testing on Android,
> while the last one is for Desktop. Did I get that right?

Yes, since there are no partials for Android, we don't to test that scenario.

Comment 8

[Andrei Vaida [:avaida]]

(In reply to Ben Hearsum (:bhearsum) from comment #6 and Chris Cooper [:coop] from comment #7)

Got it, thank you both for clarifying. We'll start testing the first two scenarios until we clear out what steps we need to follow for the third one.

Comment 9

[Rail Aliiev [:rail]]

Simon has just deployed a funsize fix (some treeherder related issues), that should make the partials work again.

I'm not sure if we can easily tests date -> nigtly migration via partial, but nightly->nightly should just work fine.

Leaving NI in place to double check this later.

Comment 10

[Chris Cooper [:coop] (he/him)]

We have TaskCluster-generated nightlies in place now for Linux (32-bit and 64-bit) and Android (ARM and x86).

Our own testing has verified that updates are working for all platforms (including a locale: es-CL), but we would still like SV to double-check. If SV could also spot-check a few locales on both linux32 and linux64, that would be appreciated.

On m-c, here are the buildids for the TC nightlies:

Linux (32 and 64):     20170118211214
Android (ARM and x86): 20170118211211

For testing partial updates, one known previous buildid you can update _FROM_ is 20170114030206. That buildid corresponds to the following linux builds:

http://archive.mozilla.org/pub/firefox/nightly/2017/01/2017-01-14-03-02-06-mozilla-central/firefox-53.0a1.en-US.linux-i686.tar.bz2
http://archive.mozilla.org/pub/firefox/nightly/2017/01/2017-01-14-03-02-06-mozilla-central/firefox-53.0a1.en-US.linux-x86_64.tar.bz2

No partials for Android, of course.

We have updates enabled right now, but the are throttled to 10%. We won't be generating new nightlies until after we do more verification tomorrow, but you should be able to update to that first set above.

Mihai: since you'll be up first among all of us tomorrow, can you field questions if required? Thanks.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Sure, am on Romanian timezone (GMT+2) and ready to help if needed.

Comment 12

[Andrei Vaida [:avaida]]

I finished testing this on Ubuntu 16.04 x86 and x64, but I'm seeing failures for the second testcase: 

	nightly → buildbot nightly (in the event of a backout)


I'm not sure if it's something that I'm doing wrong or an actual issue, here's an overview:

	UBUNTU 16.04 LTS (x86_64)
		- 53.0a1 (20170104063905, es-AR): FAILED
			no updates found via 'nightly' channel (appears up to date)

		- 53.0a1 (20170107160103, fr): FAILED
			no updates found via 'nightly' channel (appears up to date)

		- 53.0a1 (20170104160036, ko): FAILED
			no updates found via 'nightly' channel (appears up to date)
	

	UBUNTU 16.04 LTS (x86)
		- 53.0a1 (20170104160036, pl): FAILED
			no updates found via 'nightly' channel (appears up to date)

		- 53.0a1 (20170108160341, es-AR): FAILED
			no updates found via 'nightly' channel (appears up to date)

I've been tracking all my test results in this etherpad: https://public.etherpad-mozilla.org/p/1330008.

Comment 13

[Chris Cooper [:coop] (he/him)]

(In reply to Andrei Vaida, QA [:avaida] – please ni? me from comment #12)
> I finished testing this on Ubuntu 16.04 x86 and x64, but I'm seeing failures
> for the second testcase: 
> 
> 	nightly → buildbot nightly (in the event of a backout)

From IRC:

10:57 
<Callek> coop: ahh yea, we didn't prep any tc-nightly -> buildbot update paths
10:58 
<Callek> first tc-nightly is after the last bb nightly

So what you're seeing is expected, i.e. there's no update rule in place for this, and we should have said as much. Sorry. :(

> https://public.etherpad-mozilla.org/p/1330008.

Fantastic. Thanks for this.

Comment 14

[Chris Cooper [:coop] (he/him)]

Updates are working well. Thanks for the validation help here.