924503 - Turn off the B2G builds marked as REMOVE on https://wiki.mozilla.org/User:Asasaki:B2GBuilds -- proceed turning off Unagi per :stephend

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

We've been asking for a list of b2g builds we can turn off; we've gotten that list.

_ QA signoff
_ WebQA signoff
_ ATeam signoff
_ patch(es)
_ review
_ land
_ reconfig

Comment 1

[Aki Sasaki (not active)]

:tchung, :stephend, :jgriffin - does this list of REMOVE builds look ok?
(from b2g eng managers)
https://wiki.mozilla.org/User:Asasaki:B2GBuilds

Comment 2

[Tony Chung [:tchung]]

(In reply to Aki Sasaki [:aki] from comment #1)
> :tchung, :stephend, :jgriffin - does this list of REMOVE builds look ok?
> (from b2g eng managers)
> https://wiki.mozilla.org/User:Asasaki:B2GBuilds

I'm good with this list.   :stephend will make the final edits and reply back here.

Comment 3

[Jonathan Griffin (:jgriffin)]

This list implies that we're turning off all testing on mozilla-b2g18/mozilla-b2g18-v1.1.0hd, which is fine if that's the agreement with the b2g engineering managers.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Aki Sasaki [:aki] from comment #1)
> :tchung, :stephend, :jgriffin - does this list of REMOVE builds look ok?
> (from b2g eng managers)
> https://wiki.mozilla.org/User:Asasaki:B2GBuilds

LGTM, with the following caveat:

If we kill Unagi on master/Aurora for the engineering builds, before I can get Inaris to our Softvision contractors, that will severely limit/kill their Gaia UI Tests involvement.  If you would, can we kill/remove Unagi on master/Aurora, last?  Thanks!

Comment 5

[Aki Sasaki (not active)]

Per last week's Tuesday B2G meeting, we want to keep some of these old build binaries around as well (possibly as little as the final build of each type).

We want to keep unagi on m-c/aurora for last, per comment 4.

We want to keep emulator builds on 1.1hd for now, since turning them off would turn off automated testing.  We may get confirmation on this later.

Comment 6

[Aki Sasaki (not active)]

Attachment: bug924503 part 1

It's another for loop, which I don't like.
But it lets us turn these back on quickly if we need.  Come December we'll actually tear out all the 1.0.1 stuff completely, and the configs will be slightly less ugly.

I'm leaving unagi/unagi-eng on aurora+central for now.
I'm also leaving b2g18_v1_1_0_hd emulator builds, for automation testing.

Comment 7

[Aki Sasaki (not active)]

Attachment: interdiff

Builder interdiff, for a build master.

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 814678 [details] [diff] [review]
bug924503 part 1

woohoo!

Comment 9

[Ed Morley [:emorley]]

(Have CCed the sheriffs alias so we don't end up thinking builds have gone missing :-))

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 814678 [details] [diff] [review]
bug924503 part 1

https://hg.mozilla.org/build/buildbot-configs/rev/707b23430c3d

Comment 11

[John Hopkins (:jhopkins)]

in production

Comment 12

[Aki Sasaki (not active)]

X QA signoff
X WebQA signoff
X ATeam signoff
X Verify we want automated testing off on 1.1 and 1.1hd
** We kept emulator builds on 1.1hd for automated tests.
X patch(es)
X review
X land
X reconfig
X update https://wiki.mozilla.org/User:Asasaki:B2GBuilds
_ wait for WebQA to get Inaris to Softvision  <-- WE ARE HERE
_ patch to turn off unagi on m-c and m-a
_ review
_ land
_ reconfig
_ update https://wiki.mozilla.org/User:Asasaki:B2GBuilds
_ RESO FIXED

Setting needinfo? for :stephend, so we know when to turn off the unagi builds on m-c and m-a.

Comment 13

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Aki Sasaki [:aki] from comment #12)
> X QA signoff
> X WebQA signoff
> X ATeam signoff
> X Verify we want automated testing off on 1.1 and 1.1hd
> ** We kept emulator builds on 1.1hd for automated tests.
per meeting w/clee, tchung just now, we now officially *do* need to keep these emulator builds on v1.1HD. I've updated https://wiki.mozilla.org/User:Asasaki:B2GBuilds to match new reality.

> X patch(es)
> X review
> X land
> X reconfig
> X update https://wiki.mozilla.org/User:Asasaki:B2GBuilds
> _ wait for WebQA to get Inaris to Softvision  <-- WE ARE HERE
> _ patch to turn off unagi on m-c and m-a
> _ review
> _ land
> _ reconfig
> _ update https://wiki.mozilla.org/User:Asasaki:B2GBuilds
> _ RESO FIXED
> 
> Setting needinfo? for :stephend, so we know when to turn off the unagi
> builds on m-c and m-a.

Comment 14

[Aki Sasaki (not active)]

So you're ready for these to be turned off now?

I can write the patch shortly.

Comment 15

[Aki Sasaki (not active)]

Hm, now I'm wondering if this was intentional.  Was it?
Since that happened at the same time as marking bug 932381 as blocking, this might have been inadvertent...

Comment 16

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Aki Sasaki [:aki] from comment #15)
> Hm, now I'm wondering if this was intentional.  Was it?
> Since that happened at the same time as marking bug 932381 as blocking, this
> might have been inadvertent...

Yes, sorry: intentional.  We need to ensure that the in-country Marketplace Payments testers (third-party contractors) are set up properly with ezboot on Inari, since they're moving off Unagis, as our (Web QA) final dependency to moving off Unagis, wholesale.

Comment 17

[Aki Sasaki (not active)]

Attachment: unagi-configs.diff

Comment 18

[Aki Sasaki (not active)]

Attachment: unagi-mozharness.diff

Comment 19

[Aki Sasaki (not active)]

Attachment: unagi-tools.diff

Comment 20

[Aki Sasaki (not active)]

Sorry, still not clear.

Are you saying that yes, it's intentional that you cleared the needinfo flag (which you have re-set), and you want the Unagi builds off now?
The dependency seems like you may want us to wait.

For clarity's sake, could you say the equivalent of "Please turn off the Unagi builds now" when you're ready?  I'll set the review flags at that point, and we can land+reconfig+update the wiki and resolve this bug.

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 826371 [details] [diff] [review]
unagi-mozharness.diff

also removing old otoro references.

Comment 22

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Aki Sasaki [:aki] from comment #20)
> Sorry, still not clear.
> 
> Are you saying that yes, it's intentional that you cleared the needinfo flag
> (which you have re-set), and you want the Unagi builds off now?
> The dependency seems like you may want us to wait.

Apologies; the dependency I added -- bug 932381 -- needs to be fixed and its replacement in-use by third-party contractors, before I can sign off.

> For clarity's sake, could you say the equivalent of "Please turn off the
> Unagi builds now" when you're ready?  I'll set the review flags at that
> point, and we can land+reconfig+update the wiki and resolve this bug.

Yes, I can + will, sorry again for the confusion!

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 826371 [details] [diff] [review]
unagi-mozharness.diff

>diff --git a/configs/b2g/releng-otoro.py b/configs/b2g/releng-otoro.py

Might be worth renaming this to something more descriptive, at some point when we have cycles for such niceties.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 826372 [details] [diff] [review]
unagi-tools.diff

seems fine *stamp*

Comment 25

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Stephen Donner [:stephend] from comment #22)
> (In reply to Aki Sasaki [:aki] from comment #20)
> > Sorry, still not clear.
> > 
> > Are you saying that yes, it's intentional that you cleared the needinfo flag
> > (which you have re-set), and you want the Unagi builds off now?
> > The dependency seems like you may want us to wait.
> 
> Apologies; the dependency I added -- bug 932381 -- needs to be fixed and its
> replacement in-use by third-party contractors, before I can sign off.
> 
> > For clarity's sake, could you say the equivalent of "Please turn off the
> > Unagi builds now" when you're ready?  I'll set the review flags at that
> > point, and we can land+reconfig+update the wiki and resolve this bug.
> 
> Yes, I can + will, sorry again for the confusion!

sdonner: gentle ping about turning off unagi builds? I see all dep bugs now resolved, including bug#932381, hence the ask.

Comment 26

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to John O'Duinn [:joduinn] from comment #25)
> (In reply to Stephen Donner [:stephend] from comment #22)
> > (In reply to Aki Sasaki [:aki] from comment #20)
> > > Sorry, still not clear.
> > > 
> > > Are you saying that yes, it's intentional that you cleared the needinfo flag
> > > (which you have re-set), and you want the Unagi builds off now?
> > > The dependency seems like you may want us to wait.
> > 
> > Apologies; the dependency I added -- bug 932381 -- needs to be fixed and its
> > replacement in-use by third-party contractors, before I can sign off.
> > 
> > > For clarity's sake, could you say the equivalent of "Please turn off the
> > > Unagi builds now" when you're ready?  I'll set the review flags at that
> > > point, and we can land+reconfig+update the wiki and resolve this bug.
> > 
> > Yes, I can + will, sorry again for the confusion!
> 
> sdonner: gentle ping about turning off unagi builds? I see all dep bugs now
> resolved, including bug#932381, hence the ask.

Just today, I sent out the final (approved) Inari device to Intertek, to test in-country Payments; there's a meeting tomorrow to hash out how we can go forward for other third-party contractors WRT Inaris + other devices, but we should be good to switch off Unagi pretty soon.

cc:ing Edwin, whom I believe might still have a contractor, too, using an Unagi, for the time-being.

Comment 27

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Stephen Donner [:stephend] from comment #26)
> (In reply to John O'Duinn [:joduinn] from comment #25)
> > (In reply to Stephen Donner [:stephend] from comment #22)
> > > (In reply to Aki Sasaki [:aki] from comment #20)
> > > > Sorry, still not clear.
> > > > 
> > > > Are you saying that yes, it's intentional that you cleared the needinfo flag
> > > > (which you have re-set), and you want the Unagi builds off now?
> > > > The dependency seems like you may want us to wait.
> > > 
> > > Apologies; the dependency I added -- bug 932381 -- needs to be fixed and its
> > > replacement in-use by third-party contractors, before I can sign off.
> > > 
> > > > For clarity's sake, could you say the equivalent of "Please turn off the
> > > > Unagi builds now" when you're ready?  I'll set the review flags at that
> > > > point, and we can land+reconfig+update the wiki and resolve this bug.
> > > 
> > > Yes, I can + will, sorry again for the confusion!
> > 
> > sdonner: gentle ping about turning off unagi builds? I see all dep bugs now
> > resolved, including bug#932381, hence the ask.
> 
> Just today, I sent out the final (approved) Inari device to Intertek, to
> test in-country Payments; there's a meeting tomorrow to hash out how we can
> go forward for other third-party contractors WRT Inaris + other devices, but
> we should be good to switch off Unagi pretty soon.

sdonner: are you now ok if we drop the axe on these builds?


> cc:ing Edwin, whom I believe might still have a contractor, too, using an
> Unagi, for the time-being.

ewong: do you have a dependency on these builds? If so, how quickly can you switch to another supported device?

Comment 28

[Stephen Donner [:stephend] Not actively reading bugmail]

John, my extended (both Softvision + Intertek, in-country Marketplace Payments tests) + in-house team are good; thanks -- we're all on Inari and Hamachis, now.  From my end, no more need for Unagi builds!

Comment 29

[Edwin Wong [:edwong]]

I'm good with discontinuing unagi builds.  It's time for everyone to upgrade!

Comment 30

[Aki Sasaki (not active)]

Comment on attachment 826371 [details] [diff] [review]
unagi-mozharness.diff

https://hg.mozilla.org/build/mozharness/rev/d163222e0366

Comment 31

[Aki Sasaki (not active)]

Comment on attachment 826372 [details] [diff] [review]
unagi-tools.diff

https://hg.mozilla.org/build/tools/rev/a374a287890d

Comment 32

[Aki Sasaki (not active)]

Comment on attachment 826370 [details] [diff] [review]
unagi-configs.diff

Unbitrotted + landed: https://hg.mozilla.org/build/buildbot-configs/rev/2881515a3e33

Now we need a merge+reconfig and we're done.

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

What sort of retention policy should we have on the builds that have been disabled ? Should they be archived to tape/archived to Amazon S3/deleted outright ?

Comment 34

[Aki Sasaki (not active)]

I merged the mozharness portion into production.

(In reply to Nick Thomas [:nthomas] from comment #33)
> What sort of retention policy should we have on the builds that have been
> disabled ? Should they be archived to tape/archived to Amazon S3/deleted
> outright ?

I'm not entirely sure, but I know people wanted to keep at least the most recent builds.  Archiving to S3 makes sense, but would be dependent on bug 922889.

Comment 35

[Aki Sasaki (not active)]

... which is resolved ok.  To the cloud!!!

Comment 36

[Aki Sasaki (not active)]

I'm going to update the b2gbuilds wiki page, and we'll need to archive builds.
The builds are all off, though.

Comment 37

[Chris Cooper [:coop] (he/him)]

In production.