1055305 - [Flame] Create Kitkat based pvtbuilds for 2.0 and 2.1

Status: VERIFIED FIXED

(Release Engineering :: General, defect)

Description

[Tony Chung [:tchung]]

[Blocking Requested - why for this release]:

Now that we have a kitkat base image (1.4), for the Flame from T2M, we would like to generate pvtbuilds for Firefox OS engineering/QA to use.   Requesting both User and Engineering based images.

The link to base image v165 is at: https://drive.google.com/a/mozilla.com/file/d/0B7_yuXt7x6y4NE95TXVzY1NJbHc/edit?usp=sharing

I'd like to just have supported Gecko and Gaia nightly builds for 2.0 and 2.1 (aka Master/m-c) available so we can shallow flash over the base image.

Viral, Mwu, Nthomas, Catlee, what needs to be done to proceed forward?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Looks like flame-kk.xml is the manifest to use, are we done stabilizing that ? We would need devs to port that to the 2.0 branch to do builds there.

We (RelEng) don't need to sync any new repos for master, so something like apply base image to flame, use build system to pull backup, upload backup to tooltool; add in-tree files; add config to buildbot; test in staging; deploy. (From https://wiki.mozilla.org/ReleaseEngineering/How_To/Create_new_B2G_build)

Comment 2

[Tony Chung [:tchung]]

(In reply to Nick Thomas [:nthomas] from comment #1)
> Looks like flame-kk.xml is the manifest to use, are we done stabilizing that
> ? We would need devs to port that to the 2.0 branch to do builds there.
> 
> We (RelEng) don't need to sync any new repos for master, so something like
> apply base image to flame, use build system to pull backup, upload backup to
> tooltool; add in-tree files; add config to buildbot; test in staging;
> deploy. (From
> https://wiki.mozilla.org/ReleaseEngineering/How_To/Create_new_B2G_build)

Viral, mwu, are finalized on flame-kk.xml yet?   we are looking for next steps to get mozilla builds generated.

Comment 3

[Michael Wu [:mwu]]

Not ready yet.

Comment 4

[Chris AtLee [:catlee]]

Do we need builds for both flame.xml and flame-kk.xml on the same branch?

If not, we could just rename flame-kk.xml to flame.xml when ready.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Good question! We'd need to update https://hg.mozilla.org/integration/b2g-inbound/file/default/b2g/config/flame/releng-flame.tt too, for the new base image. People would need to flash again to pick up the base image ? Would an OTA of kitkat gecko+gaia continue to work fine on jb base ?

Comment 6

[Michael Wu [:mwu]]

OTA of kitkat gecko is not expected to work on JB base.

Comment 7

[:shell escalante]

Also note: we need the latest ADSP builds possible, as fixes are going in there which impact webRTC testing.  We also need to know what version from the readme notes are going in the builds (so we can know for testing if we should be seeing fixes).

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Tony Chung [:tchung] from comment #0)
> The link to base image v165 is at:
> https://drive.google.com/a/mozilla.com/file/d/0B7_yuXt7x6y4NE95TXVzY1NJbHc/
> edit?usp=sharing

Could someone either convert this to a tarball of backup-flame/, or confirm that 
  https://github.com/mozilla-b2g/device-flame/blob/master/extract-files.sh
will pull the right files if I use it (after flashing the image on my device).

Comment 9

[viral [:viralwang]]

Hi Michael,

I think we could prepare the 2.0 build after bug 1052865 (SIM card can't detect) fix.
Could you please help to create the branch on https://github.com/mozilla-b2g/device-flame?
I'm thinking we should have the branch like "kitkat-v2.0" and "kitkat-v1.4".
Thank you!

Comment 10

[Tony Chung [:tchung]]

(In reply to viral [:viralwang] from comment #9)
> Hi Michael,
> 
> I think we could prepare the 2.0 build after bug 1052865 (SIM card can't
> detect) fix.
> Could you please help to create the branch on
> https://github.com/mozilla-b2g/device-flame?
> I'm thinking we should have the branch like "kitkat-v2.0" and "kitkat-v1.4".
> Thank you!

hi viral, are you saying this bug is blocked by bug 1052865 landing first?   just clarifying what are next steps here and ETA on builds.

Comment 11

[viral [:viralwang]]

Attachment: Bug 1055305 - create flame-kk for BRANCH=v2.0

Hi Seinlin,

Could you please review the PR for branch v2.0 flame-kk?
I think we can keep device-flame same as we use for master branch.

Comment 12

[viral [:viralwang]]

Hi Michael,

I think we don't need the branch in comment 9 since v2.0 can share the master branch.
So far I'm not sure we should support v1.4 branch or not.
remove the ni flag first.
Thank you.

Comment 13

[viral [:viralwang]]

(In reply to Tony Chung [:tchung] from comment #10)
> (In reply to viral [:viralwang] from comment #9)
> > Hi Michael,
> > 
> > I think we could prepare the 2.0 build after bug 1052865 (SIM card can't
> > detect) fix.
> > Could you please help to create the branch on
> > https://github.com/mozilla-b2g/device-flame?
> > I'm thinking we should have the branch like "kitkat-v2.0" and "kitkat-v1.4".
> > Thank you!
> 
> hi viral, are you saying this bug is blocked by bug 1052865 landing first?  
> just clarifying what are next steps here and ETA on builds.

Hi Tony,

bug 1052865 already has patches with r+, I think we can provide the manifest for v2.0 now.
Thank you for your reminder.

Comment 14

[Kai-Zhen Li [:seinlin][:kli]]

Comment on attachment 8479603 [details] [review]
Bug 1055305 - create flame-kk for BRANCH=v2.0

Hi Viral, I suggest you to file a new bug such as add flame-kk manifest for v2.0 branch and which blocks this bug. Let's review the manifest in new bug then. Thanks!

Comment 15

[viral [:viralwang]]

Use bug 1059092 to create the flame-kk.xml manifest for v2.0

Comment 16

[Tony Chung [:tchung]]

Hi nick, Catlee

according to Viral, this would be your steps to generate the 2.0 and 2.1 KK images.  it's ready to go now since 1059092 has just merged:

=> "BRANCH=v2.0 ./config.sh flame-kk && ./build.sh" should be fine, please make sure we use base image v165 for backup-flame folder

Let us know what else releng needs.   Looking forward to those builds soon!

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Tony, can we deprecate the flame builds and just have flame-kk ?

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozharness] b2b_bumper config

This says we don't need to mirror any new repos down to git.mozilla.org.

Comment 19

[Tony Chung [:tchung]]

(In reply to Nick Thomas [:nthomas] from comment #17)
> Tony, can we deprecate the flame builds and just have flame-kk ?

not yet, id like to give the flame-kk images a shot before we move off of flame-jb.   also, the flame-kk image has not been widespread yet, so many engineers and community still are using it.

but i certainly agree we should have a EOL plan. 

Thanks,
Tony

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

I tried to create backup-flame/ without having to pull all the source in, using a flame flashed with the v165 base, then 
 mkdir -p device/tcm
 cd device/tcm
 git clone https://github.com/mozilla-b2g/device-flame flame
 cd flame
 ./extract-files.sh

But it ends up with
 [lots of files pulled from /system]
 pull: /system/system.ver -> ../../../backup-flame/system/system.ver
 pull: /system/build.prop -> ../../../backup-flame/system/build.prop
 1573 files pulled. 0 files skipped.
 4872 KB/s (269865462 bytes in 54.082s)
 Pulling files from ../../../backup-flame
 Pulling "libalsa-intf.so"
 cp: ../../../backup-flame/system/lib/libalsa-intf.so: No such file or directory
 Failed to pull libalsa-intf.so. Giving up.

Could you please advise if I'm doing something incorrectly, or provide a backup-flame.tar.xz from the v165.

Comment 21

[viral [:viralwang]]

Hi Nick,

Please assign the branch as follow command:
git clone https://github.com/mozilla-b2g/device-flame flame --branch kitkat

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Thanks viral, I must have missed that in the manifest. 

I've extracted the files and uploaded them to tooltool. A little further down the track b2g/config/flame-kk/releng-flame-kk.tt should look like this:
[
{
"size": 118323840, 
"digest": "f18aabdf19ce8b630d9724ecf7d3a130e6b020a9e4d4e222bba6d4b3d4303c30dc3c5162a55b0f85de926077be59e17cfe49eb7314dca7e1bb5741a260148083", 
"algorithm": "sha512", 
"filename": "backup-flame.tar.xz"
}
]

Comment 23

[Michael Shal [:mshal]]

Attachment: [m-c] flamekk

Differences from flame config:

$ diff flame/releng-flame.tt flame-kk/releng-flame-kk.tt 
2,3c2,3
< {"size": 149922032,
< "digest": "8d1a71552ffee561e93b5b3f1bb47866592ab958f908007c75561156430eb1b85a265bfc4dc2038e58dda0264daa9854877a84ef3b591c9ac2f1ab97c098e61e",
---
> {"size": 118323840,
> "digest": "f18aabdf19ce8b630d9724ecf7d3a130e6b020a9e4d4e222bba6d4b3d4303c30dc3c5162a55b0f85de926077be59e17cfe49eb7314dca7e1bb5741a260148083",


$ diff flame/config.json flame-kk/config.json
3c3
<     "tooltool_manifest": "releng-flame.tt",
---
>     "tooltool_manifest": "releng-flame-kk.tt",
34c34
<     "b2g_manifest": "flame.xml",
---
>     "b2g_manifest": "flame-kk.xml",

Comment 24

[Michael Shal [:mshal]]

Attachment: [buildbot-configs] buildbot_flamekk

There's a few things I'm not sure about here:

1) Are the enable_nightly lines needed? It sounds like from #c1 maybe not

2) Do the flame-kk and flame-kk_eng platforms need to go in this for loop?

# b2g 1.4+
for name, branch in items_before(BRANCHES, 'gecko_version', 30):
    for p in ('flame', 'flame_eng', 'linux64_gecko-debug',
              'macosx64_gecko-debug', 'linux32_gecko-debug', 'win32_gecko-debug',
              'emulator-kk', 'emulator-kk-debug'):

Comment 25

[Michael Shal [:mshal]]

bhearsum, is there a concern here that the update channel for flame and flame-kk might get mixed up? How do I verify that they are separate?

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

re updates, perhaps we should be submitting into balrog with flame-kk (this may just work, haven't checked), and devices querying with %PRODUCT% of flame-kk (harder!). In the meantime we could leave nightly builds off.

Comment 27

[Michael Shal [:mshal]]

Attachment: [buildbot-configs] buildbot_flamekk

Turning nightlies off per #c26. See also my question 2 in #c24

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

I flashed mshal's eng build from staging onto my flame, and it queried for updates with this:
 https://aus4.mozilla.org/update/3/B2G/34.0a1/20140828163852/flame/en-US/default/Boot2Gecko%202.1.0.0-prerelease/default/default/update.xml?force=1

The .../flame/en-US/default/... is the relevant part, because it's the same as the existing flame jellybean builds. default is normal for a non-nightly, the in tree patch would have it set to 'nightly' - same as JB again. So definitely work to do here, as we can't mix updates between JB and KK.

Comment 29

[Chris AtLee [:catlee]]

Comment on attachment 8480893 [details] [diff] [review]
[buildbot-configs] buildbot_flamekk

Review of attachment 8480893 [details] [diff] [review]:
-----------------------------------------------------------------

probably need to adjust try slaves below as well?

Comment 30

[Alexander Stadler]

Hi!

I've built several 2.0 betas on various platforms in the last weeks.

As its short before release and the manifest got updated I thought its time for a -kk build.

But unfortunately I'm not able to create a new "backup-flame" with the now needed files because I have no access to a v165 image or something similar (or the proprietary sources..).

I'm not authorized to access bug #1019011 (https://bugzilla.mozilla.org/show_bug.cgi?id=1019011) so that I could add my name there for access to
https://secure.pub.build.mozilla.org/tooltool/pvt/build .

tooltool-uploads.pub.build.mozilla.org propably is only for uploads
http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512 seems private.

No login for
https://intranet.mozilla.org/B2G_Team/Flame
as for
https://pvtbuilds.mozilla.org/pvt/(mozilla.org/b2gotoro/tinderbox-builds/b2g-inbound-flame-eng/latest/ ?).

Nothing like that to find at
http://ftp.mozilla.org/pub/mozilla.org/b2g/ .

Also no access to
https://drive.google.com/a/mozilla.com/file/d/0B7_yuXt7x6y4NE95TXVzY1NJbHc/edit?usp=sharing
(and no google account, but tried with a friend which has - but private file).

Can you please help me?
And aren't there more people with that problem?

I have only found the question what to do with the v165 flame image at b2g qa meetings (https://wiki.mozilla.org/B2G/QA/Meetings/2014-08-20) but not the answer.

I'm new on mobile os builds, but did many openwrt builds and tests, and i.e. was able to add support for a device (without the need for binary blobs there ;-) ).
And if time allows I would be happy to be able to contribute a bit here too.

Thank you!

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

Hi Alexander, at the moment we are getting started with KitKat for flame, and this bug is about standing up builds on the Mozilla build infrastucture. At the end of it we'll have builds for QA to use, and hopefully daily updates of gecko+gaia for interested people like yourself. But the base image from Thundersoft will need to be released by them, and I don't have any information on when or if that will happen.

Comment 32

[Michael Shal [:mshal]]

(In reply to Chris AtLee [:catlee] from comment #29)
> Comment on attachment 8480893 [details] [diff] [review]
> [buildbot-configs] buildbot_flamekk
> 
> Review of attachment 8480893 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> probably need to adjust try slaves below as well?

Are you talking about this section in b2g_config.py?

######## try
# Try-specific configs
# This is a path, relative to HGURL, where the repository is located
# HGURL  repo_path should be a valid repository
BRANCHES['try']['repo_path'] = 'try'

I didn't see the original flame or flame_eng builds there - would we need to add them all? I'm not entirely sure what that does or if it's needed.

Comment 33

[Michael Shal [:mshal]]

Attachment: [buildbot-configs] buildbot_flamekk_debug

Small followup - lightsofapollo also requested a debug version of flame-kk_eng. Looks like all this requres is another copy in b2g_config.py with a --debug arg so mozharness will enable B2G_DEBUG in the environment.

Comment 34

[Alexander Stadler]

(In reply to Nick Thomas [:nthomas] from comment #31)
> Hi Alexander, at the moment we are getting started with KitKat for flame,
> and this bug is about standing up builds on the Mozilla build infrastucture.
> At the end of it we'll have builds for QA to use, and hopefully daily
> updates of gecko+gaia for interested people like yourself. But the base
> image from Thundersoft will need to be released by them, and I don't have
> any information on when or if that will happen.

So sending (or giving access to) the already avalaible v165 base image from the link above to i.e. me is a problem? I don't see that (as long as you don't handle the source..).
(By the way - I don't need the daily builds from you, I would do them myself on demand as soon as I have the binary blobs.)
The v123 one was also published by you: https://developer.mozilla.org/en-US/Firefox_OS/Developer_phone_guide/Flame#Updating_your_Flame%27s_software. This could take a while if we wait for Thundersoft.. .

But you're right the bug is about building the -kk on Mozilla's infrastructure. Sorry if that created any inconvenience. Propably I should have created a separate "bug". (It was my first Bugzilla comment, have to get used to it.. .) But with the link here and only access needed I thought its a good place to ask (I know how to extract the files from the system ext4 sparse image).

Comment 35

[Alexander Stadler]

(In reply to Nick Thomas [:nthomas] from comment #8)
> (In reply to Tony Chung [:tchung] from comment #0)
> > The link to base image v165 is at:
> > https://drive.google.com/a/mozilla.com/file/d/0B7_yuXt7x6y4NE95TXVzY1NJbHc/
> > edit?usp=sharing
> 
> Could someone either convert this to a tarball of backup-flame/, or confirm
> that 
>   https://github.com/mozilla-b2g/device-flame/blob/master/extract-files.sh
> will pull the right files if I use it (after flashing the image on my
> device).

I would do:
simg2img system.img system.raw
sudo mount -t ext4 -o loop sys.raw sys/
and then copy the contents of sys/system/ to backup-flame/system/

Comment 36

[Michael Shal [:mshal]]

inbound: https://hg.mozilla.org/integration/mozilla-inbound/rev/85740fcec8e4

Comment 37

[Tony Chung [:tchung]]

(In reply to Alexander Stadler from comment #34
> So sending (or giving access to) the already avalaible v165 base image from
> the link above to i.e. me is a problem? I don't see that (as long as you
> don't handle the source..).
> (By the way - I don't need the daily builds from you, I would do them myself
> on demand as soon as I have the binary blobs.)
> The v123 one was also published by you:
> https://developer.mozilla.org/en-US/Firefox_OS/Developer_phone_guide/
> Flame#Updating_your_Flame%27s_software. This could take a while if we wait
> for Thundersoft.. .

Hi Alexander,  
thank you for inquiring, and showing great interest!   v123 image has actually cleared our legal agreement for redistribution by Mozilla/Thundersoft to the public.  But v165 (the Kk one), has not gone through that step yet, which is why we can't publicly redistribute it.   I do apologize for the inconvenience and trust me when i say that i wish we could be handing our contributors the needed tools like Flame images as soon as we can.  As soon as we can update the image to the public, we will update that MDN site.  thanks for your patience and please continue to contribute where you can!

Comment 38

[Alexander Stadler]

(In reply to Tony Chung [:tchung] from comment #37)
> (In reply to Alexander Stadler from comment #34

Thanks for your response!
Is there a supposed timeframe for v165 clearance?
Wouldn't it be easier if only distributing the needed files or the system folder instead the whole image? (Publishing a whole installable OS surely should be considered more carefully, as it has a much wider userbase, and so also would need more thorough testing than binary blobs I think.) So shouldn't this task be split into two? Because this issue will repeat and doesn't it slow down development if you need to wait for whole os releases just to be able to use updated drivers (also internally).
Or do I have to get one of you ;-)? Because in my opinion public is much more than handing it over to a named user for testing, so i.e. to me. No gain for anyone if only your QA can and has to test. But you surely are aware of that.. .

Comment 39

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/85740fcec8e4

Comment 40

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #26)
> re updates, perhaps we should be submitting into balrog with flame-kk (this
> may just work, haven't checked), and devices querying with %PRODUCT% of
> flame-kk (harder!). In the meantime we could leave nightly builds off.

This part of the update URL comes from PRODUCT_DEVICE in https://github.com/mozilla-b2g/device-flame/blob/master/flame.mk (you can see the full URL specification at http://mxr.mozilla.org/mozilla-central/source/b2g/app/b2g.js#568. Changing PRODUCT_DEVICE would be the easiest and clearest way to do this. Perhaps we should always append some Android version string to it?

I suggest we take this to another bug though.

Comment 41

[Michael Shal [:mshal]]

Oops, should have marked this [leave open], since we still need to land the mozharness & buildbot-configs changes.

Comment 42

[Michael Shal [:mshal]]

From IRC, it sounds like we're ok wrt try in the config:

21:03:31 <catlee-away> I'd copy flame's behaviour
21:03:38 <catlee-away> if flame isn't there, don't worry baout flame-kk

Comment 43

[Michael Shal [:mshal]]

Comment on attachment 8480256 [details] [diff] [review]
[mozharness] b2b_bumper config

checked-in nthomas' mozharness patch: https://hg.mozilla.org/build/mozharness/rev/36c0d9e32c68

Comment 44

[Jordan Lund (:jlund)]

patch(es) in production :)

Comment 45

[Michael Shal [:mshal]]

I think all that's left here is to land the two buildbot-configs patches (assuming the other one gets r+'d). nthomas, can you land those?

Comment 46

[Michael Shal [:mshal]]

Forgot to say that sources.xml was generated in b2g-inbound (I think correctly) - just waiting for the merge to m-c.

Comment 47

[Tony Chung [:tchung]]

Hey guys, when this is all done, would the builds location be at: https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/?   or someplace else?

Comment 48

[Nick Thomas [:nthomas] (UTC+12)]

Tony, that's right. We'll get a new set of directories in there using 'flame-kk' instead of 'flame' in the name.

Comment 49

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] Add more shortening and mix well

Builder names like b2g_mozilla-b2g32_v2_0_flame-kk_eng-debug_dep are over the 30 character limit, so we need more options to shrink the name.

Comment 50

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Combined patch

This combines the two earlier configs patches (attachments 8480893 & 8480980) with some tweaks:
* excludes m-b2g30_v1_4, no request for builds there
* adds explicit nightly disabling to commented mozilla-aurora block, and to mozilla-b2g32_v2_0  (hopefully we'll swap this to True later, and it will be obvious where to do so then)
* on the debug build, swaps to periodic rather than on every push (matches emulator-kk-debug)

It yields new builders for branches b2g-i, m-i, m-c, m-b2g32_v2_0, fx-team, graphics, jamun, cypress, cedar, pine, graphics, ash, maple, oak, and gum, like so:

b2g_<branch>_flame-kk_periodic
b2g_<branch>_flame-kk_eng_dep
b2g_<branch>_flame-kk_eng-debug_periodic

Comment 51

[Nick Thomas [:nthomas] (UTC+12)]

FYI, we're going to have conflicts with patches in bug 1029851, so landing this may be a little bumpy and require some checks after merge day.

Comment 52

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozilla-b2g32_v2_0] In-tree for 2.0 builds

For config.json - different update channel and l10n repos, from merging the difference in flame configs into flame-kk. Tooltool manifest is a direct copy.

Comment 53

[Chris AtLee [:catlee]]

Comment on attachment 8482049 [details] [diff] [review]
[buildbot-configs] Combined patch

Review of attachment 8482049 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/b2g_config.py
@@ +1428,5 @@
>  BRANCHES['mozilla-central']['platforms']['wasabi']['enable_nightly'] = True
>  BRANCHES['mozilla-central']['platforms']['flame']['enable_nightly'] = True
>  BRANCHES['mozilla-central']['platforms']['flame_eng']['enable_nightly'] = True
> +BRANCHES['mozilla-central']['platforms']['flame-kk']['enable_nightly'] = False
> +BRANCHES['mozilla-central']['platforms']['flame-kk_eng']['enable_nightly'] = False

do you need flame-kk_eng-debug here too?

Comment 54

[Tony Chung [:tchung]]

Hi nick, Catlee, 
any more progress and ETA on this landing?   waiting for these builds on a daily basis now.  (and now we're on mozilla-aurora, we'll need 2.1 there landed)

Comment 55

[Nick Thomas [:nthomas] (UTC+12)]

Tony, just need to land things at this point, and have TBPL deploy their support (gave them a nudge).

Comment 56

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8482044 [details] [diff] [review]
[buildbotcustom] Add more shortening and mix well

https://hg.mozilla.org/build/buildbotcustom/rev/a94a28a36c3f

Comment 57

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8482083 [details] [diff] [review]
[mozilla-b2g32_v2_0]  In-tree for 2.0 builds

https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/9651864ff035

Comment 58

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8482049 [details] [diff] [review]
[buildbot-configs] Combined patch

(In reply to Chris AtLee [:catlee] from comment #53)
> > +BRANCHES['mozilla-central']['platforms']['flame-kk']['enable_nightly'] = False
> > +BRANCHES['mozilla-central']['platforms']['flame-kk_eng']['enable_nightly'] = False
> 
> do you need flame-kk_eng-debug here too?

I left it out because False is the default for nightlies. The non-debug builds will get switched to True later.

Merged to current code and landed on default:
https://hg.mozilla.org/build/buildbot-configs/rev/f3a499341fb0

Will go live on the next reconfig, so I'm gambling that TBPL will be updated first. If not we can hide on TBPL.

Comment 59

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8480880 [details] [diff] [review]
[m-c] flamekk

Landed at comment #36.

Comment 60

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.

Comment 61

[Ryan VanderMeulen [:RyanVM]]

(In reply to Nick Thomas [:nthomas] from comment #57)
> Comment on attachment 8482083 [details] [diff] [review]
> [mozilla-b2g32_v2_0]  In-tree for 2.0 builds
> 
> https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/9651864ff035

Flame-KK builds on b2g32 are all dying with the error in the log below. I've hidden them for now.
https://tbpl.mozilla.org/php/getParsedLog.php?id=47406078&tree=Mozilla-B2g32-v2.0

Comment 62

[Ryan VanderMeulen [:RyanVM]]

catlee pointed out that config.json has a stray comma. Remove it:
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/2537ab191112

Comment 63

[Ryan VanderMeulen [:RyanVM]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #62)
> catlee pointed out that config.json has a stray comma. Remove it:
> https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/2537ab191112

Builds are green again. Unhidden on b2g32.

Comment 64

[Nick Thomas [:nthomas] (UTC+12)]

Thanks for the fix Ryan. As well as being green we're uploading sane sized files to sensible locations.

Still to do:
* get update query different for flame-kk, see comment #40, will file a dep bug
* turn on nightly builds on b2g32, m-a, m-c

Comment 65

[Nick Thomas [:nthomas] (UTC+12)]

BTW, retention for flame-kk tinderbox-builds is the same as flame, ie 12 weeks.

Comment 66

[Tony Chung [:tchung]]

Hey guys, how close are we on this?  will we be seeing nightly builds by tomorrow?

Comment 67

[Nick Thomas [:nthomas] (UTC+12)]

We have tinderbox-builds already, eg
https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/b2g-inbound-flame-kk/
https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/b2g-inbound-flame-kk-eng/
https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/b2g-inbound-flame-kk-eng-debug/

Nightlies are blocked on bug 1063237, unless we want to disable the updater or something.

Comment 68

[Tony Chung [:tchung]]

(In reply to Nick Thomas [:nthomas] from comment #26)
> re updates, perhaps we should be submitting into balrog with flame-kk (this
> may just work, haven't checked), and devices querying with %PRODUCT% of
> flame-kk (harder!). In the meantime we could leave nightly builds off.

Per IRC discussion, QA is okay with decoupling the OTA portion for now just to trade off getting flame KK images available for 2.0 and 2.1.   (2.1 higher priority is that matters).   but we'd still want bug 1063237, landed as soon as possible; pending the dependencies in the other bug.

Comment 69

[Nick Thomas [:nthomas] (UTC+12)]

tchung, should we be updating the base image from v165 to something more recent ?

Comment 70

[Tony Chung [:tchung]]

(In reply to Nick Thomas [:nthomas] from comment #69)
> tchung, should we be updating the base image from v165 to something more
> recent ?

Yes!  there is a 2.0 image now called v180.  Please use this one.
https://intranet.mozilla.org/QA/B2G_Tips_and_Tricks#latest_OEM_KitKat_Base_Image_:_v180

Comment 71

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko] Update to v180 image

v180 notes:
* d/l and flash onto flame
* extract with steps in comment #20, modified by comment #21
* compress: cd ../../..; tar -vcJf backup-flame.tar.xz backup-flame/
* upload to tooltool (https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool#How_to_upload_to_tooltool)

NB: the v180 image is an engineering build, I hope that's going to work for a non-eng build.

Comment 72

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Enable flame-kk nightlies, disable flame nightlies on b2g32 and newer

Change in builders:
< b2g_mozilla-central_flame_nightly ScriptFactory
< b2g_mozilla-central_flame_eng_nightly ScriptFactory
< b2g_mozilla-b2g32_v2_0_flame_nightly ScriptFactory
< b2g_mozilla-b2g32_v2_0_flame_eng_nightly ScriptFactory
< b2g_mozilla-aurora_flame_nightly ScriptFactory
< b2g_mozilla-aurora_flame_eng_nightly ScriptFactory
--
> b2g_mozilla-central_flame-kk_nightly ScriptFactory
> b2g_mozilla-central_flame-kk_eng_nightly ScriptFactory
> b2g_mozilla-b2g32_v2_0_flame-kk_nightly ScriptFactory
> b2g_mozilla-b2g32_v2_0_flame-kk_eng_nightly ScriptFactory
> b2g_mozilla-aurora_flame-kk_nightly ScriptFactory
> b2g_mozilla-aurora_flame-kk_eng_nightly ScriptFactory

Deployment strategy
* create a rule for product=B2G, buildTarget=flame, buildID < [last nightly], which points at the current dated blob (one per branch, higher priority than existing rule for branch)
* land + deploy this patch

Comment 73

[[e-mail] (:bhearsum)]

Comment on attachment 8492844 [details] [diff] [review]
[gecko] Update to v180 image

Review of attachment 8492844 [details] [diff] [review]:
-----------------------------------------------------------------

I can't imagine why this wouldn't work with user builds. AFAIK, eng vs user is mostly Gaia differences.

Comment 74

[[e-mail] (:bhearsum)]

Comment on attachment 8492845 [details] [diff] [review]
[buildbot-configs] Enable flame-kk nightlies, disable flame nightlies on b2g32 and newer

Review of attachment 8492845 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Nick Thomas [:nthomas] from comment #72)
> Deployment strategy
> * create a rule for product=B2G, buildTarget=flame, buildID < [last
> nightly], which points at the current dated blob (one per branch, higher
> priority than existing rule for branch)
> * land + deploy this patch

Sounds good to me. At some point we'll probably want to figure out a retirement strategy for rules such as this.

Comment 75

[[e-mail] (:bhearsum)]

Comment on attachment 8492845 [details] [diff] [review]
[buildbot-configs] Enable flame-kk nightlies, disable flame nightlies on b2g32 and newer

Review of attachment 8492845 [details] [diff] [review]:
-----------------------------------------------------------------

Hmm, you're not actually flipping plain flame to false here. r=me as long as you do that when you land.

Comment 77

[Nick Thomas [:nthomas] (UTC+12)]

You're right. Change of plan anyway, we have to support both for a while. :-(

Comment 78

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] balrog submission to separate blob for kitkat

Put on your hack-sensitive sunglasses ...

* both JB and KK query with 'flame' for build target
* JB have '(SDK 18)' or less in OS_VERSION, KK have '(SDK 19)'
* so we have to use a rule to point at different blobs

* at balrog submission, create blobs named like B2G-mozilla-aurora-kitkat-nightly-latest, with this content:
{
  "platforms": {
    "flame-kk": {
      "locales": {
        "en-US": {
          "buildID": "20140922073433",
          "platformVersion": "34.0a2",
          "displayVersion": "34.0a2",
          "appVersion": "34.0a2",
          "completes": [
            {
              "fileUrl": "http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-aurora/b2g-flame-gecko-update.mar",
              "from": "*",
              "hashValue": "4aa636b407f2800173305697...",
              "filesize": 70981888
            }
          ]
        }
      }
    },
    "flame": {
      "alias": "flame-kk"
    }
  },
  "hashFunction": "sha512",
  "name": "B2G-mozilla-aurora-kitkat-nightly-latest",
  "schema_version": 3
}
* the alias handles the disconnect between target at build and query time
* tested in staging for kitkat case, should be no-change for everything else

When we turn off JB builds we can revert the cli.py hunk, modify the SDK 19 specific rule to be for 18 and lower and point at the last update.

Comment 79

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8492844 [details] [diff] [review]
[gecko] Update to v180 image

https://hg.mozilla.org/integration/b2g-inbound/rev/eae3a4ef42fc
https://hg.mozilla.org/releases/mozilla-aurora/rev/48a77e1433c3
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/ac23a3d2b027

Comment 80

[[e-mail] (:bhearsum)]

Comment on attachment 8493422 [details] [diff] [review]
[tools] balrog submission to separate blob for kitkat

Review of attachment 8493422 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine to me for a temporary hack. I can't figure out why we can't also back out the platforms.py part of this when JB goes away, though. At that point, won't kitkat be using B2G-mozilla-aurora-nightly-latest, which will only have "flame" in it?

Comment 81

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/eae3a4ef42fc

Comment 82

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8493422 [details] [diff] [review]
[tools] balrog submission to separate blob for kitkat

https://hg.mozilla.org/build/tools/rev/9832742ed555

(In reply to Ben Hearsum [:bhearsum] from comment #80)
> Looks fine to me for a temporary hack. I can't figure out why we can't also
> back out the platforms.py part of this when JB goes away, though. At that
> point, won't kitkat be using B2G-mozilla-aurora-nightly-latest, which will
> only have "flame" in it?

The target in the b2g build system is flame-kk, which passes right through to platform name in the blob. We could modify that before calling the submitter, or leave the mapping part in.

Comment 83

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8492845 [details] [diff] [review]
[buildbot-configs] Enable flame-kk nightlies, disable flame nightlies on b2g32 and newer

Default + production:
https://hg.mozilla.org/build/buildbot-configs/rev/1017dcbb4a69
https://hg.mozilla.org/build/buildbot-configs/rev/8df139e015cf

Deployed to masters. And I've locked the updates for m-c, m-a, and m-b2g32 to the 2014092304020x nightlies (last one before this change), as a precaution.

Comment 84

[Nick Thomas [:nthomas] (UTC+12)]

Status update - I ran nightlies on m-a rev 96e15bbed7ab, but the machines didn't free enough space before hand, bug 1072048 to fix that up. 

The regular 4pm nightlies may avoid that fate, some have completed, eg
 https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-aurora-flame-kk/

NB: OTA is not yet configured for flame-kk, flame jb updates are frozen at 2014092304020x.

Comment 85

[Nick Thomas [:nthomas] (UTC+12)]

Status
* nightlies worked at 4pm
* for non-eng builds:
** set up rules in balrog for kk updates on all three branches, verified m-c and m-a work, m-a and m-b2g32 should work after next nightly
** verified flame-jb builds are still pointed to flame-jb builds, unfroze updates
** no updates supported for eng builds

* builds can be found in these directories:
master/2.2
 https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-flame-kk/
 https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-flame-kk-eng/
2.1
 https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-aurora-flame-kk/
 https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-aurora-flame-kk-eng/
2.0
 https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-b2g32_v2_0-flame-kk/
 https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-b2g32_v2_0-flame-kk-eng/

So I think we're done. Lets cover any other issues in followup bugs, including any transition strategy for JB users, and turning off JB builds.

Comment 86

[Tony Chung [:tchung]]

Super duper thanks for this, Nick and Ben!   i went ahead and full flashed a build, and it's working great for me so far.   Next time you're in town, drinks are on me!

Comment 87

[Pete Moore [:pmoore][:pete]]

Something here landed in production today: https://wiki.mozilla.org/ReleaseEngineering/Maintenance#Reconfigs_.2F_Deployments

Comment 88

[Nick Thomas [:nthomas] (UTC+12)]

Not actually, the reconfig script just doesn't know how do deal with changes that got transplanted to production already.