1111890 - [Nexus 5] Create Lollipop based pvtbuilds for 2.2

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Tony Chung [:tchung]]

+++ This bug was initially created as a clone of Bug #1055305 +++

[Blocking Requested - why for this release]:

We need to create Nexus 5 builds against pvtbuilds server, since this is a defined device for testing and development in 2.2 timeframe.   They should have builds for both Master/mozilla-central and 2.2 branch (when it becomes a branch).   This will block 2.2 work.

Who can own this?   (ie. https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/)

Comment 1

[Tony Chung [:tchung]]

marking 2.2+, since this will block testing on a affected 2.2 device.

Comment 2

[Chris AtLee [:catlee]]

I think this is best done in Task Cluster.

Comment 3

[shawn ku [:sku]]

(In reply to Tony Chung [:tchung] from comment #0)
> +++ This bug was initially created as a clone of Bug #1055305 +++
> 
> [Blocking Requested - why for this release]:
> 
> We need to create Nexus 5 builds against pvtbuilds server, since this is a
> defined device for testing and development in 2.2 timeframe.   They should
> have builds for both Master/mozilla-central and 2.2 branch (when it becomes
> a branch).   This will block 2.2 work.
> 
> Who can own this?   (ie.
> https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/)

Hi Tony:
 Since there is no WIP allowed on Moz git repo. We are still working on make build pass w/o WIP. Once we get it done, manifest for nexus5-l can be ready.
(The ETA is 2 weeks to make manifest ready per my team's perspective.)

We can have a discussion about PVT build then.

Thanks!!
Shawn

Comment 4

[bhavana bajaj [:bajaj]]

(In reply to shawn ku [:sku] from comment #3)
> (In reply to Tony Chung [:tchung] from comment #0)
> > +++ This bug was initially created as a clone of Bug #1055305 +++
> > 
> > [Blocking Requested - why for this release]:
> > 
> > We need to create Nexus 5 builds against pvtbuilds server, since this is a
> > defined device for testing and development in 2.2 timeframe.   They should
> > have builds for both Master/mozilla-central and 2.2 branch (when it becomes
> > a branch).   This will block 2.2 work.
> > 
> > Who can own this?   (ie.
> > https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/)
> 
> Hi Tony:
>  Since there is no WIP allowed on Moz git repo. We are still working on make
> build pass w/o WIP. Once we get it done, manifest for nexus5-l can be ready.
> (The ETA is 2 weeks to make manifest ready per my team's perspective.)
> 
> We can have a discussion about PVT build then.
> 
> Thanks!!
> Shawn

Tony once we are able to complete the porting work, we can talk to releng to have the pvt builds..but the first step isn't done yet :(

Comment 5

[Kai-Chih Hu [:khu]]

(In reply to bhavana bajaj [:bajaj] from comment #4)
> Tony once we are able to complete the porting work, we can talk to releng to
> have the pvt builds..but the first step isn't done yet :(

So, when we're ready to talk with releng, who could own the communication? Bhavana, are you the best one to own it? Thanks.

Comment 6

[shawn ku [:sku]]

Bug 1098983 is resolved fixed. PVT build can be proceeded IMO.

Comment 7

[Keven Kuo [:kkuo]]

(In reply to shawn ku [:sku] from comment #6)
> Bug 1098983 is resolved fixed. PVT build can be proceeded IMO.

Yes, it's time that PVT build can be processed.

--
Keven

Comment 8

[bhavana bajaj [:bajaj]]

:catlee or james, not sure if taskcluster is ready or we still need help from catlee's team to support this, so NI'ing both to get this bug moving.

Comment 9

[shawn ku [:sku]]

Please let me know if any help from gonk team is needed.
(clear ni? for myself first.)

Comment 10

[Hal Wine [:hwine] use NI!]

In addition to the manifest added in bug 1098983, the build must succeed using mozharness for either automation system to be able to schedule it.

From reading above, it is not clear if that work has been done.

ni :kkuo - can you give status of mozharness integration, please?

Comment 11

[Chris AtLee [:catlee]]

we'll most likely need binary blobs extracted from the device before we can have builds in automation. QA has generally done that for us, and then we can put them into tooltool and in-tree manifests like this:
http://hg.mozilla.org/mozilla-central/file/38e4719e71af/b2g/config/nexus-4/releng-mako.tt

James, are you in a position to take these on yet?

Comment 12

[Hal Wine [:hwine] use NI!]

To ease infrastructure load, it would be helpful if at least one of the older builds can be disabled. See bug 1121495 for list.

Comment 13

[shawn ku [:sku]]

(In reply to Hal Wine [:hwine] (use needinfo) from comment #10)
> In addition to the manifest added in bug 1098983, the build must succeed
> using mozharness for either automation system to be able to schedule it.
> 
> From reading above, it is not clear if that work has been done.
> 
> ni :kkuo - can you give status of mozharness integration, please?

Hal,
 mozharness parts were done by catlee's team.
If need device team to finish this task, please kindly let us know.

Seinlin:
 Please follow up the things we need to be done.

Comment 14

[Hal Wine [:hwine] use NI!]

(In reply to Chris AtLee [:catlee] from comment #11)
> we'll most likely need binary blobs extracted from the device before we can
> have builds in automation. QA has generally done that for us, and then we
> can put them into tooltool and in-tree manifests like this:
> http://hg.mozilla.org/mozilla-central/file/38e4719e71af/b2g/config/nexus-4/
> releng-mako.tt

From email thread after b2g-drivers meeting:

On 2015-01-27 07:44 , Naoki wrote:

Ok thanks!  I noticed that there's a config called : nexus-5-l
  ie. : ./config.sh nexus-5-l 
for a new local repo after cloning the B2G repo ( ./repo init nexus-5-l for
reconfiguring a repo) and that should automatically pull the correct lollipop
parts from what I see.

I gave it a try and it indeed does pull all the vendor drivers down.

Hal, if you need any help please let me know.  It seems that everything that needs to be done is already scripted for the blobs for the N5 to go to Lollipop.

Comment 15

[Kai-Zhen Li [:seinlin][:kli]]

Chris, I try to upload nexus-5 binary to tooltool but I don't have the permission. I think you can download it from nexus binary site. Now our porting is based on 'LRX21O'. Please ni me if necessary.

Comment 16

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I think these bugs need to land to make a proper build:
bug 1127166 and bug 1126499

Comment 17

[Kai-Zhen Li [:seinlin][:kli]]

Now temporary fix of bug 1119980 is landed, it doesn't block build.

Comment 18

[Chris AtLee [:catlee]]

Attachment: bump nexus-5-l on master

Comment 19

[Hal Wine [:hwine] use NI!]

Comment on attachment 8557260 [details] [diff] [review]
bump nexus-5-l on master

5-(lowercase-L) lgtm

Comment 20

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I think bug 1128113 needs to land as well?

Once I had that patch I was able to successfully build a N5L build.

Comment 21

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/mozharness/rev/4b6bc0a8814e

Comment 22

[Chris AtLee [:catlee]]

Attachment: in-tree configs for nexus-5

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 8559132 [details] [diff] [review]
in-tree configs for nexus-5

stamp

Comment 24

[Chris AtLee [:catlee]]

Attachment: add nexus-5 to trunk branches

Comment 25

[Chris AtLee [:catlee]]

Attachment: shorten nexus 5 slavedir names

need to shorten these otherwise we die on this builder:
b2g_mozilla-b2g34_v2_1s_nexus-5-l_eng_dep

which is weird because I'm not enabling it on that branch...I think the builder is created, but not added anywhere in misc.py

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c09b015679b2

Comment 27

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5e87f62bd6a4

Comment 28

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/08703fb7e60e

Comment 29

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Turns out that we can't recover the phone, due to the apps being on the data partiion.

Could you include B2G_SYSTEM_APPS=1 variable in the build script please? see bug 1128836 comment 22~24 for more details.

Comment 30

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Should I file a different bug for comment 29?  ( please needinfo me if that is the case )

Comment 31

[Hermes Cheng (inactive after July 27, 2015)]

Please uplift to 2.2, thanks.

Comment 32

[Chris AtLee [:catlee]]

Attachment: bump manifests on b2g37/v2.2

Comment 33

[James Lal [:lightsofapollo]]

++ I Just want to thank catlee/rail here I dropped the ball

Comment 34

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: nexus-5-l_build.patch

Patch for building with VARIANT=userdebug + B2G_SYSTEM_APPS=1 based on comment 29.

Comment 35

[Eric Chang [:ericcc] [:echang]]

The nexus-5-l.zip is only 27KB in size, it seems a little bit too small.

nexus-5-l.zip	06-Feb-2015 01:40 	27K	 

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-nexus-5-l-eng/2015/02/2015-02-05-16-04-02/

Comment 36

[Chris AtLee [:catlee]]

Comment on attachment 8560181 [details] [diff] [review]
nexus-5-l_build.patch

Review of attachment 8560181 [details] [diff] [review]:
-----------------------------------------------------------------

r+, but can you update the entries in the "zip_files" section that reference mako and replace them with hammerhead?

Comment 37

[Chris AtLee [:catlee]]

Attachment: update files to include in zip

Comment 38

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Comment on attachment 8560666 [details] [diff] [review]
update files to include in zip

Review of attachment 8560666 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.  Sorry I didn't see that section the first time around.

Comment 39

[Kai-Zhen Li [:seinlin][:kli]]

Chris, Now we can see nexus-5-l builds are on m/c. Can we also have it on v2.2? Thanks!

Comment 40

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8559778 [details] [diff] [review]
bump manifests on b2g37/v2.2

Review of attachment 8559778 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry for delay in reviewing this! I'll check it in too, to compensate. =)

Comment 41

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8559778 [details] [diff] [review]
bump manifests on b2g37/v2.2

I've landed on default, but not merged to production since there is quite a backlog (http://hg.mozilla.org/build/mozharness/graph), and this is no doubt better handled by build duty.

On default:
https://hg.mozilla.org/build/mozharness/rev/cc61099cd96a

Comment 42

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0a55cb0d339d

Comment 43

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I saw comment 39.  Do we need to do anything else to have 2.2 builds for Nexus L on pvtbuilds?  
ie is there anything else blocking us?  Please let us know.  :)

Comment 44

[Pete Moore [:pmoore][:pete]]

Hi Naoki,

We required a merge of default branch -> production branch in mozharness, and this has now been done:
http://hg.mozilla.org/build/mozharness/rev/937da8fabeaf

So I believe this change should be active now.

Thanks,
Pete

Comment 45

[Chris AtLee [:catlee]]

Are the builds on master functional now?

Comment 46

[Eric Chang [:ericcc] [:echang]]

For what I know,
1. We cannot boot from user build, but engineering build is okay.
2. We have issues when trying to run gaia-ui-test automation, issues with test cases with keyboard, we just cannot enter text with it by automation, and more than 50% of our test cases are failed.

Hermes should have more details on this.

Comment 47

[Hermes Cheng (inactive after July 27, 2015)]

(In reply to Eric Chang [:ericcc] [:echang] from comment #46)
> For what I know,
> 1. We cannot boot from user build, but engineering build is okay.
This problem is not confirmed. I will flash another phone with user build again.
> 2. We have issues when trying to run gaia-ui-test automation, issues with
> test cases with keyboard, we just cannot enter text with it by automation,
> and more than 50% of our test cases are failed.
> 
> Hermes should have more details on this.

For the pvt build on master, I will perform a test run and send out test report.

Comment 48

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Ok, thanks Pmoore.  From the sounds of things, we just need to wait for the next build cycle.

Catlee, we'll (QA: TW/US) take a look into it.

Comment 49

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Parul, if you have the time please take a look as well.

Comment 50

[Hermes Cheng (inactive after July 27, 2015)]

(In reply to Hermes Cheng[:hermescheng] from comment #47)
> (In reply to Eric Chang [:ericcc] [:echang] from comment #46)
> > For what I know,
> > 1. We cannot boot from user build, but engineering build is okay.
> This problem is not confirmed. I will flash another phone with user build
> again.
> > 2. We have issues when trying to run gaia-ui-test automation, issues with
> > test cases with keyboard, we just cannot enter text with it by automation,
> > and more than 50% of our test cases are failed.
> > 
> > Hermes should have more details on this.
> 
> For the pvt build on master, I will perform a test run and send out test
> report.


After flash my Nexus 5 with user build, the phone cannot reboot. Please see bug 1132837.

Comment 51

[Hermes Cheng (inactive after July 27, 2015)]

(In reply to Pete Moore [:pete][:pmoore] from comment #44)
> Hi Naoki,
> 
> We required a merge of default branch -> production branch in mozharness,
> and this has now been done:
> http://hg.mozilla.org/build/mozharness/rev/937da8fabeaf
> 
> So I believe this change should be active now.
> 
> Thanks,
> Pete

Hi Pete,

It seems not build on v2.2 now. Could you please check it? Thanks.


@Chris, in my last test run on pvt master build, the builds on master is ok.
* Test env info
Build ID 20150212160523
Gaia Revision 2a2b008f9ae957fe19ad540d233d86b5c0b6829e
Gaia Date 2015-02-12 07:44:13
Gecko Revision https://hg.mozilla.org/mozilla-central/rev/2f5c5ec1a24b
Gecko Version 38.0a1
Device Name hammerhead
Firmware(Release) 5.0
Firmware(Incremental) eng.cltbld.20150212.192802
Firmware Date Thu Feb 12 19:28:15 EST 2015
Bootloader HHZ12d

Comment 52

[Pete Moore [:pmoore][:pete]]

Catlee,

Should we also add something like:

BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-l']['enable_nightly'] = True
BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-l_eng']['enable_nightly'] = True
BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-l_eng']['consider_for_nightly'] = False

to:

https://hg.mozilla.org/build/buildbot-configs/rev/5e87f62bd6a4#l1.117 ?

Not sure what "consider_for_nightly" is.

Pete

Comment 53

[Kai-Zhen Li [:seinlin][:kli]]

Attachment: bug-1111890-update-binary.patch

Catlee,

We need to use nexus binary files in our build. But I didn't see them in b2g/config/nexus-5-l. I already uploaded these files to /tooltool/uploads/kli/pvt.

Could you help to review and land this if it is really missing?

Thanks!

Comment 54

[Chris AtLee [:catlee]]

(In reply to Pete Moore [:pete][:pmoore] from comment #52)
> Catlee,
> 
> Should we also add something like:
> 
> BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-l']['enable_nightly'] =
> True
> BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-
> l_eng']['enable_nightly'] = True
> BRANCHES['mozilla-b2g37_v2_2']['platforms']['nexus-5-
> l_eng']['consider_for_nightly'] = False
> 

Yes, but we'll also need config.json and releng-nexus5.tt changes ported over to mozilla-b2g37_v2_2 before these will be successful.

> to:
> 
> https://hg.mozilla.org/build/buildbot-configs/rev/5e87f62bd6a4#l1.117 ?
> 
> Not sure what "consider_for_nightly" is.

It determines which builds are considered for choosing a "good" nightly revision to use. If any build with "consider_for_nightly" has failed for a revision, then that revision won't be used for subsequent nightly builds.

Comment 55

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: https://github.com/mozilla-b2g/b2g-manifest/pull/291

Pull request to correct a nonexistant revision for v2.2 nexus.

Comment 56

[Kai-Zhen Li [:seinlin][:kli]]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #55)
> Created attachment 8566495 [details] [review]
> https://github.com/mozilla-b2g/b2g-manifest/pull/291

merged: https://github.com/mozilla-b2g/b2g-manifest/commit/17ef10caafe84b72f7e53fe4f49f298cb3830c69

Comment 57

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/9140d8c61635
https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/df38c1e22633

Note that the status flags here are being set for the in-tree patches. We'll leave the bug open until they're enabled in production.

Comment 58

[Ryan VanderMeulen [:RyanVM]]

Forgot to change the nightly-specific bits.
https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/42cf34aa81a0

Comment 59

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Thanks, Ryan!  

Ok, it sounds like we have everything in place.  We just have to wait for the builds?

Comment 60

[Chris AtLee [:catlee]]

Attachment: enable on 2.2

Comment 61

[Hermes Cheng (inactive after July 27, 2015)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #34)
> Created attachment 8560181 [details] [diff] [review]
> nexus-5-l_build.patch
> 
> Patch for building with VARIANT=userdebug + B2G_SYSTEM_APPS=1 based on
> comment 29.

Could you check in this patch for Bug 1128836?

Comment 62

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/845675ef8d56

Comment 63

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: b2gsystemapp_v2.2.patch

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
User impact if declined: pvtbuilds for 2.2 nexus-5-l won't show apps/boot properly
Testing completed: 
Risk to taking this patch (and alternatives if risky): 
String or UUID changes made by this patch:

Comment 64

[Kai-Zhen Li [:seinlin][:kli]]

When B2G_SYSTEM_APPS = 1 in userconfig, apps will always be installed to '/system/b2g/webapps'. This patch will affect to 'eng' build only, because in 'user' and 'userdebug' B2G_SYSTEM_APPS is already set to 1 by default. Is it the same with what you expected?

BTW, I think process this in a different bug could be better for tracking.

Comment 65

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I think we do want the engineering builds to be default to the system partition, don't we?  Somewhere along the lines I think we did have this discussion before on another thread when we ran out of space on the system partition...

Hermes had asked for this patch to be placed on 2.2 as well.  Could you talk to Hermes in person also to see if this is a blocker for him or not?

If we don't need this additional patch to make sure we have working user(debug) and eng builds, then I'm fine not pushing it.

Comment 66

[Hermes Cheng (inactive after July 27, 2015)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #65)
> I think we do want the engineering builds to be default to the system
> partition, don't we?  Somewhere along the lines I think we did have this
> discussion before on another thread when we ran out of space on the system
> partition...
> 
> Hermes had asked for this patch to be placed on 2.2 as well.  Could you talk
> to Hermes in person also to see if this is a blocker for him or not?
> 
> If we don't need this additional patch to make sure we have working
> user(debug) and eng builds, then I'm fine not pushing it.

Sorry, it does not block me now after I have pvt build instead of local build for v2.2.

Comment 67

[Kai-Zhen Li [:seinlin][:kli]]

Comment on attachment 8568844 [details] [diff] [review]
b2gsystemapp_v2.2.patch

Review of attachment 8568844 [details] [diff] [review]:
-----------------------------------------------------------------

I have no particular opinion on the patch and I also think the patch is also good to land. 

But I notice all other devices build didn't set B2G_SYSTEM_APPS and all apps are under /system/b2g. In nexus-5-l apps are also under /system/b2g. 

Finally, I found the answer in [1], all device eng build use releng-otoro-eng.py, and B2G_SYSTEM_APPS is set to 1 there. Now I can sure we don't need to land this patch.

[1] http://hg.mozilla.org/build/mozharness/file/f2510cc466c0/configs/b2g/releng-otoro-eng.py#l73

Comment 68

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Comment on attachment 8568844 [details] [diff] [review]
b2gsystemapp_v2.2.patch

minusing approval and review then because we don't need the patch.

Comment 69

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I tested the builds, they are flashable.

I think there's something odd about the flash script (at least running on the mac).  It didn't quite seem to flash or was taking a really long time so I ran the fastboot flash manually.

I think we might have to update the flash scripts, which we can handle in a different bug.  I think this bug can be marked resolved/verified fixed.