1154947 - Daily Engineering build and UserDebug builds needed for ignite

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect, P1)

Description

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Current ignite build is using user VARIANT which makes the phone secure ( ie not rootable )

we need an engineering build ( which has debug ) as well as a userdebug build ( which an end user can do adb root to get root access )

Comment 1

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

ie VARIANT=userdebug MOZILLA_OFFICIAL=1 ./build.sh

( as an example )

Comment 2

[Wander Lairson Costa]

I have been digging on how to replace PRODUCT_DEVICE by a custom name. It feels like we can't. gecko reads this value from ro.product.device, and it is set by the specific device Makefile. Is there an way to use a custom PRODUCT_DEVICE for balrog url path?

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Wander Lairson Costa [:wcosta] from comment #2)
> I have been digging on how to replace PRODUCT_DEVICE by a custom name. It
> feels like we can't. gecko reads this value from ro.product.device, and it
> is set by the specific device Makefile. Is there an way to use a custom
> PRODUCT_DEVICE for balrog url path?

Depending how deep you're willing to go you could modify the code that sets PRODUCT_DEVICE based on VARIANT type.

It may also be worth stepping back and seeing if updates for all of these build types are needed. Is there a reason that QA folks aren't able to flash the builds as needed?

Comment 4

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I think the only one we need OTA for is the userdebug build.

user build doesn't have root; so QA / dogfooders will have a hard time doing certain things on the phone if required to troubleshoot (getting crash id, changing prefs, etc.)

OTA banners caused issues in our automation in the past, so they disabled it for engineering builds.

see BUILDTYPE in http://source.android.com/source/building-running.html to see the difference between the builds.

Comment 5

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

You shouldn't have to dig into the code too much for changing the device name.

It should pick it up from the .config file.  That's how flame works:
MAKE_FLAGS=-j4
GECKO_OBJDIR=/Volumes/Projects/B2G_Flame_kk_master/objdir-gecko
DEVICE_NAME=flame-kk
PRODUCT_NAME=flame
B2G_UPDATER=1


Aries has : 
MAKE_FLAGS=-j6
GECKO_OBJDIR=/Volumes/Projects/B2G_Test_Repo/objdir-gecko
DEVICE_NAME=aries
PRODUCT_NAME=aries

so it should pick up the Device name there I thought?

Comment 6

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

also : adb shell getprop ro.build.type
That should tell you the build type. ( user, userdebug, eng )

Comment 7

[Wander Lairson Costa]

We could add the build type to PRODUCT_DEVICE in the update url when the build type is not "user" (and maybe only when the product is B2G). :rstrong, would that be feasible? ("git blame" told me you are the right person to ask).

Comment 8

[Robert Strong (they/them - no direct email)]

That is fine as far as the update service is concerned. It just uses the value from

Cu.import("resource://gre/modules/systemlibs.js", sysLibs);
sysLibs.libcutils.property_get("ro.product.device"));

http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#3504

As far as providing updates for specific values of PRODUCT_DEVICE in the update url bhearsum is the person in the know as to whether the server can do this.

Comment 9

[Doug Sherk (:drs) (inactive)]

I talked with :dhylands about this and I think he set what we need to do straight. Though we should test this before we go ahead with it.

For starters, we should continue to build all 3 variants nightly, so that we have full images available for flashing, and not just for OTA updates.

Also, let's take the 'eng' variant out of the picture, as OTA updates are disabled on it by default, so we don't need to supply OTA updates for it.

The problem described here -- though not specifically stated anywhere -- is that we don't know, from the server side, whether a device is loaded with a 'user' or 'userdebug' build. 'userdebug' is just a set of small tweaks to the kernel, including rooting it, and some prefs. It's otherwise the same as the 'user' variant.

Now, the thing about OTA updates is that they can only make changes to Gecko and Gaia, not the kernel. [1] Thus, if we flash a 'user' OTA update onto a device loaded with a 'userdebug' build, it won't actually switch that device from 'userdebug' to 'user'.

So based on this, we can just ship 'user' OTA updates only. Both 'user' and 'userdebug'-loaded devices will accept these.

I'm leaving :bhearsum's needinfo as I'm curious about what he says anyways.

[1] If we wanted to update the kernel, we would have to ship a FOTA update.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Doug Sherk (:drs) (use needinfo?) from comment #9)
> I talked with :dhylands about this and I think he set what we need to do
> straight. Though we should test this before we go ahead with it.
> 
> For starters, we should continue to build all 3 variants nightly, so that we
> have full images available for flashing, and not just for OTA updates.
> 
> Also, let's take the 'eng' variant out of the picture, as OTA updates are
> disabled on it by default, so we don't need to supply OTA updates for it.
> 
> The problem described here -- though not specifically stated anywhere -- is
> that we don't know, from the server side, whether a device is loaded with a
> 'user' or 'userdebug' build. 'userdebug' is just a set of small tweaks to
> the kernel, including rooting it, and some prefs. It's otherwise the same as
> the 'user' variant.
> 
> Now, the thing about OTA updates is that they can only make changes to Gecko
> and Gaia, not the kernel. [1] Thus, if we flash a 'user' OTA update onto a
> device loaded with a 'userdebug' build, it won't actually switch that device
> from 'userdebug' to 'user'.
> 
> So based on this, we can just ship 'user' OTA updates only. Both 'user' and
> 'userdebug'-loaded devices will accept these.
> 
> I'm leaving :bhearsum's needinfo as I'm curious about what he says anyways.
> 
> [1] If we wanted to update the kernel, we would have to ship a FOTA update.

I don't think the fact that we can give user and userdebug devices the same gecko+gaia bits helps much. We still want them to identify different to the update server so we can serve them different bits if we need to. And if we're doing that, it's best just to give them their own unique streams of updates (even the bits are very similar).

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #8)
> That is fine as far as the update service is concerned. It just uses the
> value from
> 
> Cu.import("resource://gre/modules/systemlibs.js", sysLibs);
> sysLibs.libcutils.property_get("ro.product.device"));
> 
> http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/
> nsUpdateService.js#3504

Based on this code, it looks like the tweak would have to happen in nsUpdateService.js or somewhere similar, since I doubt we can change the value returned by property_get. Maybe that part of the URL could be %PRODUCT_DEVICE%%FOO%, where %FOO% is set to "-userdebug" or something similar for debug builds.

> As far as providing updates for specific values of PRODUCT_DEVICE in the
> update url bhearsum is the person in the know as to whether the server can
> do this.

Once the builds identify uniquely this is easy. This mapping translates "buildbot" platforms to the %PRODUCT_DEVICE% section of the update url: https://github.com/mozilla/build-tools/blob/master/lib/python/release/platforms.py#L22 - as long as balrog-submitter.py is given the correct "buildbot" platform it will publish updates to the correct update platform.

Comment 11

[Robert Strong (they/them - no direct email)]

(In reply to Ben Hearsum [:bhearsum] from comment #10)
>...
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #8)
> > That is fine as far as the update service is concerned. It just uses the
> > value from
> > 
> > Cu.import("resource://gre/modules/systemlibs.js", sysLibs);
> > sysLibs.libcutils.property_get("ro.product.device"));
> > 
> > http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/
> > nsUpdateService.js#3504
> 
> Based on this code, it looks like the tweak would have to happen in
> nsUpdateService.js or somewhere similar, since I doubt we can change the
> value returned by property_get. Maybe that part of the URL could be
> %PRODUCT_DEVICE%%FOO%, where %FOO% is set to "-userdebug" or something
> similar for debug builds.
I would think that could be set with the value when compiling.

Comment 12

[Robert Strong (they/them - no direct email)]

If you can't be added to ro.product.device then it will probably be easiest to add a new value as the other values added for b2g in this block
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#3499

Comment 13

[Wander Lairson Costa]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #12)
> If you can't be added to ro.product.device then it will probably be easiest
> to add a new value as the other values added for b2g in this block
> http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/
> nsUpdateService.js#3499

My idea would be to make %PRODUCT_DEVICE% a combination of ro.product.device and ro.build.type. AFAIK, OTA updates are supplied for user variants, so to make sure we are not breaking existing stuff, do that only for non-user variants.

Comment 14

[[e-mail] (:bhearsum)]

(In reply to Wander Lairson Costa [:wcosta] from comment #13)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #12)
> > If you can't be added to ro.product.device then it will probably be easiest
> > to add a new value as the other values added for b2g in this block
> > http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/
> > nsUpdateService.js#3499
> 
> My idea would be to make %PRODUCT_DEVICE% a combination of ro.product.device
> and ro.build.type. AFAIK, OTA updates are supplied for user variants, so to
> make sure we are not breaking existing stuff, do that only for non-user
> variants.

This seems like a decent idea. It means extra updates to the platforms.py mappings, but it should work fine...

Comment 15

[Wander Lairson Costa]

Attachment: MozReview Request: bz://1154947/wcosta

/r/7747 - Bug 1154947 part 1: Add routes scopes to taskcluster graph.
/r/7749 - Bug 1154947 part 2: Use variant in the URL update.
/r/7751 - Bug 1154947 part 3: Add aries nightly user, userdebug and eng builds.

Pull down these commits:

hg pull -r bf69819aeecd32938f382b9e74897d42fbd0ff01 https://reviewboard-hg.mozilla.org/gecko/

Comment 16

[Wander Lairson Costa]

Attachment: MozReview Request: bz://1154947/wcosta

/r/7755 - Bug 1154947 part 1: Disable enforced OTA updates for non-emulator builds.
/r/7757 - Bug 1154947 part 2: Add non-nightly lightsaber config.

Pull down these commits:

hg pull -r 691e81db740f241fed8806881363e0d5220ae7df https://reviewboard-hg.mozilla.org/build-mozharness

Comment 17

[Wander Lairson Costa]

http://docs.taskcluster.net/tools/task-graph-inspector/#C_2qwCsPSFyCQJlQPJPIUg/QbEF7vUHQNGwXgkZ1dvErw/0

Comment 18

[Robert Strong (they/them - no direct email)]

https://reviewboard.mozilla.org/r/7749/#review6553

Please remove the empty lines that were added.

Comment 19

[Greg Arndt [:garndt]]

Comment on attachment 8598638 [details]
MozReview Request: bz://1154947/wcosta

https://reviewboard.mozilla.org/r/7745/#review6569

::: testing/taskcluster/mach_commands.py:330
(Diff revision 1)
> +            route_scopes = map(lambda route: 'queue:route:' + route, build_task['task'].get('routes', []))

I'm not aware of how scopes and routes are related, but we need to have a scope in the graph for each route within a task?

::: testing/taskcluster/scripts/phone-builder/build-lightsaber.sh:4
(Diff revision 1)
> +

In the build lightsaber nightly script, we were removing cached bits if the device was aries or shinano.  Is that not the here?

::: testing/taskcluster/tasks/builds/b2g_aries_lightsaber_eng.yml:4
(Diff revision 1)
> +    build_name: 'aries'

So I think for the other builds, we have something like "flame-kk-eng" as the build_name and the build type is 'opt'

Comment 20

[Wander Lairson Costa]

(In reply to Greg Arndt [:garndt] from comment #19)
> Comment on attachment 8598638 [details]
> MozReview Request: bz://1154947/wcosta
> 
> https://reviewboard.mozilla.org/r/7745/#review6569
> 
> ::: testing/taskcluster/mach_commands.py:330
> (Diff revision 1)
> > +            route_scopes = map(lambda route: 'queue:route:' + route, build_task['task'].get('routes', []))
> 
> I'm not aware of how scopes and routes are related, but we need to have a
> scope in the graph for each route within a task?
> 

Good question. ^jonasfj


> ::: testing/taskcluster/scripts/phone-builder/build-lightsaber.sh:4
> (Diff revision 1)
> > +
> 
> In the build lightsaber nightly script, we were removing cached bits if the
> device was aries or shinano.  Is that not the here?
> 

Good catch, fixed.

> ::: testing/taskcluster/tasks/builds/b2g_aries_lightsaber_eng.yml:4
> (Diff revision 1)
> > +    build_name: 'aries'
> 
> So I think for the other builds, we have something like "flame-kk-eng" as
> the build_name and the build type is 'opt'

Ok, fixed.

Comment 21

[Robert Strong (they/them - no direct email)]

Comment on attachment 8598638 [details]
MozReview Request: bz://1154947/wcosta

will

Comment 22

[Jonas Finnemann Jensen (:jonasfj)]

taskGraph.scopes must contain "queue:route:<...>" for all routes used by any task in the task-graph,
because creating a task with a route requires this scope.

task.scopes **only** needs to contain "queue:route:<...>" if it is a decision task that creates a task
using the route, or extends a task-graph which has the scope listed in taskGraph.scopes.
---

From what I can see you're doing the right thing here:
> route_scopes = map(lambda route: 'queue:route:' + route, build_task['task'].get('routes', []))
> graph['scopes'].extend(route_scopes)

Comment 23

[Jonas Finnemann Jensen (:jonasfj)]

Formally, docs from http://docs.taskcluster.net/queue/api-docs/#createTask, says:
> Task specific routing-keys, using the task.routes property you may define task specific routing-keys.
> If a task has a task specific routing-key: <route>, then the poster will be required to posses
> the scope queue:route:<route>.

When creating a task-graph the "poster" is the task-graph, hence, taskGraph.scopes must have the scope.

Comment 24

[Wander Lairson Costa]

Comment on attachment 8598638 [details]
MozReview Request: bz://1154947/wcosta

/r/7747 - Bug 1154947 part 1: Add routes scopes to taskcluster graph.
/r/7749 - Bug 1154947 part 2: Use variant in the URL update.
/r/7751 - Bug 1154947 part 3: Add aries nightly user, userdebug and eng builds.

Pull down these commits:

hg pull -r 55b7877665cb0968896ee69ebd9714b1155ebc18 https://reviewboard-hg.mozilla.org/gecko/

Comment 25

[Robert Strong (they/them - no direct email)]

https://reviewboard.mozilla.org/r/7749/#review6605

Ship It!

Comment 26

[Wander Lairson Costa]

Comment on attachment 8598638 [details]
MozReview Request: bz://1154947/wcosta

/r/7747 - Bug 1154947 part 1: Add routes scopes to taskcluster graph.
/r/7749 - Bug 1154947 part 2: Use variant in the URL update.
/r/7751 - Bug 1154947 part 3: Add aries nightly user, userdebug and eng builds.

Pull down these commits:

hg pull -r 41076d12a141045083b220f5f6fd6b4eacc0fb3f https://reviewboard-hg.mozilla.org/gecko/

Comment 27

[Wander Lairson Costa]

Comment on attachment 8598640 [details]
MozReview Request: bz://1154947/wcosta

/r/7755 - Bug 1154947 part 1: Disable enforced OTA updates for non-emulator builds.
/r/7757 - Bug 1154947 part 2: Add non-nightly lightsaber config.

Pull down these commits:

hg pull -r 691e81db740f241fed8806881363e0d5220ae7df https://reviewboard-hg.mozilla.org/build-mozharness

Comment 28

[Jonathan Griffin (:jgriffin)]

(In reply to Wander Lairson Costa [:wcosta] from comment #27)
> Comment on attachment 8598640 [details]
> MozReview Request: bz://1154947/wcosta
> 
> /r/7755 - Bug 1154947 part 1: Disable enforced OTA updates for non-emulator
> builds.
> /r/7757 - Bug 1154947 part 2: Add non-nightly lightsaber config.
> 
> Pull down these commits:
> 
> hg pull -r 691e81db740f241fed8806881363e0d5220ae7df
> https://reviewboard-hg.mozilla.org/build-mozharness

I'm probably not the right reviewer for this. Maybe catlee or jonasfj?

Comment 29

[Wander Lairson Costa]

Comment on attachment 8598640 [details]
MozReview Request: bz://1154947/wcosta

/r/7755 - Bug 1154947 part 1: Disable enforced OTA updates for non-emulator builds.
/r/7757 - Bug 1154947 part 2: Add non-nightly lightsaber config.

Pull down these commits:

hg pull -r 691e81db740f241fed8806881363e0d5220ae7df https://reviewboard-hg.mozilla.org/build-mozharness

Comment 30

[Pete Moore [:pmoore][:pete]]

https://reviewboard.mozilla.org/r/7757/#review6689

Looks correct to me. Essentially the same as taskcluster-lightsaber-nightly.py with balrog updates disabled.

Thanks Wander!

Comment 31

[Pete Moore [:pmoore][:pete]]

https://reviewboard.mozilla.org/r/7755/#review6691

I can rubber stamp this, but it might be worth getting a second reviewer here, e.g. Dave Hylands? (looks like he touched this code).

Comment 32

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8598640 [details]
MozReview Request: bz://1154947/wcosta

https://reviewboard.mozilla.org/r/7753/#review6693

Comment 33

[Dave Hylands [:dhylands]]

https://reviewboard.mozilla.org/r/7755/#review6739

Ship It!

Comment 34

[Wander Lairson Costa]

https://hg.mozilla.org/build/mozharness/rev/e135f9ede579
https://hg.mozilla.org/build/mozharness/rev/cf3b33c04c78

Comment 35

[Wander Lairson Costa]

https://hg.mozilla.org/integration/b2g-inbound/rev/072876f12b10
https://hg.mozilla.org/integration/b2g-inbound/rev/86d907839999
https://hg.mozilla.org/integration/b2g-inbound/rev/57ff71a5a1ff

Comment 36

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/072876f12b10
https://hg.mozilla.org/mozilla-central/rev/86d907839999
https://hg.mozilla.org/mozilla-central/rev/57ff71a5a1ff

Comment 37

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Note: https://tools.taskcluster.net/index/artifacts/#gecko.v1.cypress.latest.linux.aries/gecko.v1.cypress.latest.linux.aries.nightly-eng doesn't show a build.

I think this might be a slightly separate issue.  Need to get more info from wcosta in how he wants to handle it...

Comment 38

[Wander Lairson Costa]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #37)
> Note:
> https://tools.taskcluster.net/index/artifacts/#gecko.v1.cypress.latest.linux.
> aries/gecko.v1.cypress.latest.linux.aries.nightly-eng doesn't show a build.
> 
> I think this might be a slightly separate issue.  Need to get more info from
> wcosta in how he wants to handle it...

The eng builds seems to be broken, but I am sure I fixed this. I will give look on what I pushed to b2g-inbound.

Comment 39

[Wander Lairson Costa]

Ah, yeah, latest patch updates changed the path to images

https://tools.taskcluster.net/index/artifacts/#gecko.v1.cypress.latest.linux.aries-eng/gecko.v1.cypress.latest.linux.aries-eng.opt

Comment 40

[Peter Bylenga [:PBylenga]]

Can we back this out to recover flame 3.0 from a sanity blocker if we're not at risk to deprive Aries of builds.

Comment 41

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

backed out https://hg.mozilla.org/mozilla-central/rev/b90ebba4306d due to bug 1161579  

follow up in bug 1162202 and both fixes ( https://hg.mozilla.org/mozilla-central/rev/b90ebba4306d and bug 1162202 ) will need to land at the same time to not break things.

Comment 42

[Aus Lacroix [:aus]]

We will be able to re-land this when we land 1156816 to provide new updateURLs for the flame-kk builds.

3:04:40 PM wcosta: hrm, I have some flame nightly work for TC, but it does not have eng and userdebug builds
3:14:21 PM auswerk: gotcha
3:14:49 PM auswerk: do you have a partial patch for that specifically that you could just hand over to me? I'm happy to finish the work on this with naoki
3:16:33 PM wcosta: I landed it on m-c http://hg.mozilla.org/mozilla-central/file/5593ac626826/testing/taskcluster/tasks/builds/b2g_flame_kk_nightly.yml
3:17:06 PM wcosta: but it is not scheduled yet because we need to land bug 1156816
3:17:55 PM wcosta: bug 1156816 is ready to land, but I prefer to push it when I am back, just in case I break more things
3:20:14 PM auswerk: cool, well, we can just re-land everything at that time then
3:20:23 PM auswerk: QA can continue working with Flame (and others)

Will be helping out with this. :)

Comment 43

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/cd229a55ed1c

Comment 44

[Aus Lacroix [:aus]]

Re-landed the piece that was backed-out -- https://hg.mozilla.org/integration/b2g-inbound/rev/cd229a55ed1c

We'll see how it goes now that the dependencies from bug 1156816 that we had omitted have landed. I'm leaving this open until we verify that things are working as expected.

Comment 45

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/cd229a55ed1c

Comment 47

[Wander Lairson Costa]

Attachment: MozReview Request: Bug 1154947 part 1: Disable enforced OTA updates for non-emulator builds.

Comment 48

[Wander Lairson Costa]

Attachment: MozReview Request: Bug 1154947 part 2: Add non-nightly lightsaber config.

Comment 50

[Wander Lairson Costa]

Attachment: MozReview Request: Bug 1154947 part 1: Add routes scopes to taskcluster graph.

Comment 51

[Wander Lairson Costa]

Attachment: MozReview Request: Bug 1154947 part 2: Use variant in the URL update.

Comment 52

[Wander Lairson Costa]

Attachment: MozReview Request: Bug 1154947 part 3: Add aries nightly user, userdebug and eng builds.