Daily Engineering build and UserDebug builds needed for ignite

RESOLVED FIXED in Firefox 40

Status

Firefox OS
General
P1
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: nhirata, Assigned: aus)

Tracking

unspecified
2.2 S12 (15may)
ARM
Gonk (Firefox OS)
Dependency tree / graph

Firefox Tracking Flags

(firefox40 fixed)

Details

(Whiteboard: [spark][backout-asap])

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(5 attachments, 2 obsolete attachments)

Current ignite build is using user VARIANT which makes the phone secure ( ie not rootable )

we need an engineering build ( which has debug ) as well as a userdebug build ( which an end user can do adb root to get root access )
ie VARIANT=userdebug MOZILLA_OFFICIAL=1 ./build.sh

( as an example )
Assignee: nobody → wcosta
Priority: -- → P1
Whiteboard: [ignite]
Status: NEW → ASSIGNED
I have been digging on how to replace PRODUCT_DEVICE by a custom name. It feels like we can't. gecko reads this value from ro.product.device, and it is set by the specific device Makefile. Is there an way to use a custom PRODUCT_DEVICE for balrog url path?
Flags: needinfo?(bhearsum)
(In reply to Wander Lairson Costa [:wcosta] from comment #2)
> I have been digging on how to replace PRODUCT_DEVICE by a custom name. It
> feels like we can't. gecko reads this value from ro.product.device, and it
> is set by the specific device Makefile. Is there an way to use a custom
> PRODUCT_DEVICE for balrog url path?

Depending how deep you're willing to go you could modify the code that sets PRODUCT_DEVICE based on VARIANT type.

It may also be worth stepping back and seeing if updates for all of these build types are needed. Is there a reason that QA folks aren't able to flash the builds as needed?
Flags: needinfo?(bhearsum) → needinfo?(nhirata.bugzilla)
I think the only one we need OTA for is the userdebug build.

user build doesn't have root; so QA / dogfooders will have a hard time doing certain things on the phone if required to troubleshoot (getting crash id, changing prefs, etc.)

OTA banners caused issues in our automation in the past, so they disabled it for engineering builds.

see BUILDTYPE in http://source.android.com/source/building-running.html to see the difference between the builds.
Flags: needinfo?(nhirata.bugzilla)
You shouldn't have to dig into the code too much for changing the device name.

It should pick it up from the .config file.  That's how flame works:
MAKE_FLAGS=-j4
GECKO_OBJDIR=/Volumes/Projects/B2G_Flame_kk_master/objdir-gecko
DEVICE_NAME=flame-kk
PRODUCT_NAME=flame
B2G_UPDATER=1


Aries has : 
MAKE_FLAGS=-j6
GECKO_OBJDIR=/Volumes/Projects/B2G_Test_Repo/objdir-gecko
DEVICE_NAME=aries
PRODUCT_NAME=aries

so it should pick up the Device name there I thought?
also : adb shell getprop ro.build.type
That should tell you the build type. ( user, userdebug, eng )
We could add the build type to PRODUCT_DEVICE in the update url when the build type is not "user" (and maybe only when the product is B2G). :rstrong, would that be feasible? ("git blame" told me you are the right person to ask).
Flags: needinfo?(robert.strong.bugs)
That is fine as far as the update service is concerned. It just uses the value from

Cu.import("resource://gre/modules/systemlibs.js", sysLibs);
sysLibs.libcutils.property_get("ro.product.device"));

http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#3504

As far as providing updates for specific values of PRODUCT_DEVICE in the update url bhearsum is the person in the know as to whether the server can do this.
Flags: needinfo?(robert.strong.bugs) → needinfo?(bhearsum)
I talked with :dhylands about this and I think he set what we need to do straight. Though we should test this before we go ahead with it.

For starters, we should continue to build all 3 variants nightly, so that we have full images available for flashing, and not just for OTA updates.

Also, let's take the 'eng' variant out of the picture, as OTA updates are disabled on it by default, so we don't need to supply OTA updates for it.

The problem described here -- though not specifically stated anywhere -- is that we don't know, from the server side, whether a device is loaded with a 'user' or 'userdebug' build. 'userdebug' is just a set of small tweaks to the kernel, including rooting it, and some prefs. It's otherwise the same as the 'user' variant.

Now, the thing about OTA updates is that they can only make changes to Gecko and Gaia, not the kernel. [1] Thus, if we flash a 'user' OTA update onto a device loaded with a 'userdebug' build, it won't actually switch that device from 'userdebug' to 'user'.

So based on this, we can just ship 'user' OTA updates only. Both 'user' and 'userdebug'-loaded devices will accept these.

I'm leaving :bhearsum's needinfo as I'm curious about what he says anyways.

[1] If we wanted to update the kernel, we would have to ship a FOTA update.
(In reply to Doug Sherk (:drs) (use needinfo?) from comment #9)
> I talked with :dhylands about this and I think he set what we need to do
> straight. Though we should test this before we go ahead with it.
> 
> For starters, we should continue to build all 3 variants nightly, so that we
> have full images available for flashing, and not just for OTA updates.
> 
> Also, let's take the 'eng' variant out of the picture, as OTA updates are
> disabled on it by default, so we don't need to supply OTA updates for it.
> 
> The problem described here -- though not specifically stated anywhere -- is
> that we don't know, from the server side, whether a device is loaded with a
> 'user' or 'userdebug' build. 'userdebug' is just a set of small tweaks to
> the kernel, including rooting it, and some prefs. It's otherwise the same as
> the 'user' variant.
> 
> Now, the thing about OTA updates is that they can only make changes to Gecko
> and Gaia, not the kernel. [1] Thus, if we flash a 'user' OTA update onto a
> device loaded with a 'userdebug' build, it won't actually switch that device
> from 'userdebug' to 'user'.
> 
> So based on this, we can just ship 'user' OTA updates only. Both 'user' and
> 'userdebug'-loaded devices will accept these.
> 
> I'm leaving :bhearsum's needinfo as I'm curious about what he says anyways.
> 
> [1] If we wanted to update the kernel, we would have to ship a FOTA update.

I don't think the fact that we can give user and userdebug devices the same gecko+gaia bits helps much. We still want them to identify different to the update server so we can serve them different bits if we need to. And if we're doing that, it's best just to give them their own unique streams of updates (even the bits are very similar).

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #8)
> That is fine as far as the update service is concerned. It just uses the
> value from
> 
> Cu.import("resource://gre/modules/systemlibs.js", sysLibs);
> sysLibs.libcutils.property_get("ro.product.device"));
> 
> http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/
> nsUpdateService.js#3504

Based on this code, it looks like the tweak would have to happen in nsUpdateService.js or somewhere similar, since I doubt we can change the value returned by property_get. Maybe that part of the URL could be %PRODUCT_DEVICE%%FOO%, where %FOO% is set to "-userdebug" or something similar for debug builds.

> As far as providing updates for specific values of PRODUCT_DEVICE in the
> update url bhearsum is the person in the know as to whether the server can
> do this.

Once the builds identify uniquely this is easy. This mapping translates "buildbot" platforms to the %PRODUCT_DEVICE% section of the update url: https://github.com/mozilla/build-tools/blob/master/lib/python/release/platforms.py#L22 - as long as balrog-submitter.py is given the correct "buildbot" platform it will publish updates to the correct update platform.
Flags: needinfo?(bhearsum)
(In reply to Ben Hearsum [:bhearsum] from comment #10)
>...
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #8)
> > That is fine as far as the update service is concerned. It just uses the
> > value from
> > 
> > Cu.import("resource://gre/modules/systemlibs.js", sysLibs);
> > sysLibs.libcutils.property_get("ro.product.device"));
> > 
> > http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/
> > nsUpdateService.js#3504
> 
> Based on this code, it looks like the tweak would have to happen in
> nsUpdateService.js or somewhere similar, since I doubt we can change the
> value returned by property_get. Maybe that part of the URL could be
> %PRODUCT_DEVICE%%FOO%, where %FOO% is set to "-userdebug" or something
> similar for debug builds.
I would think that could be set with the value when compiling.
If you can't be added to ro.product.device then it will probably be easiest to add a new value as the other values added for b2g in this block
http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsUpdateService.js#3499
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #12)
> If you can't be added to ro.product.device then it will probably be easiest
> to add a new value as the other values added for b2g in this block
> http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/
> nsUpdateService.js#3499

My idea would be to make %PRODUCT_DEVICE% a combination of ro.product.device and ro.build.type. AFAIK, OTA updates are supplied for user variants, so to make sure we are not breaking existing stuff, do that only for non-user variants.
(In reply to Wander Lairson Costa [:wcosta] from comment #13)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #12)
> > If you can't be added to ro.product.device then it will probably be easiest
> > to add a new value as the other values added for b2g in this block
> > http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/
> > nsUpdateService.js#3499
> 
> My idea would be to make %PRODUCT_DEVICE% a combination of ro.product.device
> and ro.build.type. AFAIK, OTA updates are supplied for user variants, so to
> make sure we are not breaking existing stuff, do that only for non-user
> variants.

This seems like a decent idea. It means extra updates to the platforms.py mappings, but it should work fine...
Blocks: 1154767
No longer depends on: 1154767
Whiteboard: [ignite] → [spark]
Created attachment 8598638 [details]
MozReview Request: bz://1154947/wcosta

/r/7747 - Bug 1154947 part 1: Add routes scopes to taskcluster graph.
/r/7749 - Bug 1154947 part 2: Use variant in the URL update.
/r/7751 - Bug 1154947 part 3: Add aries nightly user, userdebug and eng builds.

Pull down these commits:

hg pull -r bf69819aeecd32938f382b9e74897d42fbd0ff01 https://reviewboard-hg.mozilla.org/gecko/
Attachment #8598638 - Flags: review?(robert.strong.bugs)
Attachment #8598638 - Flags: review?(garndt)
Created attachment 8598640 [details]
MozReview Request: bz://1154947/wcosta

/r/7755 - Bug 1154947 part 1: Disable enforced OTA updates for non-emulator builds.
/r/7757 - Bug 1154947 part 2: Add non-nightly lightsaber config.

Pull down these commits:

hg pull -r 691e81db740f241fed8806881363e0d5220ae7df https://reviewboard-hg.mozilla.org/build-mozharness
Attachment #8598640 - Flags: review?(bhearsum)
Comment on attachment 8598638 [details]
MozReview Request: bz://1154947/wcosta

https://reviewboard.mozilla.org/r/7745/#review6569

::: testing/taskcluster/mach_commands.py:330
(Diff revision 1)
> +            route_scopes = map(lambda route: 'queue:route:' + route, build_task['task'].get('routes', []))

I'm not aware of how scopes and routes are related, but we need to have a scope in the graph for each route within a task?

::: testing/taskcluster/scripts/phone-builder/build-lightsaber.sh:4
(Diff revision 1)
> +

In the build lightsaber nightly script, we were removing cached bits if the device was aries or shinano.  Is that not the here?

::: testing/taskcluster/tasks/builds/b2g_aries_lightsaber_eng.yml:4
(Diff revision 1)
> +    build_name: 'aries'

So I think for the other builds, we have something like "flame-kk-eng" as the build_name and the build type is 'opt'
Attachment #8598638 - Flags: review?(garndt)
(In reply to Greg Arndt [:garndt] from comment #19)
> Comment on attachment 8598638 [details]
> MozReview Request: bz://1154947/wcosta
> 
> https://reviewboard.mozilla.org/r/7745/#review6569
> 
> ::: testing/taskcluster/mach_commands.py:330
> (Diff revision 1)
> > +            route_scopes = map(lambda route: 'queue:route:' + route, build_task['task'].get('routes', []))
> 
> I'm not aware of how scopes and routes are related, but we need to have a
> scope in the graph for each route within a task?
> 

Good question. ^jonasfj


> ::: testing/taskcluster/scripts/phone-builder/build-lightsaber.sh:4
> (Diff revision 1)
> > +
> 
> In the build lightsaber nightly script, we were removing cached bits if the
> device was aries or shinano.  Is that not the here?
> 

Good catch, fixed.

> ::: testing/taskcluster/tasks/builds/b2g_aries_lightsaber_eng.yml:4
> (Diff revision 1)
> > +    build_name: 'aries'
> 
> So I think for the other builds, we have something like "flame-kk-eng" as
> the build_name and the build type is 'opt'

Ok, fixed.
Flags: needinfo?(jopsen)
Comment on attachment 8598638 [details]
MozReview Request: bz://1154947/wcosta

will
Attachment #8598638 - Flags: review?(robert.strong.bugs)
taskGraph.scopes must contain "queue:route:<...>" for all routes used by any task in the task-graph,
because creating a task with a route requires this scope.

task.scopes **only** needs to contain "queue:route:<...>" if it is a decision task that creates a task
using the route, or extends a task-graph which has the scope listed in taskGraph.scopes.
---

From what I can see you're doing the right thing here:
> route_scopes = map(lambda route: 'queue:route:' + route, build_task['task'].get('routes', []))
> graph['scopes'].extend(route_scopes)
Flags: needinfo?(jopsen)
Formally, docs from http://docs.taskcluster.net/queue/api-docs/#createTask, says:
> Task specific routing-keys, using the task.routes property you may define task specific routing-keys.
> If a task has a task specific routing-key: <route>, then the poster will be required to posses
> the scope queue:route:<route>.

When creating a task-graph the "poster" is the task-graph, hence, taskGraph.scopes must have the scope.
Comment on attachment 8598638 [details]
MozReview Request: bz://1154947/wcosta

/r/7747 - Bug 1154947 part 1: Add routes scopes to taskcluster graph.
/r/7749 - Bug 1154947 part 2: Use variant in the URL update.
/r/7751 - Bug 1154947 part 3: Add aries nightly user, userdebug and eng builds.

Pull down these commits:

hg pull -r 55b7877665cb0968896ee69ebd9714b1155ebc18 https://reviewboard-hg.mozilla.org/gecko/
Attachment #8598638 - Flags: review?(robert.strong.bugs)
Attachment #8598638 - Flags: review?(garndt)
Attachment #8598638 - Flags: review?(robert.strong.bugs) → review+
Comment on attachment 8598638 [details]
MozReview Request: bz://1154947/wcosta

/r/7747 - Bug 1154947 part 1: Add routes scopes to taskcluster graph.
/r/7749 - Bug 1154947 part 2: Use variant in the URL update.
/r/7751 - Bug 1154947 part 3: Add aries nightly user, userdebug and eng builds.

Pull down these commits:

hg pull -r 41076d12a141045083b220f5f6fd6b4eacc0fb3f https://reviewboard-hg.mozilla.org/gecko/
Attachment #8598638 - Flags: review+
Comment on attachment 8598640 [details]
MozReview Request: bz://1154947/wcosta

/r/7755 - Bug 1154947 part 1: Disable enforced OTA updates for non-emulator builds.
/r/7757 - Bug 1154947 part 2: Add non-nightly lightsaber config.

Pull down these commits:

hg pull -r 691e81db740f241fed8806881363e0d5220ae7df https://reviewboard-hg.mozilla.org/build-mozharness
Attachment #8598640 - Flags: review?(bhearsum) → review?(jgriffin)
(In reply to Wander Lairson Costa [:wcosta] from comment #27)
> Comment on attachment 8598640 [details]
> MozReview Request: bz://1154947/wcosta
> 
> /r/7755 - Bug 1154947 part 1: Disable enforced OTA updates for non-emulator
> builds.
> /r/7757 - Bug 1154947 part 2: Add non-nightly lightsaber config.
> 
> Pull down these commits:
> 
> hg pull -r 691e81db740f241fed8806881363e0d5220ae7df
> https://reviewboard-hg.mozilla.org/build-mozharness

I'm probably not the right reviewer for this. Maybe catlee or jonasfj?

Updated

3 years ago
Attachment #8598638 - Flags: review?(garndt) → review+
Comment on attachment 8598640 [details]
MozReview Request: bz://1154947/wcosta

/r/7755 - Bug 1154947 part 1: Disable enforced OTA updates for non-emulator builds.
/r/7757 - Bug 1154947 part 2: Add non-nightly lightsaber config.

Pull down these commits:

hg pull -r 691e81db740f241fed8806881363e0d5220ae7df https://reviewboard-hg.mozilla.org/build-mozharness
Attachment #8598640 - Flags: review?(jgriffin) → review?(pmoore)
https://reviewboard.mozilla.org/r/7757/#review6689

Looks correct to me. Essentially the same as taskcluster-lightsaber-nightly.py with balrog updates disabled.

Thanks Wander!
https://reviewboard.mozilla.org/r/7755/#review6691

I can rubber stamp this, but it might be worth getting a second reviewer here, e.g. Dave Hylands? (looks like he touched this code).
Comment on attachment 8598640 [details]
MozReview Request: bz://1154947/wcosta

https://reviewboard.mozilla.org/r/7753/#review6693
Attachment #8598640 - Flags: review?(pmoore) → review+
https://hg.mozilla.org/mozilla-central/rev/072876f12b10
https://hg.mozilla.org/mozilla-central/rev/86d907839999
https://hg.mozilla.org/mozilla-central/rev/57ff71a5a1ff
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
status-firefox40: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 2.2 S11 (1may)
Note: https://tools.taskcluster.net/index/artifacts/#gecko.v1.cypress.latest.linux.aries/gecko.v1.cypress.latest.linux.aries.nightly-eng doesn't show a build.

I think this might be a slightly separate issue.  Need to get more info from wcosta in how he wants to handle it...
Flags: needinfo?(wcosta)
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #37)
> Note:
> https://tools.taskcluster.net/index/artifacts/#gecko.v1.cypress.latest.linux.
> aries/gecko.v1.cypress.latest.linux.aries.nightly-eng doesn't show a build.
> 
> I think this might be a slightly separate issue.  Need to get more info from
> wcosta in how he wants to handle it...

The eng builds seems to be broken, but I am sure I fixed this. I will give look on what I pushed to b2g-inbound.
Flags: needinfo?(wcosta)
Can we back this out to recover flame 3.0 from a sanity blocker if we're not at risk to deprive Aries of builds.
Whiteboard: [spark] → [spark][backout-asap]
backed out https://hg.mozilla.org/mozilla-central/rev/b90ebba4306d due to bug 1161579  

follow up in bug 1162202 and both fixes ( https://hg.mozilla.org/mozilla-central/rev/b90ebba4306d and bug 1162202 ) will need to land at the same time to not break things.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 42

3 years ago
We will be able to re-land this when we land 1156816 to provide new updateURLs for the flame-kk builds.

3:04:40 PM wcosta: hrm, I have some flame nightly work for TC, but it does not have eng and userdebug builds
3:14:21 PM auswerk: gotcha
3:14:49 PM auswerk: do you have a partial patch for that specifically that you could just hand over to me? I'm happy to finish the work on this with naoki
3:16:33 PM wcosta: I landed it on m-c http://hg.mozilla.org/mozilla-central/file/5593ac626826/testing/taskcluster/tasks/builds/b2g_flame_kk_nightly.yml
3:17:06 PM wcosta: but it is not scheduled yet because we need to land bug 1156816
3:17:55 PM wcosta: bug 1156816 is ready to land, but I prefer to push it when I am back, just in case I break more things
3:20:14 PM auswerk: cool, well, we can just re-land everything at that time then
3:20:23 PM auswerk: QA can continue working with Flame (and others)

Will be helping out with this. :)
Assignee: wcosta → aus
Status: REOPENED → ASSIGNED
(Assignee)

Comment 44

3 years ago
Re-landed the piece that was backed-out -- https://hg.mozilla.org/integration/b2g-inbound/rev/cd229a55ed1c

We'll see how it goes now that the dependencies from bug 1156816 that we had omitted have landed. I'm leaving this open until we verify that things are working as expected.
https://hg.mozilla.org/mozilla-central/rev/cd229a55ed1c
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
Target Milestone: 2.2 S11 (1may) → 2.2 S12 (15may)
Comment on attachment 8598640 [details]
MozReview Request: bz://1154947/wcosta
Attachment #8598640 - Attachment is obsolete: true
Attachment #8598638 - Attachment is obsolete: true
Attachment #8620050 - Flags: review+
Attachment #8620051 - Flags: review+
Attachment #8620052 - Flags: review+
Attachment #8620053 - Flags: review+
Attachment #8620054 - Flags: review+
Created attachment 8620050 [details]
MozReview Request: Bug 1154947 part 1: Disable enforced OTA updates for non-emulator builds.
Created attachment 8620051 [details]
MozReview Request: Bug 1154947 part 2: Add non-nightly lightsaber config.
Comment on attachment 8598638 [details]
MozReview Request: bz://1154947/wcosta
Created attachment 8620052 [details]
MozReview Request: Bug 1154947 part 1: Add routes scopes to taskcluster graph.
Created attachment 8620053 [details]
MozReview Request: Bug 1154947 part 2: Use variant in the URL update.
Created attachment 8620054 [details]
MozReview Request: Bug 1154947 part 3: Add aries nightly user, userdebug and eng builds.
You need to log in before you can comment on or make changes to this bug.