1180707 - (2.5_LateCustomisation) late customisation

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Comment 1

[Etienne Segonzac (:etienne)]

This could be implemented as an addon to the system app.

It'll avoid "code dumping" something very carrier specific in the system app and be more transparent to the user (it'll be possible to disable the addon after the FTU).

And all of this while still fulfilling the AC.

Comment 2

[Sam Foster [:sfoster] (he/him)]

I dont see any of this as operator-specific really. The marketplace URL that responds with a list of apps to install would presumably live in a setting. (Reportedly this will be similar to operator shelf) Each operator would configure that as necessary. If that setting didn't exist, we would simply skip this whole step. I had thought of doing this like so: 

Somewhere in the FTU - once there's a connection - we request the list of apps/manifests from the marketplace. As the FTU is likely to be closed before install finishes, we need to send this list to something that can own the install - I'm not sure yet if this is the system app, the marketplace or what. So we would IAC to this app with the list of manifestURLs, it would trigger install of each and handle any error conditions. As I understand it the fallback for all failures is that the user can launch Marketplace app and find and install these apps themselves. The point is, after we have that list, this is a generic batch-install process.  

We could also defer the fetching of the manifestsUrls and have FTU just IAC to signal that this should happen, and not implement any of it in FTU. 

If we did some or all of this as an add-on, what would that look like?

Comment 3

[Etienne Segonzac (:etienne)]

(In reply to Sam Foster [:sfoster] from comment #2)
> I dont see any of this as operator-specific really. 

I meant: it's an operator use-case.
I'm sure many operator would use it though, it's a very useful feature!

But it's not something we're going to use/dogfood internally.

>The marketplace URL that
> responds with a list of apps to install would presumably live in a setting.
> (Reportedly this will be similar to operator shelf) Each operator would
> configure that as necessary. If that setting didn't exist, we would simply
> skip this whole step. I had thought of doing this like so: 
> 
> Somewhere in the FTU - once there's a connection - we request the list of
> apps/manifests from the marketplace. As the FTU is likely to be closed
> before install finishes, we need to send this list to something that can own
> the install - I'm not sure yet if this is the system app, the marketplace or
> what. So we would IAC to this app with the list of manifestURLs, it would
> trigger install of each and handle any error conditions. As I understand it
> the fallback for all failures is that the user can launch Marketplace app
> and find and install these apps themselves. The point is, after we have that
> list, this is a generic batch-install process.  
> 
> We could also defer the fetching of the manifestsUrls and have FTU just IAC
> to signal that this should happen, and not implement any of it in FTU. 

That does sound like a fair amount of code across 2 gaia apps.
1 more mozSetting hack, 1 more IAC hack.

I won't fight it. But it's fair to say that it is maintenance burden for a feature that won't get regularly dogfooded/tested.

> 
> If we did some or all of this as an add-on, what would that look like?

The addon code can live anywhere, be tweaked/forked easily by any partner, even after our code complete.
The addon listens to (presumably) the system's FTU events to trigger an XHR then some mozApps.mgmt calls, and that's it.

The system app already observes installs so the loading indicator and all we get "for free".

The user can uninstall the apps but also disable the addon after the first time run.
The only B2G part needed is a way to bundle-in default addons (a feature we would definitely use/dogfood internally).

* * *

My point is, we sure have a way to do that with settings + IAC, like we always do.
But maybe now is good time to invest in the general customization use-case (addons) instead of a very specific one :)

And again, it's not that big of a deal, I just didn't want to loose an opportunity to leverage addons just because "nobody mentioned it".

Comment 4

[Sam Foster [:sfoster] (he/him)]

I looked into this (comment #3) a little. In principle, it looks feasible to implement this with an add-on, except that add-ons are currently installed disabled. We would add the add-on to the app.list for the customization, and provided we could whitelist add-ons in the app.list (or whatever it would become once add-ons are no longer apps) AFAICT, the add-on should be able to participate in the first-use lifecyle and trigger the necessary requests and installs. 

As Etienne says, while late customization as currently defined is an operator-only use case, the ability to pre-install an add-on and truly get in on the ground floor even at first startup could prove extremely useful for devs, QA, dog-fooders etc. 

If we can go down this path, we just need that 'customization add-ons get enabled' special case. The rest neatly removes itself from our 2.5 critical path (contingent on add-ons themselves getting finalized and also making 2.5 release)

Comment 5

[Sam Foster [:sfoster] (he/him)]

Jonas, going down this route would commit us to add-ons in general being a thing in 2.5, allowing add-ons to form a part of a customization (as apps or whatever) and allowing add-ons in a customization to to be enabled by default. Also that add-ons would be injected and allowed to run code in a similarly early point in the start up lifecycle as today. Does all of that sounds reasonable for 2.5?

Comment 6

[Sam Foster [:sfoster] (he/him)]

Fabrice, do you have an opinion on doing late/local customization via an enabled-by-default add-on in the customization?

Comment 7

[[:fabrice] Fabrice Desré]

(In reply to Sam Foster [:sfoster] from comment #6)
> Fabrice, do you have an opinion on doing late/local customization via an
> enabled-by-default add-on in the customization?

I don't see any reason that would not work. The only drawback is that you'll have to wait a bit if you need more than content scripts for the new add-ons to be ready

Comment 8

[Tiffanie Shakespeare [:tif]]

Please find the spec here: 
https://drive.google.com/open?id=0B5nBT2-RgS4NcDJNSGdsMFc0Tk0

NI Matej to review strings on page 10 of the spec.

NI Naoki for assistence with the download edge cases on page 8 of the spec.

Thanks guys!

Comment 9

[Matej Novak [:matej]]

(In reply to Tiffanie Shakespeare from comment #8)
> Please find the spec here: 
> https://drive.google.com/open?id=0B5nBT2-RgS4NcDJNSGdsMFc0Tk0
> 
> NI Matej to review strings on page 10 of the spec.

All looks good, though it's my preference to say "the Marketplace" on that second screen. If this is consistent with what's elsewhere, I guess we can leave it, but it's always sounded strange to me to leave off the "the."

Comment 10

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

There's a couple of things we need to figure out if we want to use addons.

First off, we currently don't have anything like "built-in" addons. All addons show up to the user as a normal addon, and that can be disabled by the user. That is likely not something that we want for an addon which is pre-installed in order to handle this usecase.

Second, for any addon that we preinstall, it is critical that an update to FirefoxOS doesn't cause the addon or FirefoxOS to stop working properly.

The way addons work right now is that they just inject a piece of JS into one of the Gaia apps. That is a very fragile solution which is likely to break when FirefoxOS is updated since things like function names, variable names, element ids, and certified APIs can change from version to version of FirefoxOS.

A worst-case scenario here is that a built-in addon break so badly that the user is forced to enter safe-mode. This is especially bad if we fix the initial problem above since the user can't uninstall the built-in addon. And so the user would have to always start the phone in safe mode. That is likely not an acceptable state.

The new Addon implementation is supposed to help with this problem by creating certain APIs specifically designed for addons and which is kept stable for release to release. Addons using those would not break when FirefoxOS is updated, however such an addon would be limited by the capabilities of those APIs. So that's likely not going to help here.


If we can solve these problems then I think using addons is an option. But I'm not sure if it's easier to solve those problems than simply not using addons?

Comment 11

[Etienne Segonzac (:etienne)]

(In reply to Jonas Sicking (:sicking) from comment #10)
> There's a couple of things we need to figure out if we want to use addons.
> 
> First off, we currently don't have anything like "built-in" addons. All
> addons show up to the user as a normal addon, and that can be disabled by
> the user. That is likely not something that we want for an addon which is
> pre-installed in order to handle this usecase.

As long as we let the user uninstall the apps (which I think we are), why should we prevent the addon itself from being disabled?
(It's still guaranteed to run during the first run).

> 
> Second, for any addon that we preinstall, it is critical that an update to
> FirefoxOS doesn't cause the addon or FirefoxOS to stop working properly.
> 
> The way addons work right now is that they just inject a piece of JS into
> one of the Gaia apps. That is a very fragile solution which is likely to
> break when FirefoxOS is updated since things like function names, variable
> names, element ids, and certified APIs can change from version to version of
> FirefoxOS.
> 
> A worst-case scenario here is that a built-in addon break so badly that the
> user is forced to enter safe-mode. This is especially bad if we fix the
> initial problem above since the user can't uninstall the built-in addon. And
> so the user would have to always start the phone in safe mode. That is
> likely not an acceptable state.

Can we start with guidelines for built-in addons?
ie. only event listening / api calls. No dependency on the DOM etc...

> 
> If we can solve these problems then I think using addons is an option. But
> I'm not sure if it's easier to solve those problems than simply not using
> addons?

I'm worried about the maintenance cost.
It's been pointed out multiple time that we keep adding to the (already huge) system app, and I'm especially worried about adding code that won't regularly get tested under our foxfooding/beta program.

Ideally I'd like to move forward with an "addonable" piece of code.
ie. a JS file loaded into the system app, listening for FTU events and doing mozApps/mozIccManager calls.
This way we can easily extract it to an addon if we get built-in addon support.
And it'll probably be a lot easier to maintain anyway.

That said, while this approach fulfills the User Story, it doesn't fit the spec :/

Comment 12

[Etienne Segonzac (:etienne)]

(In reply to Tiffanie Shakespeare from comment #8)
> Please find the spec here: 
> https://drive.google.com/open?id=0B5nBT2-RgS4NcDJNSGdsMFc0Tk0
> 
> NI Matej to review strings on page 10 of the spec.
> 
> NI Naoki for assistence with the download edge cases on page 8 of the spec.
> 

Note that regardless of the technical implementation we choose we won't be able to diverge much between the usual app install notification/download UI and the late customization flow.

Basically, during FTU, app install requests are going to be made to gecko.
But after that everything at the homescreen and system (utility tray, banners) level will look like a regular app install, including download errors.

Comment 13

[Sam Foster [:sfoster] (he/him)]

> That said, while this approach fulfills the User Story, it doesn't fit the
> spec :/

Yeah I think we may need to revisit this spec. Basically, we are going to fetch the list of apps from the marketplace at the first opportunity - possibly during the FTU - and trigger an otherwise normal download and install process. IIRC that means no notifications during the FTU. However, it would be good to detect download/install failure and offer more specific instructions to the user than we normally do - as the user didn't trigger the installs they will have no idea what to do with errors otherwise.

Comment 14

[Tiffanie Shakespeare [:tif]]

Hey not sure who this is a question for but here goes...

In the spec, I currently have it so that a toast pops up at the end of FTE. We're in the process of revamping our notifications and these types of toasts are going away and are being replaced by regular notifications. 

So my question is, can we suppress the install notification for the apps that finish during FTE? (rather than barrage the user with notifications).

Thanks!

Comment 15

[Sam Foster [:sfoster] (he/him)]

(In reply to Tiffanie Shakespeare from comment #14)
> Hey not sure who this is a question for but here goes...
> 
> In the spec, I currently have it so that a toast pops up at the end of FTE.
> We're in the process of revamping our notifications and these types of
> toasts are going away and are being replaced by regular notifications. 
> 
> So my question is, can we suppress the install notification for the apps
> that finish during FTE? (rather than barrage the user with notifications).

I thought we were already doing this, but it appears not so I've filed bug 1189508. I'm happy to bring this into scope for this feature

Comment 16

[Tiffanie Shakespeare [:tif]]

Cool thanks!

Comment 17

[Sam Foster [:sfoster] (he/him)]

The linked specification is currently missing most of its screens ("whoops, there was a problem showing this page") can you update it Tiffanie?

Comment 18

[Tiffanie Shakespeare [:tif]]

Unfortunately, download is required. For some reason Drive is breaking the PDF and I can't seem to fix it. :(

Comment 19

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

QA needs a way to test this.  Is there a server variable/push that we could setup and do that's going to be implemented?

Comment 20

[Sam Foster [:sfoster] (he/him)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #19)
> QA needs a way to test this.  Is there a server variable/push that we could
> setup and do that's going to be implemented?

In bug 1193587 I'm using a latecustomization.manifestUrl setting, see there for the details I have - for now I'm using the operator shelves as a stand-in for the planned late customization endpoints, e.g. https://marketplace-dev.allizom.org/api/v2/feed/shelves/310/. I'll flag Chuck for more info on the Marketplace end of things.

Comment 21

[Chuck Harmston [:chuck]]

The operator shelf endpoints are a good stand-in for now; the late customization ones will be similar, though stripped down to provide just what is needed to keep the response slim. We’re flexible, and can include whatever is requested.

As far as 2.5 priorities go, getting addons into Marketplace seems to be higher than getting late customization work on our side done, so more specific implementation details likely won't be worked out for some time (though I'd be happy to take an educated guess if there are specific questions). Ping Maris (+cc) for any specifics on our roadmap.

Comment 22

[Sam Foster [:sfoster] (he/him)]

Adding bug 1203677 as a dependency. This should let is inject the late customization "free apps" step without getting all tangled up in the hardcoded steps in ftu/js/navigation.js

Comment 23

[Tiffanie Shakespeare [:tif]]

Just realized I could edit the user story so I put the links to the specs up there. NI me if there are any questions, spec updates, or patches/screenshots I need to review. 

Thanks guys!

Comment 24

[Sam Foster [:sfoster] (he/him)]

Tentatively blocking on bug 1180707. In order to do late customization as an add-on, we'll need the ability to inject some new strings into the FTU for display in the "Free Apps" panel. And if we want those localized, the l20n library allows us to do this fairly easily. Its not a hard dependency - but I'd like to keep the add-on option open.

Comment 25

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

(In reply to Etienne Segonzac (:etienne) from comment #11)

I guess we're just talking about an addon which downloads and installs some apps during FTE. If that's the case, then we don't need to worry about that addon being uninstalled later (since it no longer does anything), and we also don't need to worry about the user later updating FirefoxOS and breaking the addon (since the addon no longer needs to do anything).

Comment 26

[Sam Foster [:sfoster] (he/him)]

Tif, the spec calls for 'carrier', but looking at my test (AT&T) sim I get { carrier: '', operator: 'AT&T' }. I'm not sure in what circumstances carrier vs. operator would be defined. I'm proposing to check for carrier, then fallback to operator?

Comment 27

[David Durst [:ddurst]]

(cc ashort on the API from Marketplace)

sfoster -- are you using a stub API for now? Or are you waiting for us (Markeplace) to provide the spec?

Comment 28

[Sam Foster [:sfoster] (he/him)]

(In reply to David Durst [:ddurst] from comment #27)
> (cc ashort on the API from Marketplace)
> 
> sfoster -- are you using a stub API for now? Or are you waiting for us
> (Markeplace) to provide the spec?

I'm using the operator shelf API for now, e.g. https://marketplace-dev.allizom.org/api/v2/feed/shelves/2/
It has more data than I need but works fine for now. Mostly I need (localized/localizable) app name, manifest_url, icon(s). The url is presently coming from settings.

Comment 29

[Tiffanie Shakespeare [:tif]]

As same and I discussed in IRC - operator is the one we should go with. 

Thanks!

Comment 30

[Sam Foster [:sfoster] (he/him)]

I want to clarify how this is going to work - as implemented in bug 1193587. During the first start up, the FTU app will read a latecustomization.manifestURL setting. It will then look at the SIM for the operator/mcc/mnc. If it gets both those things, it will make request to the marketplace, building the URL + querystring based on that latecustomization.manifestURL setting value and the operator/carrier info. 

If that operator has provided late customization apps, that list gets downloaded, and each is installed. The initial list request and subsequent app downloads are zero-rated. 

The latecustomization.manifestURL setting does not have a default value. Here's where I'm not clear. If a user buys a device from a carrier, it seems reasonable (to me) to require that carrier to populate this setting on the phones they stock. If we want to avoid that and bake the marketplace URL in by default, then *all* devices at FTU will issue the request in the background.  If a new user's carrier isnt using late customization and unless they use Wifi, they will be hitting their data plan. Its a single request - but its not a result of any action the user took, and unless they disable data and don't connect to wifi (or run FTU without the SIM card) they have no way to opt-out. Maybe that's OK, I'm not sure. 

Having chatted briefly with Naoki last week, I think we may have a disconnect here. From the FTU, without the setting or some carrier-specific change to what gets flashed to the device, we have no way of knowing if late customization applies. How did we expect this to work? I can make the change to avoid needing this setting (or ensure it is pre-populated with the marketplace URL) if necessary, but I would like to understand it first.

Comment 31

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I have another set of questions.

How does late customization work in the following scenarios:
1) if the sim card is removed and then you go through the FTU and then place the SIM card in.
2) if the sim card data is turned off through the FTU.

Comment 32

[Sam Foster [:sfoster] (he/him)]

> How does late customization work in the following scenarios:
> 1) if the sim card is removed and then you go through the FTU and then place
> the SIM card in.

My understanding is that when you next open the marketplace app, you will get the banner "Your carrier has apps for you". We dont have any dectection or notification in the system app. 

> 2) if the sim card data is turned off through the FTU.

We can still get the mcc/mnc, if the user connects to a wifi network we will attempt to fetch the list and begin installing. If we never get online during the FTU, the Free Apps panel is not added and the user remains unaware until they open the Marketplace app (for which they need to be online).

Comment 33

[Wilfred Mathanaraj [:WDM]]

"bake the marketplace URL in by default, then *all* devices at FTU will issue the request in the background."

is there a lot of work to do this? I would prefer this as we offer the marketplace - we offer the shelf and we have the device OS - we should just fix it in without operator involvement.

The amount of data used up by this request is neglible - but the download of the apps is whitelisted.

Comment 34

[Sam Foster [:sfoster] (he/him)]

(In reply to Wilfred Mathanaraj [:WDM] from comment #33)
> "bake the marketplace URL in by default, then *all* devices at FTU will
> issue the request in the background."
> 
> is there a lot of work to do this? I would prefer this as we offer the
> marketplace - we offer the shelf and we have the device OS - we should just
> fix it in without operator involvement.
> 
> The amount of data used up by this request is neglible - but the download of
> the apps is whitelisted.

Ok thanks, No, its not a lot of work, just a matter of providing a default value for the setting. Thanks.

Comment 35

[Sam Foster [:sfoster] (he/him)]

(In reply to Wilfred Mathanaraj [:WDM] from comment #33)
> "bake the marketplace URL in by default, then *all* devices at FTU will
> issue the request in the background."
> 
> is there a lot of work to do this? I would prefer this as we offer the
> marketplace - we offer the shelf and we have the device OS - we should just
> fix it in without operator involvement.

Once consequence of this btw is that its not possible to get into the state where we would show the alternate "Free Apps" screen content where we suggest the user find the apps in the marketplace later. We know the operator is offering late customization apps by retrieving the manifest/app-list from the marketplace. If that list is empty or an error or we've not been able to retrieve it, we skip the panel entirely. We have no way of knowing there are apps to install until we get that response, so its a bit of a Schrödinger's cat situation. We might want to update the spec to reflect this to avoid confusion.

Comment 36

[Tiffanie Shakespeare [:tif]]

So basically we don't know if there are apps to install until we are told there are apps to install which if we are never told we don't know why (b/c there was an error or there are no apps).

Are there any scenarios where the user could still get to that screen? We know the apps are there but for some reason we aren't able to download them.

Would the Marketplace banner be unaffected? Since we aren't doing automatic downloads, if MP detects that there are apps available and are not on the device, then the banner is shown.

Thanks Sam!

Comment 37

[Sam Foster [:sfoster] (he/him)]

(In reply to Tiffanie Shakespeare [:tif] UX from comment #36)
 
> Are there any scenarios where the user could still get to that screen? We
> know the apps are there but for some reason we aren't able to download them.

We answer the "enabled?" and "are there apps" question at the same time with that request to the marketplace. I guess in theory we could lose the connection between getting the customization app list, and arriving on the free apps panel. That is worth checking. In that case we know the apps, we may or may not have been able to download their icons and we will be unable to install at that time. 

So, STR would be: 
1. Make a wifi/data connection from the FTU. 
2. Wait a few minutes on that panel, then disconnect (in the Wifi panel we don't actually have a disconnect button so you'll need to attempt to connect to another network and provide the wrong credentials)
3. Click through to the "Free Apps" panel. It should display the "install later from the marketplace" text and no app list.

Comment 38

[Sam Foster [:sfoster] (he/him)]

Paul, could you look over the Gaia-side implementation for this? In bug 1193587 we landed the ability for a late customization to install apps during FTU. In mozSettings we have a url which the FTU uses to fetch a list of manifest URLs from the marketplace. The system app is given this same list so that when the FTU triggers install(), it can automatically grants the request. This approval is contingent on the FTU running, and the manifestURL appearing in the late customization list. 

Let me know if you have any questions.

Comment 39

[Paul Theriault [:pauljt] (no longer reading bugmail)]

Freddy, can you take a look at this? This sounds ok in theory. Im a little concerned about the system app being able to silently install apps - thats new, and we should make sure that there no chance of this mechanism being abused to silently install privileged content. I guess this is all over IAC though, so only certified apps could access this functionality.

Sam - anything you could to document the flow here a little more would be helpful (maybe a bit more detail of the steps involved between the FTU and marketplace, and also the basic IAC steps would be good.)

Some thought while i've got 5 mins:

1. Manifest Url Validation
I guess the key part is here though:
https://github.com/mozilla-b2g/gaia/pull/31427/files#diff-584e849fe9a5b8eac868ccf88973dbc4R164

What does this line do:
var isCustomizing = Service.query('ftuCustomizationContains',
+                                            manifestURL);

Is there any chance that we could have an issue if an attacker create an install url like:

http://attacker.com/mainfest.webapp?http://marketplace.firefox.com/whitelistedFTUapp/manifest.webapp

That would be my main concern - IE an attacker finding a way to get their app past the whitelist, and automatically installed. 


2. Whats the impact of being able to control latecustomization.url?
I presume that the ability to change this on its own isnt a vulnerability, ie. the attack would also need to convince the user to relaunch the FTU? (though I wonder if thats launchable via web activities?). Its not a direct threat, i just want to understand how bad it would be if an app could actually change this url. (There are probably much worse things an app could do with arbitrary settings access though...)

Thats the only things that come to mind at a 10 mins glance, but freddy if you could have a look too that would be great.


Thanks!

Comment 40

[Sam Foster [:sfoster] (he/him)]

> Sam - anything you could to document the flow here a little more would be
> helpful (maybe a bit more detail of the steps involved between the FTU and
> marketplace, and also the basic IAC steps would be good.)

Sure, let me lay it out, basically there are 2 parts here on the Gaia side: the system app and the FTU app. 

The system app's FtuLauncher will conditionally launch the FTU and also start a submodule (in the system app) which is the FtuLateCustomization module. 
On the FTU side, we have a new panel LateCustomization, which fetches results from the url it finds at latecustomization.url, and sends the list over the ftucomms IAC connection. It then calls navigator.mozApps.install() for each of the manifest URL in that list. It waits for one to complete before going on to the other. 

The normal (not changed in this patch) system app lifecycle includes starting the FtuLauncher (system app) which checks if there's a ftu.manifestURL setting and if asyncStorage.getItem('ftu.enabled') is true. That triggers the launching of the FTU app. At the same time we now check the latecustomization.url setting, and if that is truthy, we start the FtuLateCustomization module (system app) which is basically an IAC client. 

FtuLateCustomization uses the existing ftu-comms connection and listens for a message on that channel with type 'late-customization-apps'. 

The system app has a dispatcher/loose-coupling mechanism - Service.query. It doesnt do IAC, its purely internal to the system app for communication between modules. A system app module can register itself to answer certain questions. If the module doesnt exist or isnt started, the answer will always be falsey. The Service.query('ftuCustomizationContains', manifestURL) gets routed to the FtuLateCustomization module. If that module is running and the manifestURL is on the list it got from the FTU, it returns true. 

In the system app's AppInstallManager is where we register the mozChromeEvent handlers to catch all attempted installs and put up the confirmation prompt. A truthy response to Service.query('ftuCustomizationContains', manifestURL) will cause a granted event to be dispatched - bypassing the normal user-facing prompt. 

When the FTU is done, we shut down the FtuLateCustomization module, so thereafter Service.query('ftuCustomizationContains', manifestURL) will always return false. Also note that the FTU app must be launched from the FtuLauncher for the FtuLateCustomization module to get started. If you did manage to set a new value for latecustomization.url and start the FTU again, it would indeed fetch results from the URL and send them over IAC. But the system app's FtuLateCustomization wouldn't be listening and you would get the normal app install prompt. 

This patch adds one more setting: latecustomization.operatorInfo. As the marketplace API requires valid mnc and mcc values, which match a late customization list configured for the given carrier, I added this override to make testing practical. You can populate your profile's setting (e.g. through custom-settings.json) with values that will get sent along in the querystring to the marketplace, so you can try known-good / known-empty variations without needing access to the particular SIM cards. 

I think that covers it. The Marketplace app uses the same data on the server, and obviously uses the mozApps api to see which apps have been installed, but doesnt otherwise interact at all with the system/FTU piece.


> Is there any chance that we could have an issue if an attacker create an
> install url like:
> 
> http://attacker.com/mainfest.webapp?http://marketplace.firefox.com/
> whitelistedFTUapp/manifest.webapp
> 
> That would be my main concern - IE an attacker finding a way to get their
> app past the whitelist, and automatically installed. 

We are trusting that navigator.mozSettings are uncompromised - mitigated to an extent by this whole process only taking place at first-run. We don't validate the URL found in that setting at all - by design, so a partner or whoever could stand up their own marketplace or similar service. We'll use this to integration test against a mock server. 

> 2. Whats the impact of being able to control latecustomization.url?
> I presume that the ability to change this on its own isnt a vulnerability,
> ie. the attack would also need to convince the user to relaunch the FTU?
> (though I wonder if thats launchable via web activities?). Its not a direct
> threat, i just want to understand how bad it would be if an app could
> actually change this url. (There are probably much worse things an app could
> do with arbitrary settings access though...)

Yeah. As I said, launching the FTU app isn't enough, you have to convince the system app to launch it for you. If you got bad settings onto a new device or newly-flashed device, then you could co-opt late customization to install things without the user having granted it. But then, if you could do that, you could presumably just install them directly on the device and save the round trip?

Comment 41

[Paul Theriault [:pauljt] (no longer reading bugmail)]

(In reply to Sam Foster [:sfoster] from comment #40)
> > Sam - anything you could to document the flow here a little more would be
> > helpful (maybe a bit more detail of the steps involved between the FTU and
> > marketplace, and also the basic IAC steps would be good.)
> 
> Sure, let me lay it out, basically there are 2 parts here on the Gaia side:
> the system app and the FTU app. 
> 
> The system app's FtuLauncher will conditionally launch the FTU and also
> start a submodule (in the system app) which is the FtuLateCustomization
> module. 
> On the FTU side, we have a new panel LateCustomization, which fetches
> results from the url it finds at latecustomization.url, and sends the list
> over the ftucomms IAC connection. It then calls navigator.mozApps.install()
> for each of the manifest URL in that list. It waits for one to complete
> before going on to the other. 
> 
> The normal (not changed in this patch) system app lifecycle includes
> starting the FtuLauncher (system app) which checks if there's a
> ftu.manifestURL setting and if asyncStorage.getItem('ftu.enabled') is true.
> That triggers the launching of the FTU app. At the same time we now check
> the latecustomization.url setting, and if that is truthy, we start the
> FtuLateCustomization module (system app) which is basically an IAC client. 
> 
> FtuLateCustomization uses the existing ftu-comms connection and listens for
> a message on that channel with type 'late-customization-apps'. 
> 
> The system app has a dispatcher/loose-coupling mechanism - Service.query. It
> doesnt do IAC, its purely internal to the system app for communication
> between modules. A system app module can register itself to answer certain
> questions. If the module doesnt exist or isnt started, the answer will
> always be falsey. The Service.query('ftuCustomizationContains', manifestURL)
> gets routed to the FtuLateCustomization module. If that module is running
> and the manifestURL is on the list it got from the FTU, it returns true. 
> 
> In the system app's AppInstallManager is where we register the
> mozChromeEvent handlers to catch all attempted installs and put up the
> confirmation prompt. A truthy response to
> Service.query('ftuCustomizationContains', manifestURL) will cause a granted
> event to be dispatched - bypassing the normal user-facing prompt. 
> 
> When the FTU is done, we shut down the FtuLateCustomization module, so
> thereafter Service.query('ftuCustomizationContains', manifestURL) will
> always return false. Also note that the FTU app must be launched from the
> FtuLauncher for the FtuLateCustomization module to get started. If you did
> manage to set a new value for latecustomization.url and start the FTU again,
> it would indeed fetch results from the URL and send them over IAC. But the
> system app's FtuLateCustomization wouldn't be listening and you would get
> the normal app install prompt. 
> 
> This patch adds one more setting: latecustomization.operatorInfo. As the
> marketplace API requires valid mnc and mcc values, which match a late
> customization list configured for the given carrier, I added this override
> to make testing practical. You can populate your profile's setting (e.g.
> through custom-settings.json) with values that will get sent along in the
> querystring to the marketplace, so you can try known-good / known-empty
> variations without needing access to the particular SIM cards. 
> 
> I think that covers it. The Marketplace app uses the same data on the
> server, and obviously uses the mozApps api to see which apps have been
> installed, but doesnt otherwise interact at all with the system/FTU piece.

Thanks for the info, that sounds like a decent plan to me.

> 
> 
> > Is there any chance that we could have an issue if an attacker create an
> > install url like:
> > 
> > http://attacker.com/mainfest.webapp?http://marketplace.firefox.com/
> > whitelistedFTUapp/manifest.webapp
> > 
> > That would be my main concern - IE an attacker finding a way to get their
> > app past the whitelist, and automatically installed. 
> 
> We are trusting that navigator.mozSettings are uncompromised - mitigated to
> an extent by this whole process only taking place at first-run. We don't
> validate the URL found in that setting at all - by design, so a partner or
> whoever could stand up their own marketplace or similar service. We'll use
> this to integration test against a mock server. 

Thanks for the info. I was mainly just checking that service.query was an exact match, not a substring match. WHich it is:
https://github.com/mozilla-b2g/gaia/pull/31427/files#diff-ca4d7a644667c2a190da1b045dec81b8R52

So all good.


> 
> > 2. Whats the impact of being able to control latecustomization.url?
> > I presume that the ability to change this on its own isnt a vulnerability,
> > ie. the attack would also need to convince the user to relaunch the FTU?
> > (though I wonder if thats launchable via web activities?). Its not a direct
> > threat, i just want to understand how bad it would be if an app could
> > actually change this url. (There are probably much worse things an app could
> > do with arbitrary settings access though...)
> 
> Yeah. As I said, launching the FTU app isn't enough, you have to convince
> the system app to launch it for you. If you got bad settings onto a new
> device or newly-flashed device, then you could co-opt late customization to
> install things without the user having granted it. But then, if you could do
> that, you could presumably just install them directly on the device and save
> the round trip?

Perfect, thanks for this detailed explanation. I've had another skim through the code to be sure but it all looks good to me.

Comment 42

[Frederik Braun [:freddy]]

This might be late feedback, but given the history of apps that come pre-installed on phones these days, do we not want to provide the user a way to skip this in the FTU?
Not sure whether selectively installing apps should be in scope, but skipping the install should be possible.

Comment 43

[Sam Foster [:sfoster] (he/him)]

(In reply to Frederik Braun [:freddyb] from comment #42)
> This might be late feedback, but given the history of apps that come
> pre-installed on phones these days, do we not want to provide the user a way
> to skip this in the FTU?
> Not sure whether selectively installing apps should be in scope, but
> skipping the install should be possible.

We have had this discussion a number of times, and I actually initial prototyped this with checkboxes next to each app. But the direction from product management has been clear - strategy/policy-wise this is no different to a market customization, it just happens later when the user gets online with their new device.

Comment 44

[Teri Wen [:twen]]

Tests for FTU part completed.
https://moztrap.mozilla.org/runtests/run/8229/env/27835/