1091786 - Remove MIME type restriction for app manifests

Status: RESOLVED WONTFIX

(Core Graveyard :: DOM: Apps, defect)

Description

[Ben Francis [:benfrancis]]

Following discussion in bug 1075716 it is proposed that we remove the restriction that app manifests must be served with a particular MIME type in order to be installed from an origin other than the origin of the manifest.

Comment 1

[Marcos Caceres [:marcosc]]

Just to be clear, you do want to enforce the MIME type for Mozilla proprietary format. This is to make it clear, through the MIME type, that this is an unregistered/vendor-specific format (as hosted apps are primarily acquired with the assistance of the marketplace). 

For the standardized format, there should be no restriction.

Comment 2

[Ben Francis [:benfrancis]]

Attachment: WIP Patch

Hi Fabrice, I thought I'd take a stab at this. Is this patch going in the right direction? What else needs doing? What testing would be required to get this landed? Is it just a case of removing tests like the one added in bug 1075716?

Comment 3

[[:fabrice] Fabrice Desré]

Comment on attachment 8566806 [details] [diff] [review]
WIP Patch

Review of attachment 8566806 [details] [diff] [review]:
-----------------------------------------------------------------

This patch removes all the content type checks. I'm fine with that, Marcos seems to think we need to keep them for owa manifests.
Jonas, what do you think?

Comment 4

[Bill Walker [:bwalker] [@wfwalker]]

(In reply to Fabrice Desré [:fabrice] from comment #3)
> Comment on attachment 8566806 [details] [diff] [review]
> WIP Patch
> 
> Review of attachment 8566806 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> This patch removes all the content type checks. I'm fine with that, Marcos
> seems to think we need to keep them for owa manifests.
> Jonas, what do you think?

The Mozilla proprietary manifest offers apps some privileges that the W3C manifest currently doesn't. The original point of these restrictions was to believe that the server owner is the one who created and posted the manifest, so we'd feel good about granting those privileges (higher limit on localStorage, etc).

Comment 5

[Marcos Caceres [:marcosc]]

(In reply to Fabrice Desré [:fabrice] from comment #3)
> This patch removes all the content type checks. I'm fine with that, Marcos
> seems to think we need to keep them for owa manifests.
> Jonas, what do you think?

As the two formats are incompatible, you will need some means to distinguish between them. You can do this through either the MIME type or through some other means (e.g., a flag) when installing through b2g. You can always determine if it's a web manifest because it will have come from a <link rel=manifest> (in the future, maybe even a HTTP `Link`). The MIME type makes the distinction explicit, but it's probably a burden for developers. So, as long as there is some clear means in b2g to distinguish the manifest types (and how they are retrieved an processed), it should be ok to drop the MIME type.

Comment 7

[Ben Francis [:benfrancis]]

So far the only incompatibility I have found between the two formats that has been problematic is that the default display mode for the W3C Manifest is "browser", but the default for mozApps is to have no browser chrome.

I'm attempting to implement the W3C manifest in a way that the mozApps manifest is just a proprietary extension of it, with the hope that over time we can deprecate the proprietary properties. This allows us to use the same code paths for the installation of apps, which has lots of benefits.

The problem with the MIME type check is that it gets applied whenever the manifest is from a separate origin to the app installing it. The way the W3C Manifest support is currently implemented in Firefox OS means that this is always the case, because it's installed from Gaia's bookmark app. Gecko doesn't know whether the manifest passed to the install() method comes from a manifest link relation so that can't be used to differentiate manifest formats.

This means that installations of W3C manifests which are not served with the optional application/manifest+json content type will fail, including some real examples of web apps in the wild like IRCCloud and CNET.

The easiest solution is to drop the MIME type restriction, which I thought Fabrice wanted to do anyway. It doesn't seem to offer any real security and is an inconvenience to developers.

Alternatively, we need to provide Gecko with another way of differentiating between manifest formats other than the optional MIME type (e.g. an argument passed to the install method), or a way for the installation to be carried out in the context of the installed app rather than the installing app.

Comment 8

[Ben Francis [:benfrancis]]

Paul, do you have an opinion on whether it's safe to drop the MIME type restriction on app manifests?

Comment 9

[Paul Theriault [:pauljt] (no longer reading bugmail)]

(In reply to Ben Francis [:benfrancis] from comment #8)
> Paul, do you have an opinion on whether it's safe to drop the MIME type
> restriction on app manifests?

If we drop the restriction, any website which allows the ability to upload files to its domain, could be made into an app without its knowledge or consent i.e. Exactly what Bill points out in comment 4. For regular web apps (i.e. not privileged) we have endeavored to prevent damage from a malicious manifest, but I don't see why we would drop this restriction. Why do you say that " it doesn't offer any real security"? 

> The problem with the MIME type check is that it gets applied whenever the
> manifest is from a separate origin to the app installing it. The way the W3C
> Manifest support is currently implemented in Firefox OS means that this is
> always the case, because it's installed from Gaia's bookmark app. Gecko
> doesn't know whether the manifest passed to the install() method comes from
> a manifest link relation so that can't be used to differentiate manifest
> formats.
> 
> This means that installations of W3C manifests which are not served with the
> optional application/manifest+json content type will fail, including some
> real examples of web apps in the wild like IRCCloud and CNET.

So are you saying: instead of enforcing the mime-type restriction, we should allow all installs, regardless of mime-type. But then decide upon installation: ie if it has a type of "application/x-web-app-manifest+json" then we treat it as a b2g manifest, otherwise treat it as a w3c manifest? 

In that case, can we just ignore all the Mozilla app manifest attributes if the mime-type is not "application/x-web-app-manifest+json"? 

That would seem safe to me (so long as w3c manifests can't later add the b2g features somehow)? Or is the intent to treat the apps the same after install?

Comment 10

[Ben Francis [:benfrancis]]

(In reply to Paul Theriault [:pauljt] from comment #9)
> If we drop the restriction, any website which allows the ability to upload
> files to its domain, could be made into an app without its knowledge or
> consent i.e. Exactly what Bill points out in comment 4. For regular web apps
> (i.e. not privileged) we have endeavored to prevent damage from a malicious
> manifest, but I don't see why we would drop this restriction.

OK, I understand.

Unfortunately this basically prevents us from using the existing mozApps installation API for installing web apps with a W3C manifest served without the (optional) Content Type header being set, because Gaia needs to provide the UI to install the app and a Gaia app is always going to be cross-origin with the app being installed.

We may need a new more privileged API which can either bypass the Content Type check, or carry out the installation as if it came from the origin of the app. In order to fully support all the CSP requirements and support edge cases like manifests behind HTTP authentication or cookies we're probably going to need a new API anyway, to allow fetching a manifest in the context of the app, rather than from the context of a Gaia app (e.g. bug 1086666 and bug 1086669). This is just going to be a lot more work.

> So are you saying: instead of enforcing the mime-type restriction, we should
> allow all installs, regardless of mime-type. But then decide upon
> installation: ie if it has a type of "application/x-web-app-manifest+json"
> then we treat it as a b2g manifest, otherwise treat it as a w3c manifest?
> 
> In that case, can we just ignore all the Mozilla app manifest attributes if
> the mime-type is not "application/x-web-app-manifest+json"?
> 
> That would seem safe to me (so long as w3c manifests can't later add the b2g
> features somehow)? Or is the intent to treat the apps the same after install?

Actually the intent was to treat the apps the same, but what you say makes sense. The problem is that the manifest formats are very similar, it's not straightforward to determine which format is being used when the Content Type is optional. I'd hoped it might be possible for the mozApps manifest format to be considered a proprietary extension of the W3C format and for it to be possible to mix and match properties, but maybe that won't work.

Which properties of the mozApps manifest are considered harmful? We could do something crude like remove the permissions property of manifests served without the right Content Type for example, but that might create confusing results for developers who can't figure out why their app won't install.


Basically we have code in Gecko to fetch and sanitise W3C manifests, but we really still need a way to expose this functionality to Gaia via an API and hook it up to the app registry.

Jonas, Fabrice, any views on how this should work?

Comment 11

[Ben Francis [:benfrancis]]

Note from talking to Fabrice, one solution may be:

Apps.mgmt.installFromPage('http://example.com/index.html')

A new privileged install API passes the document URL rather than manifest URL, then Gecko fetches the page, follows the manifest link relation and installs that way. This could allow us to bypass the MIME type check.

Comment 12

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

See here for why I think that we need a mimetype restriction for manifests:
https://github.com/w3c/manifest/issues/272#issuecomment-87495930

Comment 13

[Ben Francis [:benfrancis]]

Thanks Jonas. If we can get that into the W3C spec then that works for me!

Unfortunately currently the spec says the Content Type header is optional, and the majority of the existing manifests I have come across on the web so far do not bother setting this header. This unfortunately means that at the moment most existing web apps can not be installed in Firefox OS.

To clarify, if the manifest has to be same origin with the app and the Content Type header is compulsory, then this is effectively the same as mozApps and the Apps.mgmt.installFromPage() method would not be required, Apps.install() would suffice?

Comment 14

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Comment on attachment 8566806 [details] [diff] [review]
WIP Patch

Of course we should follow what the spec says, but for now I think we should keep this.

Comment 15

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Right. We, as the people working on the spec, need to make a decision about if a manifest represents an app or not. And if it doesn't represent an app, then why we're not just inlining the information in the page markup.

Comment 16

[Marcos Caceres [:marcosc]]

(In reply to Jonas Sicking (:sicking) from comment #15)
> Right. We, as the people working on the spec, need to make a decision about
> if a manifest represents an app or not. And if it doesn't represent an app,
> then why we're not just inlining the information in the page markup.

The manifest spec allows, and encourages, in-lining information into the page's markup (and always has!) [1]. This has always been the fallback model from the beginning: that in the absence of a manifest, you use what is in the page - and if something is missing in the manifest, you look for it in the page (e.g., the application-name, icons, or fallback to proprietary meta tags). This is why you can even have an empty manifest ("{ }") or no manifest at all and still have a totally valid/installable web application (so long as it meets the UA's installability criteria or the user wants to bookmark it manually).

This goes back to my initial resistance to having this specification at all. There is some limited value in having the manifest (as you argued on the TAG list [2] - which answers your question: "why we're not just in-lining the information in the page markup"? because, as you argued, it gives devs some tiny benefits, like cachability, reduced redundancy in markup, hints that are needed at boot time, and whatever else you listed in [2]). However, arbitrary restrictions like enforcing same-origin and imposing a MIME type just discourages developers from using a manifest - and they will hence just stick to the meta tags (because the meta tags do the same thing without any restrictions).

To address your question: "if a manifest represents an app or not"? the answer is, and has always been, and will continue to be, "no". All web pages are web apps - the presence of a manifest doesn't change that (manifest is just a limited set of metadata - some stuff that "progressively enhances" a set of documents within a range of URLs). This is the same as every iOS app is an app and every Android app is an app, no matter how crap the quality is - what makes an "app" and "app" for a platform is the use of that platform's technology (in our case, "HTML5"). What a user (or user agent) determines to be a high-quality "appy-ness" of a web site is determined by criteria (what we are calling the installability signals in the spec), but they can be mostly independent of the metadata in the manifest. 

So, we could basically have both JSON and HTML meta tags defined in the manifest spec (probably a good thing). Note that content producers, like the Financial Times, are pushing for having meta tag fallbacks for all JSON-defined metadata [3], given their not so great experience using the manifest. I've heard from others privately that if manifest are annoying to use, then they will just use the existing meta tags (which I think it's totally fair). 

I still see some value in the manifest spec: as a place to lump together metadata for a range of pages. However, I don't see a manifest being special in any way (it's just hints to a page that assists when an app is being booted-up) - and I certainly *do not* see manifests as something that makes a web application an "app". I hope we can get agreement on this view, as it would overcome a number of issues. 

So, posting your question differently: can a web site be an "application" without a manifest? I hope the answer is, "yes". It might "no" for a FxOS app (which is totally fine, IMO), but hopefully it will be "yes" for web apps.  
   

[1] https://github.com/w3c/manifest/issues/97
[2] https://lists.w3.org/Archives/Public/www-tag/2014Feb/0039.html
[3] https://github.com/w3c/manifest/issues/350

Comment 17

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

As far as I can tell, none of those benefits remain if the manifest lives on a CDN and thus changing URL any time its contents changes. I.e. as long as the manifest doesn't have an identity in the form of a stable URL.

But yes, it appears that we've always had different goals with the manifest. We should probably decide on which goals we have before we move further forward.

Comment 18

[Ben Francis [:benfrancis]]

Unassigning myself as it's not clear whether this is still needed.

Comment 19

[Paul Theriault [:pauljt] (no longer reading bugmail)]

With webapps.jsm going away as part of transition I assume this is no longer needed.

Comment 20

[Sylvestre Ledru [:Sylvestre]]

Core Graveyard / DOM: Apps is inactive. Closing all bugs in this component.