If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Allow for testing of multiple manifests on different app origins on a hosted server

VERIFIED FIXED

Status

Mozilla QA
Infrastructure
VERIFIED FIXED
6 years ago
5 years ago

People

(Reporter: jsmith, Unassigned)

Tracking

Details

(Reporter)

Description

6 years ago
A reoccurring theme of questions that appears to keep happening is a need to find a location to host application test cases on a public server, so that we can hook up these applications to places where we can install the apps for testing purposes. Likely as we go into the future, the need will grow greater overtime, as Android testing will need it, Desktop testing will need it, API testing will need it, etc. The needs for the applications themselves will grow overtime as well.

The existing solution that is built for this hosts a bunch of applications on separate ports on a server. This could work to get the apps up and running, but opening many ports overtime spawns a security risk as more ports open up overtime. A safer solution could be using sub-domains instead on a server, but we would have to build out a solution to support this.

A second question behind this is figuring out where to host test applications. A staging server does exist for web applications for testing. However, mozqa.com traditionally has been used for hosting test cases for desktop and mobile QA. An intermixing of this solution could be hosting the test applications on the staging server and ensuring that mozqa.com can point to the test application directory on the staging server. An alternative approach would be to consider using mozqa.com instead as the hosting solution - We would have to talk with Anthony Hughes if something like this is even possible.
(Reporter)

Comment 1

6 years ago
Heroku and Google App Engine have also been considered as options.
(Reporter)

Comment 2

6 years ago
Note that these test case apps do not need to test the underlying runtime itself - making mozqa.com a web app could solve that problem in a simple manner and allow us to reuse existing test cases for desktop and mobile.
(Reporter)

Updated

6 years ago
Blocks: 733631
(Reporter)

Comment 3

6 years ago
This bug might be appropriate to file under Mozilla QA --> Infrastructure, as this applies to the organization as a whole, not the web apps component itself. Can somebody confirm if that's a valid place to move this?
(Reporter)

Updated

6 years ago
No longer blocks: 733631
(Reporter)

Comment 4

6 years ago
Moving to Mozilla QA --> Infrastructure, as this is infrastructure that applies to a large majority of the QA team as a whole (e.g. desktop webRT in firefox, B2G, mobile webRT in fennec native).
Assignee: dclarke → nobody
Component: Infrastructure → Infrastructure
Product: Web Apps → Mozilla QA
QA Contact: general → infrastructure
As an example of hosting an application publicly, I've setup https://github.com/ianb/app1.ianbicking.org which publishes to http://app1.ianbicking.org via github pages.
In the case of mozqa.com we could easily create virtual hosts for sub domains and let those point to the litmus-data checkout folder under '/data/webapps/%app-id%'. That would give all apps we want to test an isolated environment.

I'm not sure how dynamic the whole process should be.
I've had a number of cases where I've needed to test something using a global visible URL, e.g., when uploading to the Marketplace.  I.e., it can't be localhost and can't be on the VPN.  In that case I find I have to upload over and over to debug, because I can't debug locally.  I just mention this because it's helpful to be able to make changes very quickly.
https://etherpad.mozilla.org/apps-test-infrastructure

I created an etherpad with some thoughts, thought this might be something to work on for the next two weeks ?
Assignee: nobody → dclarke
(Reporter)

Comment 9

6 years ago
I've added my feedback to the etherpad. In general, I'm in favor of Henrik's proposal in comment 6, as it's possible to could be quite simple to pull off with utilizing mozqa.com (a server already in our control, setup, and integrated with Mozilla's hg repository).
(Reporter)

Comment 10

6 years ago
Reword on comment 9 - In general, I'm in favor of Henrik's proposal in comment 6, as it's simple and utilizing infrastructure we've already setup and have control over (e.g. Henrik, I, and other admin it, it's integrated with Mozilla's hg repository on hg.mozilla.org).
Gregory Koberger also set this up: http://testmanifest.com/
(Reporter)

Comment 12

5 years ago
(In reply to Ian Bicking (:ianb) from comment #11)
> Gregory Koberger also set this up: http://testmanifest.com/

That's pretty cool. Greg - Curious, what's your thoughts on an approach to get test app manifests on different subdomains? I'm looking for an approach that's simple and works with our existing test infrastructure.
That site will let you use any subdomain. So, you can just do http://<random-hash>.testmanifest.com/manifest.webapp, and it will serve a working manifest. If you want to edit the manifest, just visit it in a browser.

I don't explain the site, however you can also namespace them (using a dash). So, let's say you edit abc.testmanifest.com. abc-1, abc-test, abc-alksdf, etc will all be served off a different subdomain, but will inherit the same "abc" manifest.

If you have any feature requests, I can make it happen.
(Reporter)

Comment 14

5 years ago
(In reply to Gregory Koberger (:gkoberger) from comment #13)
> That site will let you use any subdomain. So, you can just do
> http://<random-hash>.testmanifest.com/manifest.webapp, and it will serve a
> working manifest. If you want to edit the manifest, just visit it in a
> browser.

Awesome. I've hit one issue - I've modified the manifest here (http://chimpanzee6282.testmanifest.com/manifest.webapp) to allow installs from anywhere. I tried installing the app natively on firefox nightly, but I had no luck. Any ideas? If this a problem on our end (desktop webRT), ping me - I can help out with this to figure out if this is a bug on our end. If this is issue is resolved, it will solve the problem in this bug (allowing us to test manifests on different subdomains).

> 
> I don't explain the site, however you can also namespace them (using a
> dash). So, let's say you edit abc.testmanifest.com. abc-1, abc-test,
> abc-alksdf, etc will all be served off a different subdomain, but will
> inherit the same "abc" manifest.
> 
> If you have any feature requests, I can make it happen.

Feature requests:

For every subdomain created, provide a way to install the web application on firefox nightly on the subdomain. This means that if I have a manifest on http://chimpanzee6282.testmanifest.com, then include a button on the page to allow that web app to be installed using navigator.mozapps.install.

Add a level of permissions to testmanifest.com to allow a user to request a manifest, lock it with an account (maybe browser ID). A locked manifest can only be edited by the user who locked the manifest. A locked manifest is read only to everyone else. A locked manifest additionally can be released to allow users to openly read and edit the manifest on that subdomain again. The goal of this feature request then is to distinguish between public read/write vs. private write, public read.

With the top problem solved above, we solve the problem in this bug. With the other two feature requests, we make testing for our contributors with manifests really, really, easy. Thoughts?
I can fix that bug now. I'll also make it so that going to the base domain has an install button. Those are easy fixes; I'll do them now.

The second one is a bit harder, but do-able. I'll think about how to make this dead simple for both me and users.
(Reporter)

Comment 16

5 years ago
Removing the assignee. If the first problem is solved, we can close this bug as fixed.
Assignee: dclarke → nobody
First problem is fixed.
Assignee: nobody → gkoberger
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 18

5 years ago
Did some sanity checking - looks good. Will let you know if I hit any other issues. 

Btw, you should demo this at the show and tell this Friday - there's a lot of people who would find this useful!
Status: RESOLVED → VERIFIED
Greg, quick question here was say for example I have a manifest serving content on 

http://chimpanzee6282.testmanifest.com/manifest.webapp

And I wanted to test a webapp feature like screen rotation.. where would the hosted code go ? 

Generally I was hoping that chimpanzee6282 would proxy that code that was hosted on github, or create a CNAME that linked to your github account.
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
The hosting of the manifest is good i just am trying to figure out how we would use this to host 20 different test apps which all did different things.
(Reporter)

Comment 21

5 years ago
(In reply to dclarke@mozilla.com from comment #19)
> Greg, quick question here was say for example I have a manifest serving
> content on 
> 
> http://chimpanzee6282.testmanifest.com/manifest.webapp
> 
> And I wanted to test a webapp feature like screen rotation.. where would the
> hosted code go ? 

As I said in the etherpad, content-based testing can be done through mozqa.com's web application using any of the existing test cases that are hosted there. To get test cases added, file a bug in Mozilla QA --> Infrastructure for it with a diff of changes for the test case against http://hg.mozilla.org/qa/litmus-data/ and request Henrik, me, or Anthony Hughes to review. The goal this bug is trying to solve is handle testing of different manifests on different origins, not the content (as that's already resolved and proven to work).

> 
> Generally I was hoping that chimpanzee6282 would proxy that code that was
> hosted on github, or create a CNAME that linked to your github account.

A good idea for a feature request, but not needed for this bug to be closed out. The problem being solved here is testing the different manifests on different origins, not the content.

(In reply to dclarke@mozilla.com from comment #20)
> The hosting of the manifest is good i just am trying to figure out how we
> would use this to host 20 different test apps which all did different things.

Stated above - Good idea for a feature request for Greg's site, but not needed to close out this bug.

Next time btw - Leave a comment to request to reopen, but do not forcibly reopen the bug, as that creates confusion.
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
(Reporter)

Updated

5 years ago
Status: RESOLVED → VERIFIED
(Reporter)

Updated

5 years ago
Summary: Host application test cases on a public server for QA testing → Allow for testing of multiple manifests on different app origins on a hosted server
(Reporter)

Comment 22

5 years ago
Note - If we end up determining that there is a need for application hosting, let's open a bug to track it if a need arises again. However, this may not become a requirement anymore, as there's other bugs being tracked that are now talking about what resources app can load off-origin (bug 752666). When those requirements get ironed out for off-origin resources (there's a lot of debate on this right now), then we can revisit and see application hosting is needed for the QA dept specifically.
- The summary has just changed so it fits with your version of "verified". 
- I generally reopen bugs if they aren't fixed.. I don't think there is confusion in doing this. 

-----

mozqa.com < --- I thought you had dropped this as the idea, as using github or something along that line would be better as it would allow community members to create their own tests.
(Reporter)

Comment 24

5 years ago
(In reply to dclarke@mozilla.com from comment #23)
> mozqa.com < --- I thought you had dropped this as the idea, as using github
> or something along that line would be better as it would allow community
> members to create their own tests.

Nope, I actually implemented it (it's marked as reopened in bug 748207, as I need to fix the icon, but app install works and is hosted right now). Location of manifest - http://mozqa.com/data/webapps/mozqa.com/manifest.webapp. Can be installed on either app directory (https://apps.mozillalabs.com/appdir/) or mozqa.com (http://mozqa.com/data/webapps/mozqa.com/appinstall.html). 

I don't think using github would be a better option for community members in this case. Community members easily can file bugs in mozilla qa --> infrastructure containing the test case they want to add, go through the review process, and we'll push the changes to hg.mozilla.org/qa/litmus_data. After a push is made, it is automatically deployed to mozqa.com in 15 minutes. I'd say this approach as simple as it gets.
An example, last weeks show and tell, developers showed off quite a few apps that used various aspects of the infrastructure. 

I think there are some general assumptions you are making. 

#1) The app install is not the important part
#2) I don't think community members will write a specific app to test a specific feature.  We will probably get very few apps that way. 

--- Most likely you will see a developer has an issue, we will ask them to put their example on github, and we can then choose either to fork / clone the web site. And link to the app. 

    For ease I would rather just clone, whatever example they provide and link to the app. It offers a much clearer example... And then we can even see historically where the submissions came from... 

I like the idea of it being distributed, and managed by github, not by mozilla.
Github's review process is infinitely clearer than hg. mercurial's best days are behind it, and I am surprised we'd continue to tie new efforts to it.  

I would rather community members be empowered to upload test apps, and link them to litmus test cases / moztrap. 

I was hoping the "review" process would happen out of band, and not block forward progress. Especially when there is just one or two anointed reviewers.
(Reporter)

Comment 26

5 years ago
(In reply to dclarke@mozilla.com from comment #25)
> An example, last weeks show and tell, developers showed off quite a few apps
> that used various aspects of the infrastructure. 
> 
> I think there are some general assumptions you are making. 
> 
> #1) The app install is not the important part
> #2) I don't think community members will write a specific app to test a
> specific feature.  We will probably get very few apps that way. 
> 
> --- Most likely you will see a developer has an issue, we will ask them to
> put their example on github, and we can then choose either to fork / clone
> the web site. And link to the app. 
> 
>     For ease I would rather just clone, whatever example they provide and
> link to the app. It offers a much clearer example... And then we can even
> see historically where the submissions came from... 
> 
> I like the idea of it being distributed, and managed by github, not by
> mozilla.
> Github's review process is infinitely clearer than hg. mercurial's best days
> are behind it, and I am surprised we'd continue to tie new efforts to it.  
> 
> I would rather community members be empowered to upload test apps, and link
> them to litmus test cases / moztrap. 
> 
> I was hoping the "review" process would happen out of band, and not block
> forward progress. Especially when there is just one or two anointed
> reviewers.

Please take this discussion off this bug. The problem intended to be solved in this bug is solved here.
If your gh_page on GitHub has a manifest.webapp, it will be served properly.
Assignee: gkoberger → nobody
You need to log in before you can comment on or make changes to this bug.