1133711 - Remove dev_apps folder

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia, defect)

Comment 1

[Martijn Wargers (dead)]

From https://etherpad.mozilla.org/fxos-engineering-most-wanted :
Remove dev_apps folder - https://bugzilla.mozilla.org/show_bug.cgi?id=1133711
The dev_apps folder generally does not contain useful apps for developers or dogfooders. These apps should be built into the test folders as fixtures, and loaded with the necessary tests. For any apps which we might actually want to have flashed on the phone, these should be moved to the apps/ folder and enabled via the build config lists. If not many people are going to be using the app, it should be moved to fixtures and people can always symlink it to the outoftree_apps folder.

    That requires updates to build testing infra

    If there are any useful apps here we can upload them to the marketplace, and folks can link to them in a local outoftree_apps folder so they are installed when the device is flashed.

    Many of the apps are the only means to test some of the hardware, since automation is not possible (yet?), e.g. NFC. Need to know where these apps needs to be going.

    Do they need to be in the gaia repo at all? Can we install them with WebIDE or symlink them into the outoftree_apps folder for those engineers who need to use them?

    It could. Someone need to check with related QA/Gecko people to make sure we don't break their workflow.

Comment 2

[Oleg Zasypkin [:azasypkin]]

Hey Martijn and Kevin,

What would be the best choice for the app that is used for automated daily Raptor performance tests only (specifically we want to bring [1] into the tree for that purpose)?

What we want to achieve is to basically have test bench-marking app _in tree_ that is instrumented with performance marks and is supposed to be run by Raptor only. Ideally it should be flashed to device only when we run "make raptor" (under the hood it's RAPTOR=1 PERF_LOGGING=1 DEVICE_DEBUG=1 GAIA_OPTIMIZE=1 NOFTU=1 SCREEN_TIMEOUT=0 make reset-gaia).

P.S. also wondering if it's reasonable/possible to have distros/raptor/apps for the raptor-only benchmarking apps.

Thanks!

[1] https://github.com/nga-crash-tests/service-worker-benchmarks

Comment 3

[Kevin Grandon :kgrandon]

IMHO - they should not be part of gaia. I can see these being useful for platform devs to help them benchmark though. Maybe we could also set up raptor to run tests on not just gaia code, but external repos as well?

Comment 4

[Oleg Zasypkin [:azasypkin]]

(In reply to Kevin Grandon :kgrandon from comment #3)
> IMHO - they should not be part of gaia. I can see these being useful for
> platform devs to help them benchmark though. Maybe we could also set up
> raptor to run tests on not just gaia code, but external repos as well?

Let's ask Eli, he should know better than me :)

Comment 5

[:Eli Perelman]

Raptor doesn't care how the app gets on the device, just as long as it's there when it's time to test, external repo or otherwise. I just need to automate putting it on the device, and not sure how to put apps on the device outside of the normal Gaia build flow.

Comment 6

[:Eli Perelman]

For what it's worth, I'd much rather keep the apps in an external repo and just figure out how to automate installing to device.

Comment 7

[Oleg Zasypkin [:azasypkin]]

Isn't it what Spark does with remote

Comment 8

[Oleg Zasypkin [:azasypkin]]

Isn't it what Spark does with remote.list [1] and [2]?

[1] https://github.com/mozilla-b2g/gaia/blob/master/distros/spark/remote.list
[2] https://github.com/mozilla-b2g/gaia/blob/master/build/preload.js#L24

Comment 9

[Oleg Zasypkin [:azasypkin]]

So, I had a chance to play with preload-app-toolkit [1] and [2]. So it seems to be easy to integrate it with raptor workflow using build step similar to Spark's one [3] (or maybe even the same). App is successfully pre-loaded to gaia/outoftree_apps folder and installed with "make raptor". But it has the following issues/inconveniences:

1) We should store the list of the apps we want to preload somewhere. Spark has it in [4]. 

Where should we store the list of perf benchamrking apps?

2) App should be packaged into zip file and _hosted_ somehwere; The server where the package is hosted should support ETag http header for manifest.webapp (GitHub Pages don't support this :/, maybe we can relax this etag-requirement though). I see spark apps are hosted either on custom domain or in market place;

Do you have idea where we can host such apps?

3) Once app is flashed from preloaded package it doesn't have "predefined" app:// host name, it uses GUID instead, so I didn't find a way to run raptor against it (I've used name mentioned in manifest, GUID - no luck, always get timeout).

Is it possible to have "predefined" host for such apps? Is raptor supposed to deal with this somehow?

Kevin, Eli do you have any thoughts on this?

Or maybe we should not use preload script and just "git clone" required apps to "gaia/outoftree_apps" (still 1st question applies)?

Thanks!

[1] https://github.com/mozilla-b2g/preload-app-toolkit/
[2] https://github.com/mozilla-b2g/gaia/blob/master/tools/preload.py
[3] https://github.com/mozilla-b2g/gaia/blob/master/build/preload.js
[4] https://github.com/mozilla-b2g/gaia/blob/master/distros/spark/remote.list

Comment 10

[Kevin Grandon :kgrandon]

Going to see if Eli knows anything here, but perhaps we could implement a way to run tests based on app name instead of the host name? It's certainly more fragile that way though.

Comment 11

[Martijn Wargers (dead)]

You could also install the Raptor app during the setup phase. In bug 1212843, I'm planning to add an install method to gaia_test.py, but it's already easy to do now.
Or you could perhaps just push the webapp to the /data/local/webapps/ folder on the device and adjust /data/local/webapps/webapps.json, but not sure how well that would work.

Comment 12

[:Eli Perelman]

I don't really have an opinion on (1) or (2).

I could have sworn it used to be in the Raptor MDN docs, but I can't seem to find it. Anyway, Raptor currently requires an origin to test. So you could just modify your manifest.webapp [1] to set an origin to any domain, e.g. "origin": "service-worker-benchmarks.gaiamobile.org".

[1] https://github.com/nga-crash-tests/service-worker-benchmarks/blob/master/manifest.webapp

Comment 13

[Oleg Zasypkin [:azasypkin]]

(In reply to :Eli Perelman from comment #12)
> I don't really have an opinion on (1) or (2).

Kevin, sorry for bothering you again, but what do you think about (1) and (2) in comment 9?

Any pointers on how to move this forward are highly appreciated! The simplest route will be to just put this app into dev_apps folder and forget, but since this bug is filed it doesn't look like a good choice :)

Thanks!
 
> 
> I could have sworn it used to be in the Raptor MDN docs, but I can't seem to
> find it. Anyway, Raptor currently requires an origin to test. So you could
> just modify your manifest.webapp [1] to set an origin to any domain, e.g.
> "origin": "service-worker-benchmarks.gaiamobile.org".

Cool, will try that, thanks!

Comment 14

[Kevin Grandon :kgrandon]

(In reply to Oleg Zasypkin [:azasypkin][⏰UTC+1] from comment #13)
> (In reply to :Eli Perelman from comment #12)
> > I don't really have an opinion on (1) or (2).
> 
> Kevin, sorry for bothering you again, but what do you think about (1) and
> (2) in comment 9?
> 
> Any pointers on how to move this forward are highly appreciated! The
> simplest route will be to just put this app into dev_apps folder and forget,
> but since this bug is filed it doesn't look like a good choice :)

I would like to see if we can find a way to modify the origin if necessary, and install them from out of tree. I don't think we should include all of these testing apps inside of gaia.

Comment 15

[Oleg Zasypkin [:azasypkin]]

Okay, I'll wrap up something and then we'll see.

I was trying to play with "origin" and it doesn't work for packages preloaded with "preload.py" script since package that is loaded with this script is marked as "external" and our build system doesn't support external packages with "origin" o_0.

------

Alexandre and Dale, we need your advice/help here :)

TL;DR. we want to have some set of benchmarking-only apps (they can be privileged or certified), these apps should be hosted outside of main gaia tree, we'll just have a list of those apps _somewhere_. So we want to preload them only when Raptor tool needs them. This part can be achieved with preload.py script already, but Raptor needs to know exact app origin to run performance tests against it and I didn't manage to make "fixed" origin work with "preload.py" and "build/utils-xpc.js" bundle.

Alexandre,

I see you've added this check [1] a while ago, could you please let us know if it's possible to have "fixed" origin for the packaged "external" app somehow (though I haven't found any info on what "external" means exactly) ?

------

Dale,

Is it possible to mark preloaded package as "non-external" somehow so that we can have predefined "origin" (well looking at [2] - likely "no", but is it set in stone) ?

Also I see "preload.py" only works with the servers that provide "etag" HTTP header, is it possible to use/modify this script to bypass this check or work with "Last-Modified" HTTP header instead? At first I wanted to host my package at "GitHub Pages", but it doesn't support Etag :/

Any other thoughts on how we can solve this task are highly appreciated :)

Thanks!

[1] https://github.com/mozilla-b2g/gaia/blob/9908a086b2819e6a3c76de937e3958177b5933b7/build/utils-xpc.js#L408
[2] https://github.com/mozilla-b2g/gaia/blob/9908a086b2819e6a3c76de937e3958177b5933b7/tools/preload.py#L316

Comment 16

[Alexandre Poirot [:ochameau]]

(In reply to Oleg Zasypkin [:azasypkin][unavailable till Oct 23rd][⏰UTC+1] from comment #15)
> Alexandre,
> 
> I see you've added this check [1] a while ago, could you please let us know
> if it's possible to have "fixed" origin for the packaged "external" app
> somehow (though I haven't found any info on what "external" means exactly) ?

I don't remember exactly how we define what is external or not. (I'm no longer contributing much to gaia these days)
I think it is just the apps having a metadata.json, which allows various tricks.
The exception you linked as [1] just means you should set the origin in application.zip:manifest.webapp file, and not in metadata.json.
May be we should update this exception message for more clarity? (do not hesitate to submit a patch)

So if that's really what I think, you should be able to tweak your preload.py script accordingly.

Comment 17

[Dale Harvey (:daleharvey)]

Sorry for missing this

> Is it possible to mark preloaded package as "non-external" somehow so that 
> we can have predefined "origin" (well looking at [2] - likely "no", but is it set in stone) ?

preloadapp-toolkit (https://github.com/mozilla-b2g/preload-app-toolkit) was built to preload fairly arbitrary apps, if you need to patch it to enable functionality for you, fairly sure it will be accepted

Comment 18

[Oleg Zasypkin [:azasypkin]]

Thanks for you replies  Alexandre, Dale, I'll try to tinker around with these tools in bug 1219176.

Comment 19

[Kevin Grandon :kgrandon]

This is probably less of a priority now, and might be obsoleted with the transition plan anyway. Closing for now.