1196992 - Pull hosted and Marketplace apps into the stock Gaia build

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Build, defect)

Description

[Nicole Yee (:nicoleyee)]

Per Candice Serran's email:

Closing the loop on this from discussion in-person with Wilfred, Doug, Tim, and Ken.

Plan for 2.5:
Doug to implement bumper bot solution for Bugzilla Lite and Buddy Up 
Tim to review
Estimate: 2-3 days of engineering development

Comment 1

[Doug Sherk (:drs) (inactive)]

Patryk, where should these apps go on the Homescreen?

Comment 2

[Doug Sherk (:drs) (inactive)]

Attachment: [spark] Add support for bumping hosted and Marketplace apps into Gaia.

Comment 3

[GitHub Autolander Bot]

Attachment: [gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master

Comment 4

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8651956 [details] [review]
[gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master

For reference only. Holding off on review until comment 1 is answered.

Comment 5

[Patryk Adamczyk [:patryk] UX]

Technically app hierarchy is a product question, so since this is for an official (2.5) release. Wilfred would be the best to comment on comment 1.

Comment 6

[Wilfred Mathanaraj [:WDM]]

as sent via email: 

Dialer
SMS
Contacts
Email
Browser

Camera
Gallery
Music
Video
Marketplace

Calendar
Clock
Settings
FM
Vaani (TBC)

Webmaker
Buddy Up
Bugzilla Lite
Facebook
Twitter
Notes
Calculator
Costcontrol

Comment 7

[Doug Sherk (:drs) (inactive)]

Rob, Wilfred has asked via email for you to make the final decision on the Homescreen layout. Would you like to change anything in comment 6?

Comment 8

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Comment on attachment 8651954 [details] [review]
[spark] Add support for bumping hosted and Marketplace apps into Gaia.

The only comment I have is that we'll want to add all the apps that wilfred mentioned in comment 6 that isn't in gaia currently.  (or whatever the final list is )

Comment 9

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8651954 [details] [review]
[spark] Add support for bumping hosted and Marketplace apps into Gaia.

Needs to be updated with Jacqueline's answer.

Comment 10

[Doug Sherk (:drs) (inactive)]

Here's a conversation I had with sfoster about using late customization instead of bumper bot for pulling hosted and/or Marketplace apps:
http://logs.glob.uno/?c=mozilla%23gaia&s=27+Aug+2015&e=27+Aug+2015#c152607

Comment 11

[Doug Sherk (:drs) (inactive)]

Via email from Rob Macdonald [:robmac],
> The order of the list that Wilfred has proposed generally makes sense to me.
> We may want to tweak the order slightly depending on (a) whether app grouping
> is available for the default home screen and (b) whether we ship with the
> three column or four column grid.
>
> Also, I assume this is the default on master but that partners can customize
> based on the needs of their users.
> 
> Jacqueline came up with the current default order in 2.0 and may have more to
> add.
> 
> Thanks!
> Rob

Via email from Jacqueline Savory [:jsavory],
> This arrangement looks good to me as well, the priority of the apps and
> grouping of them seems appropriate so I wouldn't make any major changes.
> 
> The only tweak I might make would be after we have confirmation between the
> three or four column grid, as it might change which apps are on a row together.
> Currently, I think this looks good on the three column grid, but could use a
> little moving around for the four.
> 
> Thank you!

Comment 12

[Jacqueline Savory [:jsavory] UX]

Thanks Doug for updating the bug with the email feedback. Removing ni?, but feel free to flag me again if the default icon set is four columns instead of three.

Comment 13

[Doug Sherk (:drs) (inactive)]

Attachment: [gaia] Import hosted_apps/* folder into the build in all configurations, update default Homescreen arrangement.

Comment 14

[Doug Sherk (:drs) (inactive)]

Attachment: [spark] Add support for bumping hosted and Marketplace apps into Gaia.

Comment 15

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Comment on attachment 8651954 [details] [review]
[spark] Add support for bumping hosted and Marketplace apps into Gaia.

I am not in charge of bumper bots.

Comment 16

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8651954 [details] [review]
[spark] Add support for bumping hosted and Marketplace apps into Gaia.

Kevin has historically reviewed Spark-related patches like this for me.

Comment 17

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 8651956 [details] [review]
[gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master

For hosted apps, I prefer to check in as folder rather than binrary files, and do package during build time. Bumper bot will regularly pull new binrary files into gaia that would increase the repo size very fast.

Comment 18

[Kevin Grandon :kgrandon]

Comment on attachment 8651954 [details] [review]
[spark] Add support for bumping hosted and Marketplace apps into Gaia.

I feel like a build peer needs to review this as it feels like it's part of the build to me. I also don't know bash too well, and personally would prefer a node solution (as most gaia developers are JS hackers). Going to flag Ricky on this one instead if that's ok.

Comment 19

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8651956 [details] [review]
[gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master

I tried unrolling the zip files that 'preload.py' downloads, but there are several problems with that:

* 'metadata.json' no longer does anything, so we don't get updates for the preloaded apps via Marketplace.
* The manifests include `package_path` fields, so we'd have to parse them and strip this out. While we could do that, it's very likely that this will break in the future.
* The app origin is broken, so if it has to send requests back to a server, it doesn't work.

As much as I'd like to be able to do this, I don't think that unrolling the zips will work very well.

I looked for information about how Git handles binary diffing to see if, every time one of these zip files gets updated, it'll blow up the size of the repo. It seems that over time, the repo gets packed and Git does binary diffs between versions, [1] [2] so most of the cost is up-front. If this is true, then zipping vs unzipping will add roughly the same amount of overhead to the repo.

[1] http://stackoverflow.com/questions/17976067/will-git-store-diffs-of-binary-files-that-change-in-content-but-never-change-si
[2] http://stackoverflow.com/questions/4697216/is-git-good-with-binary-files

Comment 20

[Ricky Chien [:rickychien] (inactive)]

(In reply to Doug Sherk (:drs) from comment #19)
> Comment on attachment 8651956 [details] [review]
> [gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master
> 
> I tried unrolling the zip files that 'preload.py' downloads, but there are
> several problems with that:

I was confused here, Fred told me preload.py merely generate a new folder instead of a zip, so why you need to unroll them?
In my imagination, bumperbot will pull hosted apps into gaia everyday and these apps should be folder format. So that gaia build system is able to pack hosted apps as gaia apps.

> * 'metadata.json' no longer does anything, so we don't get updates for the
> preloaded apps via Marketplace.
> * The manifests include `package_path` fields, so we'd have to parse them
> and strip this out. While we could do that, it's very likely that this will
> break in the future.
> * The app origin is broken, so if it has to send requests back to a server,
> it doesn't work.

I don't understand that why above reasons will make us need to check in zips into gaia.

> As much as I'd like to be able to do this, I don't think that unrolling the
> zips will work very well.
> 
> I looked for information about how Git handles binary diffing to see if,
> every time one of these zip files gets updated, it'll blow up the size of
> the repo. It seems that over time, the repo gets packed and Git does binary
> diffs between versions, [1] [2] so most of the cost is up-front. If this is
> true, then zipping vs unzipping will add roughly the same amount of overhead
> to the repo.
> 
> [1]
> http://stackoverflow.com/questions/17976067/will-git-store-diffs-of-binary-
> files-that-change-in-content-but-never-change-si
> [2] http://stackoverflow.com/questions/4697216/is-git-good-with-binary-files

Comment 21

[Doug Sherk (:drs) (inactive)]

(In reply to Ricky Chien [:rickychien] from comment #20)
> I was confused here, Fred told me preload.py merely generate a new folder
> instead of a zip, so why you need to unroll them?
> In my imagination, bumperbot will pull hosted apps into gaia everyday and
> these apps should be folder format. So that gaia build system is able to
> pack hosted apps as gaia apps.

I'll be very specific, because I think we're talking about two different things here.

Marketplace is capable of serving both packaged and hosted apps. In both cases, 'preload.py' preloads some metadata via the 'metadata.json' file. Note that this is the only file that is custom written by 'preload.py'. The rest are all served statically by Marketplace, or whatever other source we're pulling them from.

There are also special cases, which are as follows
- Packaged apps: 'preload.py' preloads a stub 'update.webapp' manifest and the app package zip file. This file points to the package zip file, which has a 'manifest.webapp' manifest inside it.
- Hosted apps: 'preload.py' preloads a full 'manifest.webapp' manifest.

Note that the apps that we're preloading in this bug are a mix of hosted and packaged apps. Here's a link to BuddyUp, an example of packaged app being served on Marketplace:
https://marketplace.firefox.com/app/buddyup?src=search

Here's the manifest for that app:
https://marketplace.firefox.com/app/8d979279-a142-4fee-993b-8e7797b221a5/manifest.webapp

Note also that this manifest has a `package_path` field.

If you go here from the Firefox OS browser, you'll see that it when you download/install it, it gives you a download size. This is because it's just displaying the size of the zip file that it's serving you. Hosted apps just prompt you to install them without providing a file size, like Bugzilla Lite for example.

Based on our seeking a way to have packaged apps unrolled in the Gaia build, I tried unrolling the zipped 'application.zip' files that Marketplace serves us, but then we lose the following benefits:
- Updates from Marketplace.
- We have to change the manifest and strip out the `package_path` field before checking it into Gaia, which is hacky and prone to breaking.
- Metadata associating the app with its actual origin.
- Random things break, like image paths and such.

I'm open to alternatives here, but I don't see any reasonable ones. Do you have any thoughts on this?

Comment 22

[Ricky Chien [:rickychien] (inactive)]

Thanks for your specific explanation here!

It's really a problem when pulling hosted apps and packaged apps, we have to figure out how to deal with it for future build system. So I think it's reasonable solution to check in zip files into gaia repo as we check in marketplace app.

Comment 23

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Taking a review of this, I think we need to account for the apps that we are placing on the system partition.  This will cause an issue with flame OTA.  I think we need options for install for flame so that we keep the system partition size managable.  We already ran out of space once and it cause issues trying to get people upgraded.  bug 1181372, bug 1184980

Comment 24

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Adding Alexandre so that he's aware and if he would care to comment.  (if not no worries)

Comment 25

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8651956 [details] [review]
[gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master

Could we get this tested as an OTA update, so we can confirm whether or not comment 23 will be a problem?

This would involve starting from any relatively recent image, then serving this Gaia branch as part of an OTA update.

Comment 26

[Doug Sherk (:drs) (inactive)]

Anyone from QAnalysts, feel free to take testing comment 25 from Naoki, but he should still make the final call.

Comment 27

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

https://bugzilla.mozilla.org/show_bug.cgi?id=1199863
We already run into this issue on Flame now.

Comment 28

[Doug Sherk (:drs) (inactive)]

I have no background on this problem, but here's my understanding.

The Aries device doesn't have this problem, because the system partition is big enough by default.

The Flame, though, is running out of space on the system partition (bug 1199863). We've run into this problem in the past, and to fix it then, we expanded the system partition size.

Alex, is there anything we can do here? Can we expand the system partition again? Are there any other options? Is there anyone else whom we can ask for help on this problem?

Even if we wanted to land these changes for Aries only, we have no way of making device-specific app lists, so that's not an option.

Comment 29

[Doug Sherk (:drs) (inactive)]

Looks like there's nothing QA can do for here as this is definitely a problem, based on bug 1199863.

Comment 30

[Gabriele Svelto [:gsvelto] (PTO)]

I assume the problem here applies to engineering builds since user builds are a fair bit smaller. On a Flame using the v4 base image by default we're left with ~100M after having flashed an engineering build; for the record that's ~30MiB less than a user build. This is not enough to perform an OTA update because we need to unpack the ZIP file entirely in the free space before replacing the existing system. There's two solutions for this:

- Increase the /system partition size. While this is technically possible (we could eat into /data which is 2G currently) I don't think we can get another base build from T2M-

- Use FOTA instead of OTA. We've been saying this with Alexandre for ages, FOTA works a lot better than OTA as it needs less space and can do more. From a user-experience POV it's less nice (while we perform the installation the user cannot use the phone) but it's a ton faster and doesn't require extra space available. Things are finally moving in this direction and we've even scheduled regular meetings for discussing rolling out FOTA (minutes are here for those interested https://etherpad.mozilla.org/fxos-update ) so the issue of the space on Flame might become moot soon enough.

Comment 31

[:gerard-majax]

(In reply to Doug Sherk (:drs) from comment #28)
> I have no background on this problem, but here's my understanding.
> 
> The Aries device doesn't have this problem, because the system partition is
> big enough by default.
> 
> The Flame, though, is running out of space on the system partition (bug
> 1199863). We've run into this problem in the past, and to fix it then, we
> expanded the system partition size.
> 
> Alex, is there anything we can do here? Can we expand the system partition
> again? Are there any other options? Is there anyone else whom we can ask for
> help on this problem?
> 
> Even if we wanted to land these changes for Aries only, we have no way of
> making device-specific app lists, so that's not an option.

Two things. First, no, the trick we used safely cannot be used anymore. Repartitionning now will need to actually move partitions, which is much more dangerous.

Second, it's not clear what is the issue. Did you actually run into the issue ? We added an extra 52MB, I think it allows us to have 26MB more of apps/content when doing an OTA. How much are we actually getting now ? Are you just referring to bug 1199863 ? Because without any logcat and proper checking we can only /infer/ this is a partition size issue, and not confirm it.

Comment 32

[Doug Sherk (:drs) (inactive)]

(In reply to Alexandre LISSY :gerard-majax from comment #31)
> Two things. First, no, the trick we used safely cannot be used anymore.
> Repartitionning now will need to actually move partitions, which is much
> more dangerous.
> 
> Second, it's not clear what is the issue. Did you actually run into the
> issue ? We added an extra 52MB, I think it allows us to have 26MB more of
> apps/content when doing an OTA. How much are we actually getting now ? Are
> you just referring to bug 1199863 ? Because without any logcat and proper
> checking we can only /infer/ this is a partition size issue, and not confirm
> it.

Naoki and I just tried this on a fresh build. We started on the latest Flame build, then OTA updated to one with the bumper bot apps included. The update failed with the same error code as bug 1199863.

My suggestion for now is to just include the hosted apps only, and not the packaged apps, until we figure out a better solution. Perhaps that solution will be the one in comment 30.

Comment 33

[Doug Sherk (:drs) (inactive)]

I've updated the PRs to update only the hosted apps. Naoki, can you give QA approval on this?

Comment 34

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Comment on attachment 8651954 [details] [review]
[spark] Add support for bumping hosted and Marketplace apps into Gaia.

Just as a quick look, of looking at the code : Landing just the hosted apps of the bugzilla lite, twitter and facebook seems ok with me.

I would need to test the build process at some point as long as this doesn't break the building of gaia, it should be good to go. + for landing.

Comment 35

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8651954 [details] [review]
[spark] Add support for bumping hosted and Marketplace apps into Gaia.

https://github.com/fxos-eng/spark/commit/de4f1f65b8cf5d731759361229143c89a143136b

Comment 36

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8651956 [details] [review]
[gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master

https://github.com/mozilla-b2g/gaia/commit/03387127ae7eb865919cf3a0909036c874431778

Comment 37

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8651956 [details] [review]
[gaia] DouglasSherk:1196992-hosted > mozilla-b2g:master

Partially reverted in https://github.com/mozilla-b2g/gaia/commit/60e212543329ee48f2921e0cd92aaca4b4ce2f6a due to build failures. Ricky and I are investigating.

Ricky suggested that it may be because the `origin` attribute in 'metadata.json' isn't prefixed with "http" or "app".

Comment 38

[Doug Sherk (:drs) (inactive)]

Ricky, please leave any info you find here. I'll investigate more tomorrow.

Comment 39

[Ricky Chien [:rickychien] (inactive)]

Errors arised at https://github.com/mozilla-b2g/gaia/blob/master/build/test/integration/helper.js#L60-L63

The webapps `origin` attribute should begin with `app://` protocal unless mochitest app. However, those origin of hosted apps bebin with `https://` which generates from preload.py.

An appropriate fix is patch preload.py to generate app:// origin with a specific flag.

Comment 40

[Doug Sherk (:drs) (inactive)]

Ricky, something isn't adding up here. The "app://" scheme is for packaged apps, but the apps that we're including for now are all hosted. I don't think that the fix is to switch to the "app://" scheme.

Comment 41

[GitHub Autolander Bot]

Attachment: [gaia] DouglasSherk:1196992-hosted2 > mozilla-b2g:master

Comment 42

[Doug Sherk (:drs) (inactive)]

Attachment: Import hosted_apps/* folder into the build in all configurations, update default Homescreen arrangement.

Comment 43

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8658916 [details] [review]
[gaia] DouglasSherk:1196992-hosted2 > mozilla-b2g:master

Ricky, I made the checks for schemes a little bit less restrictive. Content-level apps can now use either the "http(s)?://" or "app://" schemes. This makes more sense to me as it is functionally correct.

Comment 44

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 8658916 [details] [review]
[gaia] DouglasSherk:1196992-hosted2 > mozilla-b2g:master

LGTM

Comment 45

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8658916 [details] [review]
[gaia] DouglasSherk:1196992-hosted2 > mozilla-b2g:master

https://github.com/mozilla-b2g/gaia/commit/1f7b19915f8146bcf1e9f938e1b9c6e4bce0bc95

Comment 46

[Doug Sherk (:drs) (inactive)]

I'm no longer working on this directly as I don't know how to deal with the issue in comment 28. If it turns out that I can help and finish this with whatever solution we come up with, I'll take this again.

Comment 47

[Julien Wajsberg [:julienw]]

Note that I landed bug 1214623 yesterday. This may free some room on Flame.
For example, the package for the SMS app is now half the previous size (I haven't checked other apps).

Comment 48

[Doug Sherk (:drs) (inactive)]

Naoki tells me that this is now safe to take to completion using the original plan. We're going to FOTA the next Flame update, so we sidestep bug 1199863.

Comment 49

[Doug Sherk (:drs) (inactive)]

Attachment: [spark] Add BuddyUp, Notes, Calculator to the bumped apps.

Comment 50

[Doug Sherk (:drs) (inactive)]

Attachment: [gaia] Add preloaded Marketplace apps into Vertical Home.

Comment 51

[Doug Sherk (:drs) (inactive)]

Attachment: [spark] Add BuddyUp, Notes, Calculator to the bumped apps.

Comment 52

[GitHub Autolander Bot]

Attachment: [gaia] DouglasSherk:1196992-hosted-packaged > mozilla-b2g:master

Comment 53

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

FYI, This is the bug that will add Buddyup to the main gaia apps.

Comment 54

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 8674513 [details] [review]
[spark] Add BuddyUp, Notes, Calculator to the bumped apps.

It seems not impact on gaia build. r+.

Comment 55

[Chris Lord [:cwiiis]]

Comment on attachment 8674514 [details] [review]
[gaia] DouglasSherk:1196992-hosted-packaged > mozilla-b2g:master

r+, but you should probably also make the relevant changes in apps/homescreen/build/default-homescreens.json

Comment 56

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Had spun a build on friday, but realized that I needed a fingerprint; creating a build today.  Was out sick yeterday.

Comment 57

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Agreed with Cwiis.  We want to move to the new homescreen by default for pin the web + other stuff.  Could this change be made please?

Comment 58

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I tried patching the second patch [https://github.com/DouglasSherk/gaia/commit/72699d8b5086d4984ecf155fff34b440b235bba3 ] 

the first patch was already committed [ https://github.com/DouglasSherk/gaia/commit/c64b41cdf188a1e51f45e8d65f088f151e320784 ] 

and made my own gaia but I don't see the buddyup app listed in the files.

Comment 59

[Doug Sherk (:drs) (inactive)]

Naoki and I discussed this on IRC. The problem was that the apps are missing from the Gaia repo, since we haven't run spark-bot's check-in process for these apps yet.

Comment 60

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Comment on attachment 8674513 [details] [review]
[spark] Add BuddyUp, Notes, Calculator to the bumped apps.

For updating the hosted files, this seems to work.  Not sure if I'm a big fan of all the separate gaia and upload; too many parts that could potentially break there.

Multiple FOTAs seem to work ok on the Flame device currently; having said that I haven't add tons to the profile and there's too many varying variables.

Comment 61

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8674513 [details] [review]
[spark] Add BuddyUp, Notes, Calculator to the bumped apps.

https://github.com/fxos-eng/spark/commit/38c667be5dcec18eee180b3fbfc232fca020a530

Comment 62

[Doug Sherk (:drs) (inactive)]

Bumped remotely hosted apps for the first time using attachment 8674513 [details] [review] here:
https://github.com/mozilla-b2g/gaia/commit/ec2b1bbba3d23f1c224874b3bae928b1b2d675fa

Comment 63

[Doug Sherk (:drs) (inactive)]

Comment on attachment 8674514 [details] [review]
[gaia] DouglasSherk:1196992-hosted-packaged > mozilla-b2g:master

https://github.com/mozilla-b2g/gaia/commit/aac4d416b6665e908589ae28f47f3388e17fca3f

Comment 64

[Kevin Grandon :kgrandon]

This broke a few tests, so backing out for now. Please use a pull request to re-land and ensure tests are green.

Backout: https://github.com/mozilla-b2g/gaia/commit/90d30c756b5c4cb9c4968e676d42c5da38b86ef0
For breakage: https://treeherder.mozilla.org/logviewer.html#?job_id=580026&repo=gaia-master

Backout: https://github.com/mozilla-b2g/gaia/commit/db3b2e26c7f360a024088ccf073d454af9b842bc
For breakage: https://treeherder.mozilla.org/logviewer.html#?job_id=580030&repo=gaia-master

Some of these tests are a bit strange for having hard-coded values. I think we can probably just update them, but I'm not entirely sure about the strategy to bump/update these going forward. Doug - I'll leave it to you, thanks.

Comment 65

[Nicole Yee (:nicoleyee)]

Hi all, Drs is sick with the flu today and I imagine he will need a couple of days to recover. Do we have any back up plans to get this fixed. Unfortunately, we are really cutting it close for getting Bugzilla Lite into the 2.5 release with this bug and need this resolved by Friday. Any ideas? I'm not to sure who to need info so please need info accordingly.

Comment 66

[Kevin Grandon :kgrandon]

I'll take a look.

Comment 67

[GitHub Autolander Bot]

Attachment: [gaia] KevinGrandon:bug_1196992_reland > mozilla-b2g:master

Comment 68

[Nicole Yee (:nicoleyee)]

Thank you, Kevin!

Comment 69

[Kevin Grandon :kgrandon]

Re-landed: https://github.com/mozilla-b2g/gaia/commit/2d8465a779fbddfa5a3d748fc40f0382686f5be5

Comment 70

[Gabriela [:gaby2300]]

Which apps should start appearing in the future FOTA updates?