847830 - Use same mechanism as firefox/fennec for customization

Status: RESOLVED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Attachment: sample distribution.json

We use CUSTOMIZE environment variable for customization now, but we should use same mechanism as firefox/fennec.

fennec use distribution/distribution.json[1] for bookmarks, Preferences and Themes and allow putting extensions in distribution/ to preload extensions.

because there are many apps in gaia, not just a browser app, I propose use distribution.json to describe how to put customization resources to correct places, and we can remove app-specificed code in build/applications-data.js and build/manifest-zip.js.

attachment is sample distribution.json, and [2] is customize-sample.zip for current customization framework.

[1] https://wiki.mozilla.org/Mobile/Distribution_Files
[2] https://www.dropbox.com/s/9qi2i84p6iuyaj3/customize-sample.zip

Fabrice and Rick, could you give us suggestion for this work? thanks!

Comment 1

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Let's not put the destination information to the distribution.json, as that will likely change in the future. For example, your simple file directs build script to copy the wallpapers into two apps. If there is a 3rd app need the same files, all of the distribution.json out there in partners' repo will require upgrade.

IMO all of the destination/source info should stay in build scripts. I know it's very ugly currently to put such app-specific info in the addToZip() function, but put that into distribution.json is nowhere better....

The current "customize" framework looks good. We should aim to support that in the partner repo w/o asking them doing a lot of changes in near terms, given the fact our partner is already working with it. I don't see any use case for another description file in that dir. Change the dir/variable name here, and we could find a way to fix addToZip() once we could think of something.

Thanks for your consideration!

Comment 2

[Tim Guan-tin Chien [:timdream] (please needinfo)]

The work in this bug should something like

- s/CUSTOMIZE/GAIA_DISTRIBUTION_DIR/
- GAIA_DISTRIBUTION_DIR default to $GAIA_DIR/destination/
- /destination entry in .gitignore

Comment 3

[[:fabrice] Fabrice Desré]

(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TPE) from comment #2)
> The work in this bug should something like
> 
> - s/CUSTOMIZE/GAIA_DISTRIBUTION_DIR/
> - GAIA_DISTRIBUTION_DIR default to $GAIA_DIR/destination/
> - /destination entry in .gitignore

I agree with that, if we do s/destination/distribution

Comment 4

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Fabrice Desré [:fabrice] from comment #3)
> I agree with that, if we do s/destination/distribution

Yeah, typo, my bad.

Comment 5

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

nominating this for tef+ because we should keep same customization/distribution environment variable for all branches.

Comment 6

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Attachment: PR: https://github.com/mozilla-b2g/gaia/pull/8593

Comment 7

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Comment on attachment 723867 [details]
PR: https://github.com/mozilla-b2g/gaia/pull/8593

r+ if you rename these functions in question (and if I understand that correctly)

Comment 8

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

merged.

master:
https://github.com/mozilla-b2g/gaia/commit/41384d66173335d62292d9c571c826768e11dd8a

v1.0.1:
https://github.com/mozilla-b2g/gaia/commit/b12d129f14b7d9eb3d259eabfae5cce46adb6ec2

v1-train:
https://github.com/mozilla-b2g/gaia/commit/26828e665158191ea941366c393848b043ae340d

Comment 9

[Dave Hylands [:dhylands]]

Reverted on v1.0.1
https://github.com/mozilla-b2g/gaia/pull/8631
https://github.com/mozilla-b2g/gaia/commit/97c8d8a6249156043d348543f10fdc8117facacb

Revetred on v1-train:
https://github.com/mozilla-b2g/gaia/pull/8632
https://github.com/mozilla-b2g/gaia/commit/9639a27130d5b14f9b3f494e0c91c3fd06e78c62

Comment 10

[Dave Hylands [:dhylands]]

I think I asked for retriggers of Bg, Ng and NgL on b2g18 and b2g18-v1.0.1 on TBPL properly.

I also accidentally retriggered a Bg before I delivered my changed (I didn't realize that just clicking on the + actually retriggered, I was expecting some type of confirmation)

Comment 11

[Dave Hylands [:dhylands]]

oh yeah - RyanVM requested someone to back out the v1.0.1 and v1-train changes from comment 8 due to suspected gaia bustage. I volunteered to do it for him (in case anybody reading this bug is wondering whats going on)

Comment 12

[Ryan VanderMeulen [:RyanVM]]

FYI, this was causing bustage of the desktop B2G builds on the b2g18 branches, hence the backout. This probably affects mozilla-central (gaia-master) as well, but we didn't run into it because we don't actually generate B2G desktop builds there. Anyway, we should probably back this out on master too and reopen the bug.

Here's a log of the bustage.
https://tbpl.mozilla.org/php/getParsedLog.php?id=20607916&tree=Mozilla-B2g18

run-js-command  applications-data
make[7]: Leaving directory `/e/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/gaia'
e:\builds\moz2_slave\m-b18-w32_g-000000000000000000\build\obj-firefox\b2g\gaia\Makefile:41:0: command 'make -j1 -C e:/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/gaia profile' failed, return code 2
e:\builds\moz2_slave\m-b18-w32_g-000000000000000000\build\config\makefiles\target_libs.mk:60:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/build/pymake/pymake/../make.py -C gaia libs' failed, return code 2
e:\builds\moz2_slave\m-b18-w32_g-000000000000000000\build\config\makefiles\target_libs.mk:18:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/build/pymake/pymake/../make.py -C b2g libs' failed, return code 2
e:\builds\moz2_slave\m-b18-w32_g-000000000000000000\build\config\rules.mk:587:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/build/pymake/pymake/../make.py libs_tier_app' failed, return code 2
e:\builds\moz2_slave\m-b18-w32_g-000000000000000000\build\config\rules.mk:552:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/build/pymake/pymake/../make.py  tier_app' failed, return code 2
e:\builds\moz2_slave\m-b18-w32_g-000000000000000000\build\client.mk:351:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/build/pymake/pymake/../make.py -j1 -C obj-firefox' failed, return code 2
e:\builds\moz2_slave\m-b18-w32_g-000000000000000000\build\client.mk:152:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/build/pymake/pymake/../make.py -f e:/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/client.mk realbuild' failed, return code 2
build/utils.js:103: Error:  -*- build/utils.js: Invalid file path (/e/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/gaia/distribution, homescreens.json)
build/utils.js:103: [Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsILocalFile.initWithPath]"  nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)"  location: "JS frame :: build/utils.js :: getFile :: line 94"  data: no]
build/utils.js:103: 
make[7]: *** [applications-data] Error 3

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Confirmed that this is busted on gaia master as well. Please backout and reopen.

Comment 14

[Dave Hylands [:dhylands]]

Reverted on master:
https://github.com/mozilla-b2g/gaia/pull/8639
https://github.com/mozilla-b2g/gaia/commit/2203dcb9ac14c8033a366bec72a3347e0c9acd7b

Reopened

Comment 15

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #12)
> FYI, this was causing bustage of the desktop B2G builds on the b2g18
> branches, hence the backout. This probably affects mozilla-central
> (gaia-master) as well, but we didn't run into it because we don't actually
> generate B2G desktop builds there. Anyway, we should probably back this out
> on master too and reopen the bug.
> 
> Here's a log of the bustage.
> https://tbpl.mozilla.org/php/getParsedLog.php?id=20607916&tree=Mozilla-B2g18
> 

Hey Ryan, thanks for the info. Do you happen to know if there is a try server available for test out Gaia?

In the mean time, Yuren is in the process of setting-up Windows VM locally with ievms ...

Comment 16

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Oh, and, sorry about not caught the error when I review the code at first place. :-/

Comment 17

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

this is not only a file separator issue, seems realpath command of makefile return string like "/e/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/gaia/distribution"

not 

"e:/builds/moz2_slave/m-b18-w32_g-000000000000000000/build/gaia/distribution"

WIP here: https://github.com/yurenju/gaia/commit/f9db7268b107ca51ed5932ef88d642ee43398352

Comment 18

[Ryan VanderMeulen [:RyanVM]]

(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TPE) (PTO Mar 22 - Apr 7) from comment #15)
> (In reply to Ryan VanderMeulen [:RyanVM] from comment #12)
> > FYI, this was causing bustage of the desktop B2G builds on the b2g18
> > branches, hence the backout. This probably affects mozilla-central
> > (gaia-master) as well, but we didn't run into it because we don't actually
> > generate B2G desktop builds there. Anyway, we should probably back this out
> > on master too and reopen the bug.
> > 
> > Here's a log of the bustage.
> > https://tbpl.mozilla.org/php/getParsedLog.php?id=20607916&tree=Mozilla-B2g18
> > 
> 
> Hey Ryan, thanks for the info. Do you happen to know if there is a try
> server available for test out Gaia?
> 
> In the mean time, Yuren is in the process of setting-up Windows VM locally
> with ievms ...

Like I mentioned in comment 12, we unfortunately don't build desktop B2G builds on m-c, so I don't think they will build on Try either :(. I'm going to be pinging releng about this today. FWIW, the localizer builds that were still running on gaia-central went green after 14, so thanks Dave!

Comment 19

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Yuren states that he has tested the WIP code posted on comment 17, but I cannot verify that (I asked him for the VM files but I didn't copy the files correctly :'( ).

Given the fact there is no environment readily available, I'll post a pull request of his work for review today anyway. Running out of ideas :-/

Comment 20

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Attachment: Github: https://github.com/mozilla-b2g/gaia/pull/8663

https://github.com/mozilla-b2g/gaia/pull/8663

Fabrice, would you mind review this patch? Thanks. This is put Yuren's WIP [1] on top of the original commit [2]. The path issue doesn't happen in Python but in Makefile, and the wrapper adopted is a well-known and tested workaround to that.

[1] https://github.com/yurenju/gaia/commit/f9db7268b107ca51ed5932ef88d642ee43398352
[2] https://github.com/mozilla-b2g/gaia/commit/41384d66173335d62292d9c571c826768e11dd8a

Comment 21

[[:fabrice] Fabrice Desré]

Comment on attachment 725237 [details]
Github: https://github.com/mozilla-b2g/gaia/pull/8663

Looks good to me. Crossing fingers ;)

Comment 22

[Ryan VanderMeulen [:RyanVM]]

Of course, if it breaks, we won't know since we don't currently generate desktop B2G builds on inbound/m-c and we fixed the bug where we were using gaia-master on the b2g18 branch builds \o/

Comment 23

[Ryan VanderMeulen [:RyanVM]]

Actually, we do generate nightly B2G desktop builds on m-c, but they're hidden by default. Once this push shows up in the hg mirror of gaia-master, I'll trigger a new build and see what happens.

Comment 24

[[:fabrice] Fabrice Desré]

https://github.com/mozilla-b2g/gaia/commit/57e17b813e7244e320ad454e1007efbda572ec4a

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Unfortunately, I won't be able to generate a usable nightly off this push due to some unrelated bustage. We'll have to wait until tomorrow to see what happens with the regularly-scheduled ones.

Comment 26

[Tim Guan-tin Chien [:timdream] (please needinfo)]

https://github.com/mozilla-b2g/gaia/commit/57e17b813e7244e320ad454e1007efbda572ec4a

landed on master in comment 24.

Comment 27

[Tim Guan-tin Chien [:timdream] (please needinfo)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=20691539&tree=Mozilla-B2g18

Looks like the build is a success (I am looking at the right log right?)

Comment 28

[Ryan VanderMeulen [:RyanVM]]

b2g18 uses v1-train for desktop builds (as your log shows). As I mentioned in comment 23, this needs to be verified on mozilla-central (which is now broken yet again for unrelated issues :(...)

Comment 29

[Ryan VanderMeulen [:RyanVM]]

I *think* you're OK to uplift this now. We're still having issues with the m-c B2G desktop builds, but it's during the test phase after building.

Comment 30

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Merged again.

v1-train:
https://github.com/mozilla-b2g/gaia/commit/ac8b9dd9290b0e520933d4ce534d029ad36f5109

v1.0.1:
https://github.com/mozilla-b2g/gaia/commit/28a90691f03a5b59cb140ed19003a66bbeb6448d

Comment 31

[Ryan VanderMeulen [:RyanVM]]

It stuck this time :)

Comment 32

[Cody Roesch [:croesch]]

Can you please provide steps to verify this fix - as we will blackbox test from the UI?

Comment 33

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

I installed a windows with mingw, cygwin, python, etc.. for installing gaia and verify it. it's really complex...

Comment 34

[amiller]

No Test case creation is needed in moztrap for this issue.

Comment 35

[John Hammink]

Please verify this fix.

Comment 36

[John Hammink]

Please disregard my last comment.