848604 - Gaia build process needs a better process for abstracting app directories

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[[e-mail] [:onecyrenus]]

The gaia build process, currently has a set of rules inside the Makefile & build/utils.js which define how all the apps are built / delivered on Firefox OS. 

The problem with aforementioned system is that it is inflexible, and requires new directories to be added and new logic to be added each time you want to build a slightly different configuraiton.

Comment 1

[[e-mail] [:onecyrenus]]

The basic system I am proposing is the following basic file format which stores the information about the apps being installed. 

config/dogfood.json or engineering.json 
{
  "apps" : [
    {"internal" : "apps/bluetooth"},
    {"internal" : "apps/browser"},...etc.etc
    {"external" : "external-dogfood-apps/hoststubtest"},

}
The make process reads in this file, and copies everything into a _mkapps/internal 
or _mkapps/external directory. 

make
pass the GAIA_APP_SRCDIRS two directories always _mkapps/internal & _mkapps/external
no more need to propogate further information to utils.js re: production / dogfood.


fabrice generally want to know if this design seems generally acceptable, as I am nearing having something workable.

Comment 2

[[e-mail] [:onecyrenus]]

Generally will make things backward compatible by choosing the config file based upon what environment variables are set ..eg production / dogfood, or whichever config file is set as an enviornment variable.

Comment 3

[[e-mail] [:onecyrenus]]

Attachment: build system to use json file to decide which apps go into the build.

Fabrice, 
   Would like you to consider this patch, it offloads the apps / directory choices to a json file, now having said that there is a lot more we can do with this.  Packages of apps, do we want build configuration to be stored in the json file as well. Ability to pull from github for off source apps.

Comment 4

[Jason Smith [:jsmith]]

Yuren - I know you've done quite a lot work on partner customizations. Will this patch impact some of the customization work you've built?

Comment 5

[[e-mail] [:onecyrenus]]

Yurenju, 
   If you could annotate bug with your design docs that would be great. 

Please note this bug is just a general proposal, still ironing out how we want to accomplish this task.

Comment 6

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

looks good for me.

CC Rudy since he is working on bug 838098 for preload apps from GAIA_DISTRIBUTION_DIR.

It would be great if we can read config file from GAIA_DISTRIBUTION_DIR/apps.json while it exists. and please set feedback? to me when you submit patch/pull request for landing this work, I will check compatibility with GAIA_DISTRIBUTION_DIR environment variable.

thanks!

Comment 7

[[e-mail] [:onecyrenus]]

Looking at the patch that is on bug 838098, I am not really happy with that approach. 

----

GAIA_DISTRIBUTION_DIR/apps.json  <--- This isn't listed on 
https://wiki.mozilla.org/B2G/MarketCustomizations#Customization_Overview

so my thought is that this is a new requirement, I will add this to my patch to assume apps.json is there, could someone update the wiki so it is a known expectation, because the build will fail if apps.json is not there, and that environment variable is set. 


(In reply to Yuren Ju [:yurenju] (PTO during 3/15-3/18) from comment #6)
> looks good for me.
> 
> CC Rudy since he is working on bug 838098 for preload apps from
> GAIA_DISTRIBUTION_DIR.
> 
> It would be great if we can read config file from
> GAIA_DISTRIBUTION_DIR/apps.json while it exists. and please set feedback? to
> me when you submit patch/pull request for landing this work, I will check
> compatibility with GAIA_DISTRIBUTION_DIR environment variable.
> 
> thanks!

Comment 8

[[e-mail] [:onecyrenus]]

Attachment: updated patch uses GAIA_DISTRIBUTION_DIR

Details of this patch: 
1) GAIA_DISTRIBUTION_DIR if this is set, it will attempt to use
GAIA_DISTRIBUTION_DIR/apps.json
2) If PRODUCTION / DOGFOOD / DEMO is set we will use the associated json file
3) If GAIA_APP_CONFIG is set we will use that. 
4) If none of these are set an engineering build with all apps will be created. 

This should be backward compatible with current usage.

Comment 9

[Daniel Coloma:dcoloma]

(In reply to [e-mail]  [:onecyrenus] from comment #8)
> Created attachment 725163 [details] [diff] [review]
> updated patch uses GAIA_DISTRIBUTION_DIR
> 
> Details of this patch: 
> 1) GAIA_DISTRIBUTION_DIR if this is set, it will attempt to use
> GAIA_DISTRIBUTION_DIR/apps.json
> 2) If PRODUCTION / DOGFOOD / DEMO is set we will use the associated json file
> 3) If GAIA_APP_CONFIG is set we will use that. 
> 4) If none of these are set an engineering build with all apps will be
> created. 
> 
> This should be backward compatible with current usage.

Anyhow, is this really a 1.0.1 blocker?

Comment 10

[Michael Vines [:m1] [:evilmachines]]

Comment on attachment 725163 [details] [diff] [review]
updated patch uses GAIA_DISTRIBUTION_DIR

Review of attachment 725163 [details] [diff] [review]:
-----------------------------------------------------------------

::: Makefile
@@ -86,3 @@
>  endif
>  
> -ifneq ($(GAIA_OUTOFTREE_APP_SRCDIRS),)

What happened to GAIA_OUTOFTREE_APP_SRCDIRS?  We uses this here to support apps that are not sitting within the gaia/ subdirectory.  Is this still supported via other means?

Comment 11

[Michael Vines [:m1] [:evilmachines]]

Comment on attachment 725163 [details] [diff] [review]
updated patch uses GAIA_DISTRIBUTION_DIR

Review of attachment 725163 [details] [diff] [review]:
-----------------------------------------------------------------

::: Makefile
@@ -86,3 @@
>  endif
>  
> -ifneq ($(GAIA_OUTOFTREE_APP_SRCDIRS),)

What happened to GAIA_OUTOFTREE_APP_SRCDIRS?  We uses this here to support apps that are not sitting within the gaia/ subdirectory.  Is this still supported via other means?

Comment 12

[[:fabrice] Fabrice Desré]

Comment on attachment 725163 [details] [diff] [review]
updated patch uses GAIA_DISTRIBUTION_DIR

Review of attachment 725163 [details] [diff] [review]:
-----------------------------------------------------------------

::: Makefile
@@ -1,1 @@
> -###############################################################################

nit: no need to remove that line

@@ -86,3 @@
>  endif
>  
> -ifneq ($(GAIA_OUTOFTREE_APP_SRCDIRS),)

I agree, we need a way to let people add apps from somewhere else.

@@ +233,5 @@
> +		type=$${A[0]}; \
> +		app_name=$${A[1]}; \
> +		cp -rf $$app_name $(MAKE_APPS_DIR)/$$type/ ; \
> +	done; \
> +	ln -s $(CURDIR)/shared $(MAKE_APPS_DIR)/

Why do we need that?

::: build/read-configs.py
@@ +18,5 @@
> +  data = json.load(json_data)
> +  json_data.close()
> +  for app in data['apps']:
> +    for key, value in app.iteritems():
> +      print key + ':' + value 

nit: trailing ws

::: configs/dogfood.json
@@ +27,5 @@
> +    {"internal" : "showcase_apps/crystalskull"},
> +    {"internal" : "showcase_apps/cubevid"},
> +    {"internal" : "showcase_apps/twittershare"}
> +  ]
> +}

I'm not a big fan of this json structure. What about something like:

{
 "internal": ["apps/bluetooth", "apps/browser", ...],
 "external": ["external-apps/marketplace", "external-apps/m.here.com", ...]
}

Comment 13

[[e-mail] [:onecyrenus]]

(In reply to Fabrice Desré [:fabrice] from comment #12)
> Comment on attachment 725163 [details] [diff] [review]
> updated patch uses GAIA_DISTRIBUTION_DIR
> 
> Review of attachment 725163 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: Makefile
> @@ -1,1 @@
> > -###############################################################################
> 
> nit: no need to remove that line
> 
> @@ -86,3 @@
> >  endif
> >  
> > -ifneq ($(GAIA_OUTOFTREE_APP_SRCDIRS),)
> 
> I agree, we need a way to let people add apps from somewhere else

Wouldn't the way to add an app to the build, be to edit the json file, and add it there. You can maintain your own json file for the apps you want to push to the device. ?

> 
> @@ +233,5 @@
> > +		type=$${A[0]}; \
> > +		app_name=$${A[1]}; \
> > +		cp -rf $$app_name $(MAKE_APPS_DIR)/$$type/ ; \
> > +	done; \
> > +	ln -s $(CURDIR)/shared $(MAKE_APPS_DIR)/
> 
> Why do we need that?
> 
This is part of the build process that is already there, ex: look at the gallery app Makefile, it includes resources from ../../shared/js.... so possibly would need to pass in a shared directory location to the makefile in order for the process to work as expected. 

> ::: build/read-configs.py
> @@ +18,5 @@
> > +  data = json.load(json_data)
> > +  json_data.close()
> > +  for app in data['apps']:
> > +    for key, value in app.iteritems():
> > +      print key + ':' + value 
> 
> nit: trailing ws
> 
> ::: configs/dogfood.json
> @@ +27,5 @@
> > +    {"internal" : "showcase_apps/crystalskull"},
> > +    {"internal" : "showcase_apps/cubevid"},
> > +    {"internal" : "showcase_apps/twittershare"}
> > +  ]
> > +}
> 
> I'm not a big fan of this json structure. What about something like:
> 
> {
>  "internal": ["apps/bluetooth", "apps/browser", ...],
>  "external": ["external-apps/marketplace", "external-apps/m.here.com", ...]
> }

Can change the structure, that is fine. 

Including out of src apps, i generally feel that this should be done through the json file mechanism...

Comment 14

[Michael Vines [:m1] [:evilmachines]]

(tef- as I won't see why this is mandatory for v1.0.1.  Please feel free to re-nom with justification if desired)

Comment 15

[[e-mail] [:onecyrenus]]

Michael, 
  Purpose of this patch is to allow partners to be able to customize which apps go on the phone.  imho should be considered part of the device customization framework. 

Unsure what our partner requirements are, but if they include deciding what apps to deploy on the device, then i think this is tef+. 

Currently the only mechanism available is to delete the apps from the gaia source tree, and build / deploy.

Comment 16

[Michael Vines [:m1] [:evilmachines]]

Thanks.  "Unsure what our partner requirements are" is a little scary because it implies we don't know that what we are building will ultimately be useful for v1.0.1.  Where are the requirements for this bug coming from then?   

Also GAIA_OUTOFTREE_APP_SRCDIRS is one mechanism that is currently available to add new apps outside of the gaia/ directory, but looks like this is being removed in this patch?  This will cause churn on v1.0.1 that I'd like to avoid if at all possible.

Comment 17

[[e-mail] [:onecyrenus]]

Michael, 
  So to answer your question for GAIA_OUTOFTREE_APP SRCDIRS, you can include whichever apps you want inside the json file. 
  The other part to GAIA_OUTOFTREE apps that doesn't work, and was the reason for me removing it is there is no ability to have an external out of gaia tree app.  So you end up with a name that hints you can include any kind of out of tree app, but in reality only handles internal apps. 
  My perspective is that GAIA_OUTOFTREE_APP_SRCDIRS is confusing and generally masks the problem, which is as an app developer i need a way to push my internal / external app to the device. I believe this process should be separate from gaia make.

Comment 18

[[:fabrice] Fabrice Desré]

I don't think Michael uses GAIA_OUTOFTREE_APP_SRCDIRS to push apps to the device, but to build images that include apps that are not part of gaia. That is a very legitimate use case, and we really need to support that with as few changes on their side as possible.

Comment 19

[[e-mail] [:onecyrenus]]

Fabrice, do you mean no changes from their side or as few changes from their side. 

This functionality is still supported, but you'd maintain the list of apps you wanted on device inside the json file, and they can most definitely be out of the src directory. 

For expediency sake i'll just add this back in.

Comment 20

[[:fabrice] Fabrice Desré]

(In reply to [e-mail]  [:onecyrenus] from comment #19)
> Fabrice, do you mean no changes from their side or as few changes from their
> side. 

As few as possible.

> This functionality is still supported, but you'd maintain the list of apps
> you wanted on device inside the json file, and they can most definitely be
> out of the src directory. 

Can you provide an example? That would help a lot I think.

Comment 21

[Michael Vines [:m1] [:evilmachines]]

(tef-, again :(   Sorry guys, I'm really not feeling this at all right now.  We are still talking about perturbing the v1.0.1 build for a new feature with unclear requirements.  That makes three reasons to take a pass at this point.)

Comment 22

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 725163 [details] [diff] [review]
updated patch uses GAIA_DISTRIBUTION_DIR

clean feedback flag because this patch got r-. set again when you have another patch.

Comment 23

[[e-mail] [:onecyrenus]]

Attachment: gaia make fixes to allow for your config directory to be well specified

Fixed what i believe to be the general concerns
#1) It is possible for you to include out of source tree apps using environment variables. 
Ex: export GAIA_OUTOFTREE_APP_SRCDIRS=/media/b2g/twittershare2/
Symlinks the out of source app try into your make directory _mkapps/internal
#2) It is possible for you to include out of source tree apps by editing the configuration json file ex: ....
                    "apps/video",
                    "/home/myuser/myapp/" ]
This will copy the app into the build directory into _mkapps/internal each time you rebuild. Slight semantic differnce, but wanted to preserve all previous notions of the environment variable. 
(My belief is that apps are small enough they should be copied into _mkapps/internal. As you will then have a record of exactly what was pushed to device even if you are mucking around in the source tree. 
#3) It is possible to include apps by linking to their url. 
     "external" : [ "external-apps/marketplace",
                    "external-apps/m.here.com",
                    "http://marketplace.firefox.com/manifest.webapp"]
Creating a dummy metadata.json file with the origin set. 
#4) Also this patch moves the entire build process to a build directory, so that the make process isn't touching source.

Comment 24

[[e-mail] [:onecyrenus]]

Will also run this through try tonight, and post the results.

Comment 25

[[e-mail] [:onecyrenus]]

(In reply to [e-mail]  [:onecyrenus] from comment #24)
> Will also run this through try tonight, and post the results.

Gaia changes don't have a mechanism to be pushed to the try server ....

Comment 26

[[e-mail] [:onecyrenus]]

Attachment: gaia pull request

Same patch as before but in the form of a gaia pull request.

Comment 27

[[e-mail] [:onecyrenus]]

Attachment: pointer to gaia pull request

removed the need to specify external or internal in the configuration file.

Comment 28

[[e-mail] [:onecyrenus]]

This last patch basically subsumes the work done by yurenju in the below commit

https://github.com/yurenju/gaia/commit/a62d990266c6f3661a5c712bc5d52eb49d6cdf52

Comment 29

[[e-mail] [:onecyrenus]]

Any eta on this ?

Comment 30

[[:fabrice] Fabrice Desré]

Hey David, sorry for being late there. I'll try to do it this week. I like a lot the idea of moving everything out to a build/ directory btw.

Comment 31

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Hi dclarke,

Yuren and I had a lengthy conversation to sync up what we had done v.s. your approach here. I am looking into your work right now to see if it fits all the requirements, especially around distributions.

Our approach in Taipei was progressive enhancement -- land changes in different bugs one-by-one to ensure everyone could follow up -- but if you have already done all the changes and we could still make sure it's safe enough for v1.0.1, then it doesn't make sense not to land it.

Comment 32

[[e-mail] [:onecyrenus]]

Tim, 
   That is great, would appreciate any extra eyes on this bug.

Comment 33

[[:fabrice] Fabrice Desré]

Comment on attachment 730956 [details]
pointer to gaia pull request

It looks like Tim started to review on github

Comment 35

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

I wrote what we plan for this bug on mozilla wiki:
https://wiki.mozilla.org/B2G/MarketCustomizations#Plan

feel free to update wiki page when this feature available :)

Comment 36

[Rudy Lu [:rudyl] (inactive after 2015/7/3)]

Hi all,

I can help look into this to see what's left to do if no one is actively working on this.
Please take this back if any concerns.

Thanks.

Comment 37

[[e-mail] [:onecyrenus]]

Rudy, sounds good this patch got stuck in review, so probably a fresh pair of eyes and a different perspective is what is needed.

Comment 38

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

steal from Rudy ;-)

Comment 39

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Don't forget to update https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/Gaia/Hacking after this lands.

Comment 40

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Attachment: Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427

Pointer to Github pull-request

Comment 41

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 763428 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427

There are two majar changes by this pull request:
1. use build/apps-[production|engineering|dogfood|demo|.list for picking apps.
2. use GAIA_DISTRIBUTION_DIR/apps.list if the environment variable defined

Comment 42

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

remind myself: we also need to update https://wiki.mozilla.org/B2G/MarketCustomizations

Comment 43

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Attachment: apps.list

you also can use specific app list to pick apps, download the attachment and type:
GAIA_APP_CONFIG=<path-to-apps.list> make reset-gaia

you will have a gaia only contains system, homescreen & settings app :)

Comment 44

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Comment on attachment 763428 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427

The patch looks generally alright as we have addresses all the use cases we could think of/find.

Please add the part we discussed so we could backward-compatible GAIA_APP_SRCDIRS with a warning message.

Comment 45

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

PR updated accroding Tim's review

Comment 46

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

Comment on attachment 763428 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427

I'm fine with what Tim and Alex think are OK with. The current system does not scale in many situations (mostly my bad) so do whatever you think is smarter :)

Comment 47

[Alexandre Poirot [:ochameau]]

Comment on attachment 763428 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427

Thanks for this great work and sorry for the delay in reviewing the patch.
I spent some time testing various environment and it appears to break Windows support (yes, some people build gaia on Windows ;)).

That would be really great if we can keep it running on this platform.

Comment 48

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

I'll find a windows VM to test it and thank you for investigating on windows!

Comment 49

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

I have updated PR, but we still need James's feedback on Pull Request page. also set ni? here.

https://github.com/mozilla-b2g/gaia/pull/10427#discussion_r4862925

Comment 50

[James Lal [:lightsofapollo]]

Looks good to me!

Comment 51

[Alexandre Poirot [:ochameau]]

Comment on attachment 763428 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/10427

Thanks for the fixes, you should go ahead and land this.
Do not forget to watch TBPL results, this is the kind of changes that can easily break test environments!

Comment 52

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

I'll watch TBPL for this and Askeing will help us to verify on Linux & Windows again.

Thank you Alex, Tim & Vivien!

Merged!
https://github.com/mozilla-b2g/gaia/commit/f1a33dc54c644dd07e5503a0de66c7ba0b7679d9

Comment 53

[Kim Moir [:kmoir] ET]

I think this change caused bug 887761

Comment 54

[Ed Morley [:emorley]]

(In reply to Kim Moir [:kmoir] from comment #53)
> I think this change caused bug 887761

Have reverted this for now:
https://github.com/mozilla-b2g/gaia/commit/b716b8391a8c9ad1ed97a9abc326c8edeb992556

The manifest auto-updater will soon check into birch, we'll need to wait for a desktop B2G clobber build after that to confirm it has fixed it.

Comment 55

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Thank you Kim & Ed, I'm investigating.

do we have the way to test on TPBL before merge to gaia master?

Comment 56

[Kim Moir [:kmoir] ET]

I talked to Hal, he said you could specify a different gaia rev in the gaia.json file (in gecko/firefox) and then run a build on try.

Comment 57

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

I can't reproduce on my local machine by:
- make -C <GAIA_DIR> profile
- <B2G_DIR>/build.sh
- pull gecko from mozilla-central & build with a mozconfig from https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Using_the_B2G_desktop_client

Kim, Where can I find build script for TPBL? I would like to reproduce it on local machine.

Comment 58

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

I got it.
http://hg.mozilla.org/build/mozharness/file/84187194b050/scripts/b2g_build.py

Comment 59

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

actually I don't have gaia rev before push to gaia master. talked to kanru and he said we may use repo_path.

Comment 60

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Attachment: https://github.com/mozilla-b2g/gaia/pull/10880

this is the same Pull request as attachment 763428 [details] but rebase to master & resolved some conflict.

I can't reproduce TBPL error on bug 887761, and then I triggered those TBPL build again I got another errors about webapp-optimize.

- https://tbpl.mozilla.org/?rev=3b955f306226
- https://tbpl.mozilla.org/?rev=aeae60c65cc7&tree=Mozilla-Inbound
- https://tbpl.mozilla.org/?rev=e5f664c8b66d&tree=Mozilla-Inbound

Alexandre & Tim, Do you have any suggestion?

Comment 61

[Alexandre Poirot [:ochameau]]

The following error:
build/webapp-optimize.js:487: ReferenceError: GAIA_CONCAT_LOCALES is not defined

is most likely due to merge issue against bug 853933.
But looking at your pull request, this GAIA_CONCAT_LOCALES seems to be correctly set.

Comment 62

[Tim Guan-tin Chien [:timdream] (please needinfo)]

I am not familiar with TPBL :'(

Jonathan, would you be able to help out and take a look of the log?

Comment 63

[Ed Morley [:emorley]]

Note the error only occurred on clobber builds.

Comment 64

[Alexandre Poirot [:ochameau]]

I'm not sure Jonathan can really help us figuring out what is being broken in gaia build system only on tbpl.
I think what would help is giving access to Yurun access to a build slave and help him build gaia the same way our automated scripts do.

It should be quite easy to figure out what is going wrong once we reproduce it and can instrument gaia.

Comment 65

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

sounds helpful if I can access to build slave and reproduce this bug :D

and I'm also clone tbpl and try to setup environment, maybe I can reproduce on my local machine.

Comment 66

[Ed Morley [:emorley]]

(In reply to Yuren Ju [:yurenju] from comment #65)
> and I'm also clone tbpl and try to setup environment, maybe I can reproduce
> on my local machine.

TBPL is just a dashboard for displaying results, so isn't what you want to set up locally. Your best bet is to request access to a releng machine. See:
https://wiki.mozilla.org/Platform/2013-07-02#Automation_Corner_.28ctalbert.2Fjoduinn.29

Comment 67

[Jonathan Griffin (:jgriffin)]

Having access to a build slave won't help you produce the builds in the same way that buildbot does.  It would be quite difficult to try and reproduce that locally, even using a build slave, I believe, unless the error is just a matter of the build slaves being "slow" compared to the typical developer machine.

I'm cc'ing some rel-eng folks who are involved in the builds, maybe they can help.

catlee, do you have any idea about what's going on with the B2G build error in e.g., 

https://tbpl.mozilla.org/?rev=aeae60c65cc7&tree=Mozilla-Inbound

Comment 68

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

bug 892318 filed for requesting a releng machine.

Comment 69

[Alexandre Poirot [:ochameau]]

Oh, there is really something weird with tbpl tests.
I'm seing the GAIA_CONCAT_LOCALES exception on a push-to-try (that has nothing to do with gaia!):
  https://tbpl.mozilla.org/?tree=Try&rev=736c69ef34dd

Can it be that bug 853933 regressed tbpl, but that we don't see it unless pushing manually to try???

Comment 70

[Alexandre Poirot [:ochameau]]

Ok, I don't know what is going on with the test runner, but something bad happens.
This following run:
  https://tbpl.mozilla.org/?tree=Try&rev=736c69ef34dd
is based on an old gecko, around June 25th, its gaia.json targets a gaia revision around the same date.
But we are seeing an exception about GAIA_CONCAT_LOCALES, that is something brand new, that only exists in gaia since bug 853933 landed on July 4th. There is no way to see this tring appear on a gaia from June 25th.

I have no idea what exactly happens, but it looks like gaia/Makefile file is from June 25th, but all files from gaia/build/ folder is from master (or from a revision after July 4th)

I've just pushed the same gecko patch but against a recent gecko/gaia just to see what will happen:
  https://tbpl.mozilla.org/?tree=Try&rev=b71104170ebf

Given these strange behaviors, I don't think it has anything to do with your patch Yuren.

Comment 71

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

sounds not my fault :D

Ed, can I land this commit again?

Comment 72

[Ed Morley [:emorley]]

(In reply to Yuren Ju [:yurenju] from comment #71)
> sounds not my fault :D
> 
> Ed, can I land this commit again?

If the issue is fixed, sure :-)
(Though there aren't b2g desktop builds in the comment 70 try run...)

Comment 73

[Chris AtLee [:catlee]]

> catlee, do you have any idea about what's going on with the B2G build error
> in e.g., 
> 
> https://tbpl.mozilla.org/?rev=aeae60c65cc7&tree=Mozilla-Inbound

I don't sorry. That looks like it's deep in the gaia build system, which I don't know much about.

Comment 74

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

I have same conclusion as Alexandre said on comment 70, bug 887761 looks like a test runner bug because Makefile and scripts in build/ came from different date, that's why I can't reproduce bug 887761 on my local machine even I use the same commit hashtag from gaia-integration with mercurial.

remove dependency with bug 887761 and I'll land this soon.

Comment 75

[Ed Morley [:emorley]]

(In reply to Yuren Ju [:yurenju] from comment #74)
> I have same conclusion as Alexandre said on comment 70, bug 887761 looks
> like a test runner bug because Makefile and scripts in build/ came from
> different date, that's why I can't reproduce bug 887761 on my local machine
> even I use the same commit hashtag from gaia-integration with mercurial.
> 
> remove dependency with bug 887761 and I'll land this soon.

Note even if this is a test runner bug, if this patch is enough to trigger it, then we'll have to backout again, removed dependency or not.

Comment 76

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Merged again.

https://github.com/mozilla-b2g/gaia/commit/b54c694

Comment 77

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

got new result.

now we have webapp-optimize error but not applications-data.

https://tbpl.mozilla.org/?rev=3b955f306226

Comment 78

[Ed Morley [:emorley]]

(In reply to Yuren Ju [:yurenju] from comment #77)
> got new result.
> 
> now we have webapp-optimize error but not applications-data.
> 
> https://tbpl.mozilla.org/?rev=3b955f306226

That TBPL link is from June?

Comment 79

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Oh sorry I gave a wrong link. I thought retrigger build process will get recent gaia from '/integration/gaia-central', but looks not.

Comment 80

[Antonio Manuel Amaya Calvo (:amac)]

With this change applied, I can't have apps as symbolic links anymore (it was working before). That is, I used to be able to do:

cd apps; ln -s /whereverMyNewAppIs/myappDir .; cd ..; make profile

and I would get a myappDir.gaiamobile.org app included on the profile. Now it doesn't work.

Comment 81

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Hi Antonio, could you file a bug for it and CC me? thanks!

Comment 82

[Antonio Manuel Amaya Calvo (:amac)]

(In reply to Yuren Ju [:yurenju] from comment #81)
> Hi Antonio, could you file a bug for it and CC me? thanks!
Created bug 902402.