951450 - Flashing customizations currently fails with exception: Error: Invalid position for app Twitter

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::System, defect)

Description

[John Hammink]

Tested on both 1.2 and 1.3 build (11/15 and 12/9 base images) on Buri

STR:

1.) Flash the latest build for the version you are aiming to test
2.) On a clean system, pull Gaia locally with git installed by running "git clone https://github.com/mozilla-b2g/gaia.git"
3.) Pull the reference customization locally with git installed by running "git clone https://github.com/mozilla/qa-testcase-data.git"
4.) In the Gaia directory, run "git checkout branchname", where branchname equals the branch you are testing (e.g. if you want 1.2, you would run git checkout v1.2).  I tried with both 1.2 and 1.3
5.) In the Gaia directory, run "MOZILLA_OFFICIAL=1 GAIA_DISTRIBUTION_DIR=../qa-testcase-data/customization/reference make production" 

Expected:  You should get the reference customization installed to the device without errors.

Actual:   Make production command fails (full dump here: http://pastebin.mozilla.org/3799067)

Exception: Error: Invalid position for app Twitter
validateHomescreen/</<@resource://gre/modules/commonjs/toolkit/loader.js -> file:///Users/johnhammink/Documents/gits/gaia/build/variant.js:358
validateHomescreen/<@resource://gre/modules/commonjs/toolkit/loader.js -> file:///Users/johnhammink/Documents/gits/gaia/build/variant.js:339
validateHomescreen@resource://gre/modules/commonjs/toolkit/loader.js -> file:///Users/johnhammink/Documents/gits/gaia/build/variant.js:309
execute@resource://gre/modules/commonjs/toolkit/loader.js -> file:///Users/johnhammink/Documents/gits/gaia/build/variant.js:414
@-e:1

make: *** [local-apps] Error 3

Comment 1

[John Hammink]

This is a definite blocker, as it (or the issue at the root of it) is preventing SIM customization testing.

Comment 2

[Antonio Manuel Amaya Calvo (:amac)]

Albert, Carmen, can you take a look at this?

Comment 3

[Jason Smith [:jsmith]]

So twitter has a valid position here:

        {
          "id": "Twitter",
          "screen": 1,
          "location": 4
        },

That would place Twitter in the 5th location on the 2nd screen. The default 2nd screen places 11 apps on that screen - that means placing Twitter at the 5th location is possible, so this shouldn't error out.

This is a regression from bug 929572.

Comment 4

[Albert [:albert]]

I can reproduce for 1.3 but 1.2 works for me. The thing is that:

- v1.2 with e.me enabled -> do not allow to place apps in the first page.
- v1.2 with e.me disabled && v1.3 -> allow apps in the first page.

Comment 5

[Albert [:albert]]

(In reply to Albert [:albert] from comment #4)
> I can reproduce for 1.3 but 1.2 works for me. The thing is that:
> 
> - v1.2 with e.me enabled -> do not allow to place apps in the first page.
> - v1.2 with e.me disabled && v1.3 -> allow apps in the first page.

Sorry, I was testing in master instead of 1.3. I can't reproduce for 1.2 and 1.3, it works as expected.

However master is broken due to bug 929602, so I opened bug 951681 to fix it.

Comment 6

[Albert [:albert]]

Rafa, can you check this?

Comment 7

[Albert [:albert]]

Using the single variant repo at https://github.com/telefonicaid/firefoxos-gaia-testsbuild all work, the problem comes when using https://github.com/mozilla/qa-testcase-data.git

The init.json generated at build time is empty, just contains the apps for the dock, for this reason location 4 throws invalid position error.

Comment 8

[John Hammink]

Here's the build information for the phone

Buri 1.3; (11/15 1.2 base build with december 18 gecko and gaia on top)

Gaia      a99b23e73fe5630a877004746d0e7fcec1b6d653
Gecko     http://hg.mozilla.org/releases/mozilla-aurora/rev/369bdbff6c38
BuildID   20131218004002
Version   28.0a2
ro.build.version.incremental=eng.archermind.20131114.105818
ro.build.date=Thu Nov 14 10:58:33 CST 2013
Shallow Flash Done!

Comment 9

[John Hammink]

I will try the setup with TEF's repo.  Thanks, Albert.

Comment 10

[Albert [:albert]]

(In reply to Albert [:albert] from comment #7)
> Using the single variant repo at
> https://github.com/telefonicaid/firefoxos-gaia-testsbuild all work, the
> problem comes when using https://github.com/mozilla/qa-testcase-data.git
> 
> The init.json generated at build time is empty, just contains the apps for
> the dock, for this reason location 4 throws invalid position error.

init.json does not contain all the applications that will be in the homescreen, in this case having an empty homescreens.json applications defined in build/apps-xxxx.list are also installed. So maybe we need to open a bug to fill init.json with these apps.

Comment 11

[Jason Smith [:jsmith]]

(In reply to Albert [:albert] from comment #10)
> (In reply to Albert [:albert] from comment #7)
> > Using the single variant repo at
> > https://github.com/telefonicaid/firefoxos-gaia-testsbuild all work, the
> > problem comes when using https://github.com/mozilla/qa-testcase-data.git
> > 
> > The init.json generated at build time is empty, just contains the apps for
> > the dock, for this reason location 4 throws invalid position error.
> 
> init.json does not contain all the applications that will be in the
> homescreen, in this case having an empty homescreens.json applications
> defined in build/apps-xxxx.list are also installed. So maybe we need to open
> a bug to fill init.json with these apps.

So are you implying this is a bug in the customized build setup or a bug in the test customization? I can't tell by your comments.

Comment 12

[Jason Smith [:jsmith]]

(In reply to John Hammink from comment #9)
> I will try the setup with TEF's repo.  Thanks, Albert.

That's a significantly different workflow - that's not the right path to resolution here to be able to ensure we've tested the right workflows in the 1st test run. I think you instead should be changing the customization and flashing that to try to workaround the problem here.

Comment 13

[Albert [:albert]]

(In reply to Jason Smith [:jsmith] from comment #11)
> (In reply to Albert [:albert] from comment #10)
> > (In reply to Albert [:albert] from comment #7)
> > > Using the single variant repo at
> > > https://github.com/telefonicaid/firefoxos-gaia-testsbuild all work, the
> > > problem comes when using https://github.com/mozilla/qa-testcase-data.git
> > > 
> > > The init.json generated at build time is empty, just contains the apps for
> > > the dock, for this reason location 4 throws invalid position error.
> > 
> > init.json does not contain all the applications that will be in the
> > homescreen, in this case having an empty homescreens.json applications
> > defined in build/apps-xxxx.list are also installed. So maybe we need to open
> > a bug to fill init.json with these apps.
> 
> So are you implying this is a bug in the customized build setup or a bug in
> the test customization? I can't tell by your comments.

Is a bug in the make process.

Comment 14

[John Hammink]

If you can give me some heads up as to how to change the customization - and or investigate more deeply here.  I've tried removing "twitter" from variant.json, but I'm still seeing the same issue.  Will check back tomorrow.

Comment 15

[rafael.marquez]

I have executed the test using mozilla and Telefonica repositories  in v1.2 and v1.3 branches

**Mozilla

Repo: https://github.com/mozilla/qa-testcase-data.git
Command to flash the device: MOZILLA_OFFICIAL=1 GAIA_DISTRIBUTION_DIR="URL"/qa-testcase-data/customization/reference make production
V1.2 -> Works well 
V1.3 -> Works bad  


**Telefónica

Repo: https://github.com/telefonicaid/firefoxos-gaia-testsbuild.git
Command to flash the device: GAIA_DISTRIBUTION_DIR="URL"/firefoxos-gaia-testsbuild  PRODUCTION=1 make reset-gaia
V1.2 -> Works well 
V1.3 -> Works well


Albert has found a problem and he's reviewing what happens .

Comment 16

[Albert [:albert]]

To summarize the problems I've seen:

1 - Master does not work because a wrong init.json - bug 929602 modified some dependencies of gaia Makefile and currently single variant script is run before init.json is generated.

2 - The validation function for applications position is taking the dock's apps (element 0 in grid array) as a screen apps.

3 - If homescreens.json is empty (https://github.com/mozilla/qa-testcase-data/blob/gh-pages/customization/reference/homescreens.json), the init.json used to validate positions is also empty. Although not configured in the homescreens.json, there are applications that are installed ($GAIA_HOME/build/apps-production.list).

Point 1 is out of the scope of this bug, so I created bug 951681.

I'm working on a patch to fix points 2 and 3.

Comment 17

[Albert [:albert]]

Attachment: Patch

Comment 18

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Hi Albert,

for (3), can we solve this problem with extracting homescreen.json from applications-data.js[1]? then we can use this file to generate homescreen object from applications-data.js:customizeHomescreen()


[1] https://github.com/mozilla-b2g/gaia/blob/master/build/applications-data.js#L140-L178

Comment 19

[Albert [:albert]]

(In reply to Yuren Ju [:yurenju] from comment #18)
> Hi Albert,
> 
> for (3), can we solve this problem with extracting homescreen.json from
> applications-data.js[1]? then we can use this file to generate homescreen
> object from applications-data.js:customizeHomescreen()
> 
> 
> [1]
> https://github.com/mozilla-b2g/gaia/blob/master/build/applications-data.
> js#L140-L178

No, we can't because homescreen hardcoded in [1] because we can have applications that are not present but will be installed, for example if you launch a 'make production' command the build will contain the applications found in paths defined in [2] and maybe (sure) that some are not in the homescreen.json

Both (2 and 3) are currently fixed in the patch, don't you like the implemented solution?

[2] https://github.com/mozilla-b2g/gaia/blob/master/build/apps-production.list

Comment 20

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

I'm just thinking the better way to solve it, but looks your pr is the best way.

I'll review it later, thanks.

Comment 21

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 8350274 [details]
Patch

r=yurenju, please address some nits which I mention on github, thanks.

and George is implementing unit tests infrastructure for build scripts. we may add some tests for variant.js after he finish bug 905495.

Comment 22

[Albert [:albert]]

I though in other possible solutions, and one was to consider using GAIA_APPDIRS variable to override the homescreen in [1], but then we will have dependencies problems, because variant.js should run before applications-data.js to include single variant apps, so it would mean to merge applications-data with variant, can be crazy...

Comment 23

[Albert [:albert]]

Master: https://github.com/mozilla-b2g/gaia/commit/1ca7f67b46cf840202bb122273d9de51ba4e3026

Comment 24

[Albert [:albert]]

Master: https://github.com/mozilla-b2g/gaia/commit/1ca7f67b46cf840202bb122273d9de51ba4e3026

Comment 25

[Albert [:albert]]

Comment on attachment 8350274 [details]
Patch

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): bug 929572
[User impact] if declined: can't create single variant builds
[Testing completed]:
[Risk to taking this patch] (and alternatives if risky): low
[String changes made]: no

Comment 26

[Jason Smith [:jsmith]]

Marking blocking+ for critical functional regression - go ahead & land this for 1.3.

Comment 28

[John Ford [:jhford] CET/CEST Berlin Time]

Uplifted 1ca7f67b46cf840202bb122273d9de51ba4e3026 to:
v1.3: d2d10f0f7ea8df03e4e368671f86a9809265b9da

Comment 29

[Albert [:albert]]

Comment on attachment 8350274 [details]
Patch

Uplift already done, so clearing approval flag.

Comment 30

[Jason Smith [:jsmith]]

A sanity test of this with the QA Test Case Data customization shows this working. This one bug found while testing this in bug 957857 though - looks like preloaded contacts are no longer working.

Comment 31

[Jason Smith [:jsmith]]

(In reply to Jason Smith [:jsmith] from comment #30)
> A sanity test of this with the QA Test Case Data customization shows this
> working. This one bug found while testing this in bug 957857 though - looks
> like preloaded contacts are no longer working.

Actually, looks like this only works with a AT&T US SIM. The T-Mobile US SIM customization ran into a bit more trouble - see bug 957866.