1205257 - Spark distro does not rebuild fine

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Build, defect)

Description

[:gerard-majax]

This has been a long standing issue for me (and contributors).

STR:
 0. GAIA_DISTRIBUTION_DIR=distros/spark ./build.sh
 1. hack hack hack
 2. GAIA_DISTRIBUTION_DIR=distros/spark ./build.sh

Expected:
 Build succeeds

Actual:
 Build will fail on copying Gaia's default to $(PRODUCT_OUT)/system/b2g/defaults/. I have to remove everything from $(PRODUCT_OUT) to get a new build.

This is making my life miserable and loosing me precious time.

Comment 1

[:gerard-majax]

Who is responsible for this?

Comment 2

[:gerard-majax]

> cp -r /home/alex/codaz/Mozilla/b2g/devices/XperiaZ3c/B2G.L/gaia/profile/defaults/* out/target/product/aries/system/b2g/defaults/
> cp: impossible d'évaluer «/home/alex/codaz/Mozilla/b2g/devices/XperiaZ3c/B2G.L/gaia/profile/defaults/*»: Aucun fichier ou dossier de ce type
> make: *** [out/target/product/aries/system/gecko] Erreur 1
> make: *** Attente des tâches non terminées....

Sometimes I have to remove all out/ ...

Comment 3

[Adam Farden [:adfad666]]

try reverting https://github.com/mozilla-b2g/gonk-misc/commit/cd913c475eca6724a11ac0184fd78b449a481c23

This solved build issues I had due to a probable race condition, I noticed the error messages involve a lack of profiles.tar.gz (assuming this is the same build issue of course).

Comment 4

[Adam Farden [:adfad666]]

It seems the prefs are being built, deleted and then rebuilt before the profile.tar.gz is created. The first build will satisfy the dependency for gecko, but it's then suddenly missing due to the deletion and rebuild, and so gecko fails.

Comment 5

[Ricky Chien [:rickychien] (inactive)]

I don't verify it actually from b2g build script. According to comment 3, it probably break from gonk-misc.

Alex, does comment 3 solve your issue?

Comment 6

[:gerard-majax]

(In reply to Ricky Chien [:rickychien] from comment #5)
> I don't verify it actually from b2g build script. According to comment 3, it
> probably break from gonk-misc.
> 
> Alex, does comment 3 solve your issue?

Maybe, I don't know and I don't have time. But I don't accept this as a solution: we should have proper dependencies. Thing is, this is only breaking reliably when doing Spark build ...

Comment 7

[Ricky Chien [:rickychien] (inactive)]

If breaking is due to spark build, I think it's resonable causing by a race condition. Since spark will take a long time to download online resources and it probably cause race condition.

Comment 8

[:gerard-majax]

(In reply to Ricky Chien [:rickychien] from comment #7)
> If breaking is due to spark build, I think it's resonable causing by a race
> condition. Since spark will take a long time to download online resources
> and it probably cause race condition.

That's possible. Nobody wants to fix this ? It's really really time consuming and it is making my life miserable when working on other bugs ...

Comment 9

[Ricky Chien [:rickychien] (inactive)]

I can try to fix this according to comment 3.

Comment 10

[Ricky Chien [:rickychien] (inactive)]

I cannot reproduce it on my PC. 

Alex, I just tried to  GAIA_DISTRIBUTION_DIR=distros/spark ./build.sh twice but didn't get any failure. Could you tell me how to hack so that I can reproduce it?

Or you can try comment 3 to `cd gonk-misc` and revert that commit as Adam said.

Comment 11

[:gerard-majax]

I'm testing, but I will not revert anything. Ricky, can you make sure you have enough CPU power? I have 4 CPUs and I know Adam reproduced that on more powerful hardware.

Comment 12

[Ricky Chien [:rickychien] (inactive)]

I'm running on 4 cores intel i7 CPU, but cannot reproduce it every time.

Comment 13

[Ricky Chien [:rickychien] (inactive)]

Do you want to update B2G repo with git pull && ./repo sync -d?

Comment 14

[:gerard-majax]

(In reply to Ricky Chien [:rickychien] from comment #13)
> Do you want to update B2G repo with git pull && ./repo sync -d?

Yeah, don't worry, that is fine on that side.

Comment 15

[:gerard-majax]

I do confirm 100% repro on my side ...

Comment 16

[:gerard-majax]

Attachment: build-error.log

Comment 17

[:gerard-majax]

I have spotted that some of the downloads happening on Gaia side might take quite some time. Anyway, you have the build log when reproducing the failure in attachment 8673601 [details]. I am not doing anything special, it's a clean checkout of aries-l device.

Comment 18

[Ricky Chien [:rickychien] (inactive)]

It's reproducible in aries-l build. (I tried emulator build but it doesn't complain.)
Sometimes I get errors as same as yours but sometimes not (not clobber build).

And then I've spotted an another failure in my log the same as yours after reverting https://github.com/mozilla-b2g/gonk-misc/commit/cd913c475eca6724a11ac0184fd78b449a481c23.

> make[2]: *** [modules] Error 1
> make[1]: *** [sub-make] Error 2
> make[1]: Leaving directory `/home/ricky/Documents/B2G/kernel'
> make: [TARGET_KERNEL_BINARIES] Error 2 (ignored)
> make  -C kernel O=/home/ricky/Documents/B2G/out/target/product/aries/obj/KERNEL_OBJ INSTALL_MOD_PATH=../../system ARCH=arm CROSS_COMPILE="/home/ricky/Documents/B2G/prebuilts/misc/linux-x86/ccache/ccache arm-eabi-" modules_install
> make[1]: Entering directory `/home/ricky/Documents/B2G/kernel'
> 
> The present kernel configuration has modules disabled.
> Type 'make config' and enable loadable module support.
> Then build a kernel with module support enabled.
> 
> make[2]: *** [modules_install] Error 1
> make[1]: *** [sub-make] Error 2
> make[1]: Leaving directory `/home/ricky/Documents/B2G/kernel'

I have no idea what happened in B2G build system, but I'm pretty sure it works well when trying to build spark in Gaia side.

Unassigned myself because I guess it is a B2G build system bug and unfortunately being triggered only in spark build and I'm not the expert of B2G build :(

Comment 19

[:gerard-majax]

Ricky, please continue, that error is unrelated.

Comment 20

[Ricky Chien [:rickychien] (inactive)]

Please ignore the build errors on comment 18, it appears everytime but output still success.

I notice that build failure on comment 2 will arise when preload.py takes quite long time to download app. However, b2g build fails when this case and gaia build doesn't.

A workaround for you is to skip downloading step of preload tool if you aren't patching gaia.

For example, cd B2G/gaia and remove https://github.com/mozilla-b2g/gaia/blob/a9c86a906a2dcda363df542521d497539a83dfdd/build/preload.js#L24-L27

it works for me!

Comment 21

[Ricky Chien [:rickychien] (inactive)]

If we want to fix download problem for preload.py, we could ask someone who is familiar with https://github.com/mozilla-b2g/preload-app-toolkit.

Comment 22

[Ricky Chien [:rickychien] (inactive)]

preload.py is sometimes slowly when fetching app but I can fetch those apps quickly by myself by using firefox...

Comment 23

[:gerard-majax]

Right, it makes all sense, but I think the fact that preload.py takes some time is something we should just take into account. Preloading is used by Spark so I don't really think we can afford to skip it if I want to build Spark ...

Ricky, I think the issue is really that we have preload.py running and not being properly accounted for as a Make target dependency, hence why the breakage. Can you check if that is something on Gaia side? Thanks!

Comment 24

[Ricky Chien [:rickychien] (inactive)]

Your right, preload.js doesn't being given any target dependency [1]. I'll figure out how to sort it out. thanks!


[1] https://github.com/mozilla-b2g/gaia/blob/master/Makefile#L414

Comment 25

[GitHub Autolander Bot]

Attachment: [gaia] rickychien:reload-dependency > mozilla-b2g:master

Comment 26

[Ricky Chien [:rickychien] (inactive)]

Comment on attachment 8674918 [details] [review]
[gaia] rickychien:reload-dependency > mozilla-b2g:master

Julien, would you mind taking a look for this small patch? thanks!

Alexandre, could you please help me apply this patch to verify on your environment? thanks!

Comment 27

[:gerard-majax]

Comment on attachment 8674918 [details] [review]
[gaia] rickychien:reload-dependency > mozilla-b2g:master

Four consecutive successfull rebuilds, that looks good to me. Adam, maybe you can stop reverting and instead apply that one :)

Comment 28

[Adam Farden [:adfad666]]

Comment on attachment 8674918 [details] [review]
[gaia] rickychien:reload-dependency > mozilla-b2g:master

Working well after several rebuilds!

Comment 29

[Julien Wajsberg [:julienw]]

Comment on attachment 8674918 [details] [review]
[gaia] rickychien:reload-dependency > mozilla-b2g:master

r=me

thx !

Comment 30

[Ricky Chien [:rickychien] (inactive)]

Land in master:

https://github.com/mozilla-b2g/gaia/commit/e6d59c5a6fe0d42419aa48a519cfeee61fe3b0b7

Comment 31

[Adam Farden [:adfad666]]

Either this was not fixed properly or it has regressed by another change, but I just hit the profiles not found error when running this after a clean build:

export VARIANT=userdebug &&
export B2G_UPDATER=1 &&
export B2G_SYSTEM_APPS=1 &&
export B2G_UPDATE_CHANNEL=default &&
export B2G_FOTA_FULLIMG_PARTS="/boot:boot.img /system:system.img /recovery:recovery.img" &&
export MOZ_TELEMETRY_REPORTING=1 &&
export MOZ_CRASHREPORTER_NO_REPORT=1 &&
export GAIA_DISTRIBUTION_DIR=distros/spark &&
export GAIA_OPTIMIZE=1 &&
export LOCALE_BASEDIR=$PWD/gaia/locales &&
export LOCALES_FILE=$PWD/gaia/locales/languages_all.json &&
export GAIA_DEFAULT_LOCALE=en-GB &&
export ENABLE_DEFAULT_BOOTANIMATION=true &&
./build.sh gecko-update-fota-fullimg

Comment 32

[[:fabrice] Fabrice Desré]

Not sure why we try to maintain the spark distro. It's going away.

Comment 33

[Julien Wajsberg [:julienw]]

I believe that if there is an issue it will happen for any use of GAIA_DISTRIBUTION_DIR, so including for partner distributions.

Comment 34

[Ricky Chien [:rickychien] (inactive)]

Is it reproducible every time on your machine with a clean build?