Closed Bug 887761 Opened 11 years ago Closed 11 years ago

B2G desktop clobber builds fail with "make[7]: *** [applications-data] Error 3" after "build/utils.js:219: ReferenceError: GAIA_APPDIRS is not defined"

Categories

(Firefox OS Graveyard :: Gaia, defect)

Product:
Firefox OS Graveyard
Component:
Gaia
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [buildduty])

Attachments

(1 file)

pullrequest 11056
168 bytes, text/html
Pike
: review+
Details 
b2g_mozilla-inbound_linux64_gecko build on 2013-06-27 01:01:24 PDT for push 59f88c16fca7

slave: bld-linux64-ec2-031

https://tbpl.mozilla.org/php/getParsedLog.php?id=24653499&tree=Mozilla-Inbound

{
2013-06-27 05:02:40 (4.40 MB/s) - `xulrunner-24.0a1.en-US.linux-x86_64.sdk.tar.bz2' saved [65344299/65344299]

tar xjf xulrunner*.tar.bz2 && rm xulrunner*.tar.bz2
test -d profile || mkdir -p profile
run-js-command  applications-data
build/utils.js:219: ReferenceError: GAIA_APPDIRS is not defined
build/utils.js:258: TypeError: Gaia is undefined
make[7]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/gaia'
make[7]: *** [applications-data] Error 3
make[6]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/obj-firefox/b2g/gaia'
make[6]: *** [libs] Error 2
make[5]: *** [libs] Error 2
make[5]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/obj-firefox/b2g'
make[4]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/obj-firefox'
make[4]: *** [libs_tier_app] Error 2
make[3]: *** [tier_app] Error 2
make[3]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/obj-firefox'
make[2]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build/obj-firefox'
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make[1]: Leaving directory `/builds/slave/m-in-l64_g-0000000000000000000/build'
make: *** [build] Error 2
State Changed: unlock buildroot
program finished with exit code 2
elapsedTime=3168.958343
========= Finished compile failed (results: 2, elapsed: 52 mins, 50 secs) (at 2013-06-27 02:02:52.275564) =========
}
NB this only appears to happen on clobber builds.
b2g_mozilla-central_linux32_gecko_localizer nightly on 2013-06-27 03:12:24 PDT for push 3b955f306226

slave: bld-linux64-ix-004

https://tbpl.mozilla.org/php/getParsedLog.php?id=24655145&tree=Mozilla-Central
Talked to :ochameau  in #b2g, he suggested bug 848604 might be the cause of this issue.  Moving bug to b2g gaia
Component: Release Engineering: Automation (General) → Gaia
Product: mozilla.org → Boot2Gecko
QA Contact: catlee
Version: other → unspecified
Blocks: 848604
See Also: → 880773
See bug 848604 comment 70.
At the end, I think that bug 848604 as well as bug 853933 are just exposing some issues with build/test scripts.
Hi Ed, I got a loaner machine, Do you know how to build b2g as same as automation build mechanism?
Flags: needinfo?(emorley)
I'm sorry, I don't. You'll need to ask someone from releng.
Flags: needinfo?(emorley)
Hi Chris, do you know about that?
Flags: needinfo?(catlee)
Basically copy/paste from the log.

The B2G desktop builds are just like regular builds, except with a different mozconfig. You can copy b2g/configs/mozconfigs/$platform/nightly to .mozconfig to get started.
Flags: needinfo?(catlee)
Assignee: nobody → yurenju.mozilla
after investigated I found some thing strange.
I use below build to reproduce bug:
https://tbpl.mozilla.org/?rev=56423bbe9a13&tree=Mozilla-Inbound

target gaia revision of this build is c59267134f80ccb38353a46c51b210009dba0391 which doesn't include changes of bug 848604
but below command copy another revision of gaia code (include commit of bug 848604) and replaced to build/gaia:

    mock_mozilla -r mozilla-centos6-i386 --cwd /builds/slave/m-in-linux32_g-000000000000000/build --unpriv --shell '/usr/bin/env HG_SHARE_BASE_DIR="/builds/hg-shared" LOCALES_FILE="locales/languages_dev.json" CCACHE_BASEDIR="/builds/slave/m-in-linux32_g-000000000000000" SYMBOL_SERVER_HOST="symbolpush.mozilla.org" SYMBOL_SERVER_USER="ffxbld" CCACHE_DIR="/builds/ccache" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" SYMBOL_SERVER_SSH_KEY="/home/cltbld/.ssh/ffxbld_dsa" LOCALE_BASEDIR="/builds/slave/m-in-linux32_g-000000000000000/build-gaia-l10n" PATH="/tools/python27-mercurial/bin:/tools/python27/bin:${PATH}:/tools/buildbot/bin" TINDERBOX_OUTPUT="1" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_ffx/" MOZ_OBJDIR="obj-firefox" MOZ_CRASHREPORTER_NO_REPORT="1" CCACHE_COMPRESS="1" LC_ALL="C" CCACHE_UMASK="002" make -f client.mk build '"'"'MOZ_BUILD_DATE=20130717051921'"'"''

but when I executed this command again then everything is fine. hope this information is useful for tracking this bug.
Assignee: yurenju.mozilla → nobody
Aki, Do you have any thoughts about that?

The try run investigated by Yuren:
  https://tbpl.mozilla.org/?rev=56423bbe9a13&tree=Mozilla-Inbound
Fails here (fedora opt Bg):
  https://tbpl.mozilla.org/php/getParsedLog.php?id=25376313&tree=Mozilla-Inbound
With an exeption message that proves that gaia/build/utils.js file has bug 848604 revision applied:
  build/utils.js:219: ReferenceError: GAIA_APPDIRS is not defined
  (GAIA_APPDIRS is a new variable introduce only by bug 848604)
*but* this try run is against this mozilla-central changeset 56423bbe9a13:
  https://hg.mozilla.org/integration/mozilla-inbound/file/56423bbe9a13/b2g/config/gaia.json
  { "revision": "c59267134f80ccb38353a46c51b210009dba0391", 
    "repo_path": "/integration/gaia-central" }
So we should be having following gaia revision:
  http://hg.mozilla.org/integration/gaia/file/c59267134f80/build/utils.js
And you can see that GAIA_APPDIRS doesn't exists in this revision.

There is an obvious revision issue in test runner script, where we end up with mixed up revision between gaia Makefile (that is at root gaia folder) and files in gaia's build/ folder.

We can suspect gaia.json to badly update the gaia folder, but the logs seems to say that the correct hg commands are being run without any particular error:
  command: START

  command: hg pull -r c59267134f80ccb38353a46c51b210009dba0391 http://hg-
  internal.dmz.scl3.mozilla.com//integration/gaia-central
  command: cwd: /builds/hg-shared/integration/gaia-central
  command: output:
  pulling from http://hg-internal.dmz.scl3.mozilla.com//integration/gaia-central
  no changes found
  command: END (0.84s elapsed)

  Trying to share /builds/hg-shared/integration/gaia-central to /builds/slave
  /m-in-linux32_g-000000000000000/build/gaia

  command: START
  command: hg share -U /builds/hg-shared/integration/gaia-central /builds/slave
  /m-in-linux32_g-000000000000000/build/gaia
  command: cwd: /builds/slave/m-in-linux32_g-000000000000000/build
  command: output:
  command: END (0.24s elapsed)

  command: START
  command: hg update -C -r c59267134f80ccb38353a46c51b210009dba0391
  command: cwd: /builds/slave/m-in-linux32_g-000000000000000/build/gaia
  command: output:
  4512 files updated, 0 files merged, 0 files removed, 0 files unresolved
  command: END (23.33s elapsed)

I tend to think there is something going wrong in this code:
  http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#1213
But I have absolutely no idea how to trigger this code, if we want to reproduce this locally.

Chris, Aki, Could you either take a look at this, or find someone to do so, and/or help Yuren to debug gaia's folder update/cloning?
Flags: needinfo?(aki)
A point of clarification: mozilla-inbound != Try, so this is a standard depend build, not a Try build.

I kicked off a staging mozilla-inbound build.  The slave had no existing build/ directory in /builds/slave/m-in-linux32_g-000000000000000 .

The gaia source step updated to c59267134f80ccb38353a46c51b210009dba0391 .

I ssh'ed in, and did an |hg ident| in /builds/slave/m-in-linux32_g-000000000000000/build/gaia and got c59267134f80.  An |hg diff| gave me nothing.  A |sum build/utils.js| gave me "26242     9" and there's no GAIA_APPDIRS in that file.

I'm letting it run to see if something in the build updates gaia mid-build.
Flags: needinfo?(aki)
[cltbld@bld-centos6-hp-006.build.scl1.mozilla.com gaia]$ sum build/utils.js
48595     8
[cltbld@bld-centos6-hp-006.build.scl1.mozilla.com gaia]$ !vi
vi build/utils.js
[cltbld@bld-centos6-hp-006.build.scl1.mozilla.com gaia]$ hg ident
20b7d51bd03d+ tip


During the vi step, I verified that GAIA_APPDIRS exists in that file.

After the |hg ident|, I did an |hg diff|; the diff appears to all be l10n-related.

My suspicion has been that gaia l10n has been causing these issues; I think I just verified.
Thanks a lot for looking at this!

Are you able to reproduce this "GAIA_APPDIRS is not defined" exception?
(In reply to Alexandre Poirot (:ochameau) from comment #29)
> Thanks a lot for looking at this!
> 
> Are you able to reproduce this "GAIA_APPDIRS is not defined" exception?

Yes.

Stas: any way we can prevent the gaia-l10n makefile step from updating gaia?
Flags: needinfo?(stas)
Oh would it be because of this command?
https://github.com/mozilla-b2g/gaia/blob/master/Makefile#L299
(I'm not a hg expert, does hg update, just reset everything to the tip?)

That seems to be executed only when we are using hg instead of git 
*and* when we pass a LOCALE_BASEDIR variable.
That looks right.
Attached file pullrequest 11056 — 
So I think we do want another hg command here.
But we should really find resources/time to come up with gaia objdir in order to prevent this kind of hack...
Attachment #778060 - Flags: review?(stas)
Comment on attachment 778060 [details]
pullrequest 11056

Looks good to me, r=me, I think revert is what we want. Using `hg update --clean .` with '.' as revision would have worked, too. The differences between the two are subtle, and according to hg help only affect ongoing merges.

http://stackoverflow.com/questions/4681939/revert-vs-update-for-uncommited-changes backs that up.
Attachment #778060 - Flags: review?(stas) → review+
Flags: needinfo?(stas)
sry, gotta hide from tbpl. needinfo me if you need further feedback.
master: 4183c938541b4a3aeb8ba67eb81c86f6e35e6625
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Thanks again Aki to help us figuring out what was going wrong!!
It looks like the patch that was merged was broken. It had 2 spaces rather than a tab.

I just pushed:

https://github.com/mozilla-b2g/gaia/pull/11136
https://github.com/mozilla-b2g/gaia/commit/79d65c8ee9a49e6cbeb66cb2586badd7d7665bc4

which replaces the spaces with a tab.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: