Closed Bug 908691 Opened 11 years ago Closed 11 years ago

Windows desktop B2G builds are broken again

Categories

(Firefox OS Graveyard :: GonkIntegration, defect)

Product:
Firefox OS Graveyard
Component:
GonkIntegration
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: RyanVM, Assigned: julienw)

References

Details

A yet-to-be-determined Gaia change has once again broken Windows desktop B2G builds, as visible on the m-c nightly builds.

https://tbpl.mozilla.org/php/getParsedLog.php?id=26920697&tree=Mozilla-Central

b2g_mozilla-central_win32_gecko nightly on 2013-08-23 04:02:03 PDT for push fb2318875cd4
slave: w64-ix-slave75

make[7]: Leaving directory `/e/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/gaia'
e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\obj-firefox\b2g\gaia\Makefile:38:0: command 'make -j1 -C e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/gaia profile' failed, return code 2
e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\config\makefiles\target_libs.mk:34:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py -C gaia libs' failed, return code 2
evaluation from e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\obj-firefox\Makefile:107:24:25:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py -C b2g libs' failed, return code 2
evaluation from e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\obj-firefox\Makefile:107:24:13:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py libs_tier_app' failed, return code 2
e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\obj-firefox\Makefile:104:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py  tier_app' failed, return code 2
e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\client.mk:372:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py -j4 -C obj-firefox' failed, return code 2
e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\client.mk:172:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py -f e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/client.mk realbuild' failed, return code 2
/bin/bash: xulrunner-sdk-26/.url: No such file or directory
make[7]: *** [install-xulrunner-sdk] Error 1
program finished with exit code 2
hmm, bustage from bug 908580 maybe?
Summary: Windows desktop B2G builds broken again → Windows desktop B2G builds are broken again
Might be related to bug 908698.
I supposed busted nightlies should be considered a blocker.
Severity: normal → blocker
Depends on: 909190
Assignee: nobody → yurenju.mozilla
will backout the various patches.
Assignee: yurenju.mozilla → felash
Discussed this with Ryan, we can wait until bug 909190 lands, because the patch is nearly ready, which should fix the Windows build.

Yuren, I'm putting a needinfo for you here so that you try to land bug 909190 quickly, and you update this bug when bug 909190 lands.

Thanks all, and next time I'll take a lot more care testing such patches on several environments.
Assignee: felash → yurenju.mozilla
Flags: needinfo?(yurenju.mozilla)
I was told this morning that this would be top priority when Taipei woke up, which is why we agreed to not go the backout route. Yet here we are, 9+ hours since that conversation with no activity. Is it time to start backing things out to get these builds green again?
Flags: needinfo?(felash)
Flags: needinfo?(fabrice)
bug 909190 was landed, let's wait TBPL result to see if bug 909190 can fix this issue.
Status: NEW → ASSIGNED
Flags: needinfo?(yurenju.mozilla)
Flags: needinfo?(felash)
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #9)
> I was told this morning that this would be top priority when Taipei woke up,
> which is why we agreed to not go the backout route. Yet here we are, 9+
> hours since that conversation with no activity. Is it time to start backing
> things out to get these builds green again?

Ah I backed out bug 900504 over this, since it was the last to touch the file in question. Guessing not related then? (That and the retriggers since are busted; but I'd put that down to retriggering too soon)
probably unrelated, yep.
bug 900504 is another story, so I thought bug 909190 solved this issue.
Looks like they're still busted. From today's nightlies:
https://tbpl.mozilla.org/php/getParsedLog.php?id=27112332&tree=Mozilla-Central

mv: cannot stat `/xulrunner-sdk': No such file or directory
make[7]: Leaving directory `/e/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/gaia'
e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\obj-firefox\b2g\gaia\Makefile:38:0: command 'make -j1 -C e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/gaia profile' failed, return code 2
e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\config\makefiles\target_libs.mk:34:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py -C gaia libs' failed, return code 2
evaluation from e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\obj-firefox\Makefile:106:24:25:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py -C b2g libs' failed, return code 2
evaluation from e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\obj-firefox\Makefile:106:24:13:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py libs_tier_app' failed, return code 2
e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\obj-firefox\Makefile:103:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py  tier_app' failed, return code 2
rm: cannot lstat `': No such file or directory
make[7]: *** [install-xulrunner-sdk] Error 1
e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\client.mk:372:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py -j4 -C obj-firefox' failed, return code 2
e:\builds\moz2_slave\m-cen-w32_g-ntly-0000000000000\build\client.mk:172:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/build/pymake/pymake/../make.py -f e:/builds/moz2_slave/m-cen-w32_g-ntly-0000000000000/build/client.mk realbuild' failed, return code 2
program finished with exit code 2
I'll revert all the patches, please give me one ot two hours as that's quite complicated now.

We'll need to know how to actually reproduce the environment you use to build. Could make.py be the culprit here ? I know Yuren tried his build on the gnu make from msys.
Official builds use pymake. AFAIK, gnu make isn't even officially supported for Windows builds anymore.
I get it, but do you have precise steps to reproduce the environment used for Windows builds ?
Install Mozillabuild? Windows B2G builds run on the same slaves as all the other Windows builds.
Reverted bug 906316 and all dependent patches.
Blocks: 879370
No longer depends on: 879370
ok, the key error in the new problem is "/bin/bash: mktemp: command not found".

We don't have this in MozillaBuild.

Any alternative to using mktemp on windows ? Or could we add it on the build environment ?

The good news is that I reproduce the exact same problem here. :-)
Assignee: yurenju.mozilla → felash
Flags: needinfo?(fabrice)
I haven't yet tested this in Gaia, but this Python one-liner *should* provide the equivalent functionality:

  python -c 'from tempfile import mkstemp; print mkstemp()[1]'
Thanks Myk! I don't know yet if I want to add one more dependency on python here, but why not.

I also see that I can use "$RANDOM", so maybe `mkdir "$TEMP/gaia.tmp.$RANDOM` is enough for our needs here.

But I'd rather have mktemp.exe in MozillaBuild...
Filed bug 910637 for this.
(In reply to Julien Wajsberg [:julienw] from comment #22)
> Thanks Myk! I don't know yet if I want to add one more dependency on python
> here, but why not.

The script in question is a shell script, and as long as you depend on the shell, you might as well depend on Python too.


> I also see that I can use "$RANDOM", so maybe `mkdir
> "$TEMP/gaia.tmp.$RANDOM` is enough for our needs here.
> 
> But I'd rather have mktemp.exe in MozillaBuild...

Sure, although it looks like the script also depends on optipng, advpng, and bc, which aren't in MozillaBuild either.
(In reply to Myk Melez [:myk] [@mykmelez] from comment #24)
> (In reply to Julien Wajsberg [:julienw] from comment #22)
> > Thanks Myk! I don't know yet if I want to add one more dependency on python
> > here, but why not.
> 
> The script in question is a shell script, and as long as you depend on the
> shell, you might as well depend on Python too.

Erm, actually I'm not sure the script to which I am referring is the actual problem.  I see mktemp in tools/png_recompress.sh, but I don't actually see where that script is invoked, and it doesn't seem to get invoked during a standard build on my Windows box.

grep also claims that build/busybox-armv6l contains mktemp, but that's a binary, and presumably it is calling the libc function.

These are the errors I get when building the Gaia tip:

/bin/sh: buildapps-engineering.list: No such file or directory
c:\Users\myk\Mozilla\gaia\Makefile:523:0$ test -d profile || mkdir -p profile
run-js-command  applications-data
Looking for packaged app: c:UsersmykMozillagaia\apps\communications\update.webapp
Invalid JSON file : c:UsersmykMozillagaia\apps\communications\update.webapp
build/utils.js:109: Error:  -*- build/utils.js: [Exception... "Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIFileInputStream.init]"  nsresult: "0x80520012 (NS_ERROR_FILE_NOT_FOUND)"  location: "JS frame :: build/utils.js :: getFileContent :: line 93"  data: no]
build/utils.js:109: file not found: c:UsersmykMozillagaia\apps\communications\update.webapp
build/utils.js:109:
c:\Users\myk\Mozilla\gaia\Makefile:524:0: command 'echo "run-js-command  applications-data"; JS_CONSTS=' const GAIA_DIR = "c:\Users\myk\Mozilla\gaia"; const PROFILE_DIR = "c:\Users\myk\Mozilla\gaia\\profile"; const PROFILE_FOLDER = "profile"; const GAIA_SCHEME = "app://"; const GAIA_DOMAIN = "gaiamobile.org"; const DEBUG = 0; const LOCAL_DOMAINS = 1; const DESKTOP = 0; const DEVICE_DEBUG = 0; const HOMESCREEN = "app://system.gaiamobile.org"; const GAIA_PORT = ""; const GAIA_LOCALES_PATH = "locales"; const LOCALES_FILE = "shared/resources/languages.json"; const BUILD_APP_NAME = "*"; const PRODUCTION = "0"; const GAIA_OPTIMIZE = "0"; const GAIA_DEV_PIXELS_PER_PX = "1"; const DOGFOOD = "0"; const OFFICIAL = ""; const GAIA_DEFAULT_LOCALE = "en-US"; const GAIA_INLINE_LOCALES = "1"; const GAIA_CONCAT_LOCALES = "1"; const GAIA_ENGINE = "xpcshell"; const GAIA_DISTRIBUTION_DIR = "c:\Users\myk\Mozilla\gaia\\distribution"; const GAIA_APPDIRS = ""; ';  ./xulrunner-sdk/bin/xpcshell -e "$JS_CONSTS" -f build/utils.js "build/applications-data.js"' failed, return code 3
(In reply to Myk Melez [:myk] [@mykmelez] from comment #25)
> These are the errors I get when building the Gaia tip:

Erm, sorry, that was on an older branch.  Here's what I get on the Gaia tip:

make.py[0]: Entering directory 'c:\Users\myk\Mozilla\gaia'
/bin/sh: buildapps-engineering.list: No such file or directory
c:\Users\myk\Mozilla\gaia\Makefile:482:0$ echo "$BUILD_CONFIG" > c:\Users\myk\Mozilla\gaia\\build\\config.js
/bin/sh: c:UsersmykMozillagaiabuildconfig.js: Permission denied
c:\Users\myk\Mozilla\gaia\Makefile:482:0: command 'echo "$BUILD_CONFIG" > c:\Users\myk\Mozilla\gaia\\build\\config.js' failed, return code 1
(In reply to Myk Melez [:myk] [@mykmelez] from comment #26)
> Here's what I get on the Gaia tip:

However, it only fails with pymake; it builds fine with regular make:

08-29 04:48 > pymake
make.py[0]: Entering directory 'c:\Users\myk\Mozilla\gaia'
/bin/sh: buildapps-engineering.list: No such file or directory
c:\Users\myk\Mozilla\gaia\Makefile:482:0$ echo "$BUILD_CONFIG" > c:\Users\myk\Mozilla\gaia\\build\\config.js
/bin/sh: c:UsersmykMozillagaiabuildconfig.js: Permission denied
c:\Users\myk\Mozilla\gaia\Makefile:482:0: command 'echo "$BUILD_CONFIG" > c:\Users\myk\Mozilla\gaia\\build\\config.js' failed, return code 1
08-29 04:48 > make
echo "$BUILD_CONFIG" > c:\\Users\\myk\\Mozilla\\gaia\\build\\config.js
test -d profile || mkdir -p profile
run-js-command  applications-data
…
That $BUILD_CONFIG problem comes from Bug 905096, don't know if we should backout it then ?

So, Myk, sorry for being confusing :) Here is a little background about my comments. This bug was originally created because bug 906316 and its follow-up broke the windows build. My comments were only targeted at that patch, which you don't see in the current tip because I reverted it yesterday. We used mktemp in that patch.

The script you're talking about is (afaik) only launched directly, and is not part of the build process.

Hope this makes all this clearer.
so, windows desktop builds work now, thanks to the revert in comment 23.

Filed bug 910694 for the pymake bug.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.