Closed Bug 971803 Opened 10 years ago Closed 9 years ago

Intermittent Windows Desktop B2G build failures like "make[7]: *** [all] Error 1" "make[6]: *** [app-makefiles] Error 1" from "cp: will not create hard link"

Categories

(Firefox OS Graveyard :: General, defect)

Product:
Firefox OS Graveyard
Component:
General
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: RyanVM, Unassigned)

References

Details

(Keywords: intermittent-failure) 

These pop up intermittently and usually go away when retriggered. Not sure who should look into this.

https://tbpl.mozilla.org/php/getParsedLog.php?id=34564941&tree=Mozilla-Inbound

b2g_mozilla-inbound_win32_gecko build on 2014-02-12 07:55:18 PST for push 6cd04a76fa13
slave: w64-ix-slave129

make[7]: Leaving directory `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/apps/camera'
make[7]: Entering directory `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/apps/clock'
make[7]: Leaving directory `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/apps/clock'
make[6]: Leaving directory `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia'
c:\builds\moz2_slave\m-in-w32_g-0000000000000000000\build\obj-firefox\b2g\gaia\Makefile:30:0: command 'C:/mozilla-build/msys/local/bin/make.exe -j1 -C c:/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia profile' failed, return code 2
evaluation from c:\builds\moz2_slave\m-in-w32_g-0000000000000000000\build\config\recurse.mk:189:53:14:0: command 'C:/mozilla-build/python27/python.exe c:/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/build/pymake/pymake/../make.py -C gaia libs' failed, return code 2
c:\builds\moz2_slave\m-in-w32_g-0000000000000000000\build\config\recurse.mk:162:0: command 'C:/mozilla-build/python27/python.exe c:/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/build/pymake/pymake/../make.py -C b2g libs' failed, return code 2
c:\builds\moz2_slave\m-in-w32_g-0000000000000000000\build\config\rules.mk:599:0: command 'C:/mozilla-build/python27/python.exe c:/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/build/pymake/pymake/../make.py libs' failed, return code 2
c:\builds\moz2_slave\m-in-w32_g-0000000000000000000\build\client.mk:398:0: command 'C:/mozilla-build/python27/python.exe c:/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/build/pymake/pymake/../make.py -j4 -C obj-firefox' failed, return code 2
c:\builds\moz2_slave\m-in-w32_g-0000000000000000000\build\client.mk:185:0: command 'C:/mozilla-build/python27/python.exe c:/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/build/pymake/pymake/../make.py -f c:/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/client.mk realbuild' failed, return code 2
make[7]: *** No rule to make target `../../shared/js/format.js/Makefile', needed by `../../build_stage/clock/js/startup.js'.  Stop.
make[6]: *** [app-makefiles] Error 1
https://tbpl.mozilla.org/php/getParsedLog.php?id=34685549&tree=B2g-Inbound

Vivien, any ideas who might be able to help here?
Flags: needinfo?(21)
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #3)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=34685549&tree=B2g-Inbound
> 
> Vivien, any ideas who might be able to help here?

I think that this is partly related to the alameda stuff. So jburke may know better what's going on here.
Flags: needinfo?(21) → needinfo?(jrburke)
This looks like a clock/Makefile issue. It looks like that main target is looking for a "../../shared/js/format.js/Makefile" file, but that is not going to exist. It almost seems like there is something weird with the generation of SHARED_SOURCES, that is what I would think would generate the ../../shared/js/format.js' part, but not sure how Makefile gets appended.

Both camera and email use a similar Makefile construction for SHARED_RESOURCES, so I would expect to see failures in those cases too if it was a shared issue. Looking at the capture above, it seems like camera completed successfully. Also weird that it is intermittent and only on Windows. Maybe a version issue with whatever shim is used on Windows to get Makefile support?

I am getting out of my depth on the Makefile situation, going to ask :mcav for ideas and put this in the Clock component. However, :mcav, if you do not have ideas let's talk more in IRC to see if we can get ideas from other productivity folks.
Component: General → Gaia::Clock
Flags: needinfo?(jrburke) → needinfo?(m)
All of these but the first file appear to be cp freaking out about hard-links.  The Makefile problem from the first one looks like either a python/pymake proble, an underlying file-system freak-out like whatever is causing these other failures, and/or some rogue background process doing crazy stuff to the file-system.

Ryan, do you know if the file-system in use is even NTFS and capable of having hard-links?

http://mingw-users.1079350.n2.nabble.com/will-not-create-hard-link-td7368611.html 

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #1)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=34616873&tree=Mozilla-
> Central

Building system app to build_stage...
cp: will not create hard link `/c/builds/moz2_slave/m-cen-w32_g_lz-ntly-0000000000/build/gaia/build_stage/system/test/marionette/fakemusic' to directory `/c/builds/moz2_slave/m-cen-w32_g_lz-ntly-0000000000/build/gaia/build_stage/system/resources/images/backgrounds'


(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #2)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=34677715&tree=B2g-Inbound

cp -rp ../../shared ../../build_stage/camera/shared
cp: will not create hard link `../../build_stage/camera/shared/style/drawer/images/ui' to directory `../../build_stage/camera/shared/js/device_storage'
cp: will not create hard link `../../build_stage/camera/shared/style/time_selector/images/icons' to directory `../../build_stage/camera/shared/style/buttons/images/ui'


(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #3)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=34685549&tree=B2g-Inbound

cp: will not create hard link `/c/builds/moz2_slave/b2g-in-w32_g-00000000000000000/build/gaia/build_stage/communications/gmail' to directory `/c/builds/moz2_slave/b2g-in-w32_g-00000000000000000/build/gaia/build_stage/communications/ftu/test/marionette'
Component: Gaia::Clock → General
Flags: needinfo?(m)
I would assume they are NTFS, but I'm not overly familiar with how our test slaves are configured. John, can you help answer the questions in comment 6? :)
Flags: needinfo?(jhopkins)
NTFS does support creating hard links, eg. via the mklink command:

c:\Users\cltbld\tmp>mklink /h b a
Hardlink created for b <<===>> a

When I list those files from msys, I can see they share the same 'inode':

59658559 -rw-r--r-- 2 cltbld Administrators 13 Feb 18 06:33 a
59658559 -rw-r--r-- 2 cltbld Administrators 13 Feb 18 06:33 b

I can create a new hard link via 'ln b c' and once again the inodes match.
When changing one file's contents, they all change (as they should since they are hard linked).

Page [1] says of MozillaBuild: "It's the old tools in particular that can cause problems, since they often don't behave as expected, are buggy, or don't support command line arguments that have been taken for granted for years. For example, copying a source tree using cp -rf src1 src2 does not work correctly because of an old version of cp (it gives "cp: will not create hard link" errors for some files)."

Have we only recently started using hard-linked filed in our build tree?


[1] https://developer.mozilla.org/en-US/docs/Simple_Firefox_build/Windows_Firefox_build
Flags: needinfo?(jhopkins)
So, Gaia itself does not seem to be creating hard-links explicitly, though there are a few spots that should not affect these code-paths where symbolic links are used.

http://en.wikipedia.org/wiki/NTFS_symbolic_link says:
The default security settings in Windows Vista/Windows 7 disallow non-elevated administrators and all non-administrators from creating symbolic links. This behavior can be changed running "secpol.msc" the Local Security Policy management console (under: Security Settings\Local Policies\User Rights Assignment\Create symbolic links). It can be worked around by starting cmd.exe with Run as administrator option or the runas command.

Do we know that symbolic links don't get automatically converted to hardlinks or anything like that?

My quick audit was to check invocations of "ln " in the core Gaia Makefile, app makefiles, and logic in the "build/" subdir to make sure they used "-s".  Likewise, our "cp " invocations don't include "-l" which would imply hard-links.


I tried to find out how we could try upgrading the 'cp' in MozillaBuild, but am now confused.  In some other bugzilla bugs, I saw mention of gnuwin32 providing 'cp' or something like that, but it's not clear to me how gnuwin32 gets into MozillaBuild from the repo at http://hg.mozilla.org/mozilla-build/ to determine what versions are in use.
$ cp --version
cp (GNU coreutils) 5.97

We're using an ancient release of MSYS in the current MozillaBuild distribution - version 1.0.11. Unfortunately, MSYS upgrades have been minefields in the past, so they've basically not been done. I'm starting to look into what it would take to migrate over to MSYS2, but that's not going to be short-term.
contextifying (below), it's all hard-link problems.  2 are from the Gaia Makefile copying stuff, 1 is from a per-app Makefile copying stuff.

The most immediate workaround since it's not particularly obvious where the hard-links are coming from would seem to be to change our uses of verbatim "cp" to $(CP) and have CP on windows use some node.js-implemented copy function.

Of course, the begged question is where are the hard-links coming from.  I'm going to mail dev-gaia to see if anyone would know why those seem to be happening.


(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #11)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=34919177&tree=Mozilla-
> Inbound

Building communications app to build_stage...
cp: will not create hard link `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/build_stage/communications/import/test/unit' to directory `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/build_stage/communications/contacts/locales/fb'

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #12)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=34922289&tree=Mozilla-
> Inbound

Building homescreen app to build_stage...
cp: will not create hard link `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/build_stage/homescreen/everything.me/js/helpers' to directory `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/build_stage/homescreen/build'
cp: will not create hard link `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/build_stage/homescreen/everything.me/modules/CollectionsSuggest' to directory `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/build_stage/homescreen/collections/music'
cp: will not create hard link `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/build_stage/homescreen/everything.me/modules/Searchbar' to directory `/c/builds/moz2_slave/m-in-w32_g-0000000000000000000/build/gaia/build_stage/homescreen/collections/sports'

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #13)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=34924283&tree=Mozilla-
> Inbound

cp -rp ../../shared ../../build_stage/camera/shared
cp: will not create hard link `../../build_stage/camera/shared/style/switches/images/switch' to directory `../../build_stage/camera/shared/style/headers'
Summary: Intermittent Windows Desktop B2G build failures like "make[7]: *** No rule to make target `../../shared/js/format.js/Makefile', needed by `../../build_stage/clock/js/startup.js'. Stop." → Intermittent Windows Desktop B2G build failures like "make[7]: *** [all] Error 1" "make[6]: *** [app-makefiles] Error 1" from "cp: will not create hard link"
(In reply to TBPL Robot from comment #28)
> make[7]: *** No rule to make target `autoconfig/msn.com/ui', needed by
> `../../build_stage/email/js/mail_app.js'.  Stop.
> make[6]: *** [app-makefiles] Error 1

So this type of error (as opposed to the "oh no, hard links!" error) sounds a lot like John Ford's mention of bugs in the inode emulation layer at https://groups.google.com/d/msg/mozilla.dev.gaia/6FwDftBbsHU/79v3U3B3W9kJ which also explains the hard links problem.

That specific problem suggests that the 'make' in use is subject to the inode emulation issue too.  Are we using the wrong make?  Should we be using pymake to build gaia?  Or possibly do we have 2 makes (not counting pymake) in mozilla-build and we're using the wrong one?
this bug should be fixed on bug 1035722 since we will rewrite copy behavious in javascript.
Depends on: 1035722
Depends on: 1029385
No longer depends on: 1035722
Inactive; closing (see bug 1180138).
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.