Last Comment Bug 1195442 - OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/objdir-tb'
: OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-d-0...
Status: RESOLVED FIXED
:
Product: Release Engineering
Classification: Other
Component: General Automation (show other bugs)
: unspecified
: Unspecified Unspecified
-- blocker (vote)
: ---
Assigned To: Philipp Kewisch [:Fallen]
: Chris AtLee [:catlee]
:
Mentors:
: 1197594 (view as bug list)
Depends on: 1035099
Blocks: 1214403 1204381
  Show dependency treegraph
 
Reported: 2015-08-17 11:35 PDT by aleth [:aleth]
Modified: 2015-12-16 05:52 PST (History)
13 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
WiP - v1 (4.50 KB, patch)
2015-08-25 11:52 PDT, Philipp Kewisch [:Fallen]
no flags Details | Diff | Splinter Review
Fix it in buildbotcustom - v2 (3.14 KB, patch)
2015-09-06 03:24 PDT, Philipp Kewisch [:Fallen]
rail: review+
philipp: checked‑in-
Details | Diff | Splinter Review
Fix it in buildbotcustom - v3 (3.24 KB, patch)
2015-09-13 14:42 PDT, Philipp Kewisch [:Fallen]
rail: review+
Details | Diff | Splinter Review
Fix it in buildbotcustom - v4 (4.57 KB, patch)
2015-10-13 13:18 PDT, Philipp Kewisch [:Fallen]
rail: review+
jlund: checked‑in+
Details | Diff | Splinter Review

Description User image aleth [:aleth] 2015-08-17 11:35:24 PDT
mozpack.errors.ErrorMessage: Error: Error while running startup cache precompilation
make[2]: *** [stage-package] Error 1

Looks like xpcshell is failing for some reason.
Local "mach package" works fine.
Comment 1 User image aleth [:aleth] 2015-08-17 11:42:40 PDT
Full log e.g. https://treeherder.mozilla.org/logviewer.html#?job_id=22754&repo=comm-central

Executing /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/i386/../x86_64/dist/bin/xpcshell -g /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/x86_64/dist/Daily.app/Contents/Resources -a /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/x86_64/dist/Daily.app/Contents/Resources -f /builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/toolkit/mozapps/installer/precompile_cache.js -e precompile_startupcache("resource://gre/");
resource://gre/components/ActivityMessageConfigurator.js
resource://gre/components/ActivityProxy.js
resource://gre/components/ActivityRequestHandler.js
resource://gre/components/ActivityWrapper.js
resource://gre/components/AlarmsManager.js
resource://gre/components/AppsService.js
resource://gre/components/BrowserElementParent.js
resource://gre/components/CSSUnprefixingService.js
resource://gre/components/ChromeNotifications.js
resource://gre/components/ColorAnalyzer.js
resource://gre/components/ConsoleAPIStorage.js
resource://gre/components/ContactManager.js
resource://gre/components/ContentProcessSingleton.js
resource://gre/components/CrashService.js
resource://gre/components/DOMSecureElement.js
resource://gre/components/DataReportingService.js
resource://gre/components/DataStoreImpl.js
resource://gre/components/DownloadLegacy.js
resource://gre/components/DownloadsStartup.js
resource://gre/components/FeedProcessor.js
resource://gre/components/FormAutofillContentService.js
resource://gre/components/FormAutofillStartup.js
resource://gre/components/FormHistoryStartup.js
resource://gre/components/InterAppCommService.js
resource://gre/components/InterAppConnection.js
resource://gre/components/InterAppMessagePort.js
resource://gre/components/MainProcessSingleton.js
resource://gre/components/MozKeyboard.js
resource://gre/components/NetworkGeolocationProvider.js
resource://gre/components/NotificationStorage.js
resource://gre/components/PACGenerator.js
resource://gre/components/PageThumbsProtocol.js
resource://gre/components/PeerConnection.js
resource://gre/components/PermissionPromptService.js
resource://gre/components/PermissionSettings.js
resource://gre/components/PhoneNumberService.js
resource://gre/components/PlacesCategoriesStarter.js
resource://gre/components/PresentationDeviceInfoManager.js
resource://gre/components/PrivateBrowsingTrackingProtectionWhitelist.js
resource://gre/components/Push.js
resource://gre/components/PushClient.js
resource://gre/components/PushNotificationService.js
resource://gre/components/RequestSyncManager.js
resource://gre/components/RequestSyncScheduler.js
resource://gre/components/ResourceStatsManager.js
resource://gre/components/SettingsManager.js
resource://gre/components/SiteSpecificUserAgent.js
resource://gre/components/SlowScriptDebug.js
resource://gre/components/SystemMessageCache.js
resource://gre/components/SystemMessageInternal.js
resource://gre/components/SystemMessageManager.js
resource://gre/components/SystemUpdateManager.js
resource://gre/components/TCPPresentationServer.js
resource://gre/components/TCPServerSocket.js
resource://gre/components/TCPSocket.js
resource://gre/components/TCPSocketParentIntermediary.js
resource://gre/components/TelemetryStartup.js
resource://gre/components/TestInterfaceJS.js
resource://gre/components/TestInterfaceJSMaplike.js
resource://gre/components/TestStartupCacheTelemetry.js
resource://gre/components/UnifiedComplete.js
resource://gre/components/WebVTTParserWrapper.js
resource://gre/components/Webapps.js
resource://gre/components/XULStore.js
resource://gre/components/aboutRedirector.js
resource://gre/components/addonManager.js
resource://gre/components/amContentHandler.js
resource://gre/components/amInstallTrigger.js
resource://gre/components/amWebInstallListener.js
resource://gre/components/captivedetect.js
resource://gre/components/contentAreaDropListener.js
resource://gre/components/crypto-SDR.js
resource://gre/components/defaultShims.js
resource://gre/components/facebook.js
resource://gre/components/folderLookupService.js
resource://gre/components/glautocomp.js
resource://gre/components/gtalk.js
resource://gre/components/htmlMenuBuilder.js
resource://gre/components/httpd.js
resource://gre/components/imAccounts.js
resource://gre/components/imCommands.js
resource://gre/components/imContacts.js
resource://gre/components/imConversations.js
resource://gre/components/imCore.js
resource://gre/components/imIncomingServer.js
resource://gre/components/imProtocolInfo.js
resource://gre/components/irc.js
resource://gre/components/jsconsole-clhandler.js
resource://gre/components/jsmimeemitter.js
resource://gre/components/logger.js
resource://gre/components/mailContentHandler.js
resource://gre/components/mailGlue.js
resource://gre/components/marionettecomponent.js
resource://gre/components/mdn-service.js
resource://gre/components/messageWakeupService.js
resource://gre/components/mimeJSComponents.js
resource://gre/components/msgAsyncPrompter.js
resource://gre/components/msgOAuth2Module.js
resource://gre/components/multiprocessShims.js
resource://gre/components/newMailNotificationService.js
resource://gre/components/newsblog.js
resource://gre/components/nsAbAutoCompleteMyDomain.js
resource://gre/components/nsAbAutoCompleteSearch.js
resource://gre/components/nsAbLDAPAttributeMap.js
resource://gre/components/nsAbLDAPAutoCompleteSearch.js
resource://gre/components/nsActivity.js
resource://gre/components/nsActivityManager.js
resource://gre/components/nsActivityManagerUI.js
resource://gre/components/nsAsyncShutdown.js
resource://gre/components/nsBlocklistService.js
resource://gre/components/nsBlocklistServiceContent.js
resource://gre/components/nsBox.js
resource://gre/components/nsContentDispatchChooser.js
resource://gre/components/nsContentPrefService.js
resource://gre/components/nsCrashMonitor.js
resource://gre/components/nsDefaultCLH.js
resource://gre/components/nsDownloadManagerUI.js
resource://gre/components/nsFormAutoComplete.js
resource://gre/components/nsFormHistory.js
resource://gre/components/nsHandlerService.js
resource://gre/components/nsHelperAppDlg.js
resource://gre/components/nsHightail.js
resource://gre/components/nsINIProcessor.js
resource://gre/components/nsInputListAutoComplete.js
resource://gre/components/nsLDAPProtocolHandler.js
resource://gre/components/nsLivemarkService.js
resource://gre/components/nsLoginInfo.js
resource://gre/components/nsLoginManager.js
resource://gre/components/nsLoginManagerPrompter.js
resource://gre/components/nsMailDefaultHandler.js
resource://gre/components/nsMailNewsCommandLineHandler.js
resource://gre/components/nsMsgTraitService.js
resource://gre/components/nsNewsAutoCompleteSearch.js
resource://gre/components/nsPlacesAutoComplete.js
resource://gre/components/nsPlacesExpiration.js
resource://gre/components/nsPrompter.js
resource://gre/components/nsSMTPProtocolHandler.js
resource://gre/components/nsSearchService.js
resource://gre/components/nsSearchSuggestions.js
resource://gre/components/nsSetDefaultMail.js
resource://gre/components/nsTaggingService.js
resource://gre/components/nsTerminatorTelemetry.js
resource://gre/components/nsURLFormatter.js
resource://gre/components/nsUpdateService.js
resource://gre/components/nsUpdateServiceStub.js
resource://gre/components/nsUpdateTimerManager.js
resource://gre/components/nsUrlClassifierHashCompleter.js
resource://gre/components/nsUrlClassifierLib.js
resource://gre/components/nsUrlClassifierListManager.js
resource://gre/components/nsWebHandlerApp.js
resource://gre/components/odnoklassniki.js
resource://gre/components/offlineStartup.js
resource://gre/components/recording-cmdline.js
resource://gre/components/simpleServices.js
resource://gre/components/skype.js
resource://gre/components/smileProtocolHandler.js
resource://gre/components/smime-service.js
resource://gre/components/steelApplication.js
resource://gre/components/storage-json.js
resource://gre/components/twitter.js
resource://gre/components/txEXSLTRegExFunctions.js
resource://gre/components/xmpp.js
resource://gre/components/yahoo.js
resource://gre/modules/AboutReader.jsm
resource://gre/modules/ActivitiesService.jsm
resource://gre/modules/ActivitiesServiceFilter.jsm
resource://gre/modules/AddonManager.jsm
resource://gre/modules/AddonWatcher.jsm
resource://gre/modules/AlarmDB.jsm
resource://gre/modules/AlarmService.jsm
AlarmService: init()
AlarmService: _restoreAlarmsFromDb()
resource://gre/modules/AppConstants.jsm
resource://gre/modules/AppDownloadManager.jsm
resource://gre/modules/AppsServiceChild.jsm
resource://gre/modules/AppsUtils.jsm
resource://gre/modules/ArrayBufferUtils.jsm
resource://gre/modules/AsyncShutdown.jsm
resource://gre/modules/AsyncSpellCheckTestHelper.jsm
resource://gre/modules/AutoCompleteE10S.jsm
resource://gre/modules/BackgroundPageThumbs.jsm
resource://gre/modules/Battery.jsm
resource://gre/modules/BigInteger.jsm
resource://gre/modules/BinarySearch.jsm
resource://gre/modules/BookmarkHTMLUtils.jsm
resource://gre/modules/BookmarkJSONUtils.jsm
resource://gre/modules/Bookmarks.jsm
resource://gre/modules/BrowserElementPromptService.jsm
resource://gre/modules/BrowserUtils.jsm
resource://gre/modules/CertUtils.jsm
resource://gre/modules/CharsetMenu.jsm
resource://gre/modules/ChromeManifestParser.jsm
resource://gre/modules/ClientID.jsm
resource://gre/modules/ClusterLib.js
resource://gre/modules/ColorAnalyzer_worker.js
resource://gre/modules/ColorConversion.js
resource://gre/modules/CommonDialog.jsm
resource://gre/modules/CompatWarning.jsm
resource://gre/modules/ContactDB.jsm
resource://gre/modules/ContactService.jsm
resource://gre/modules/ContentPrefInstance.jsm
resource://gre/modules/ContentPrefService2.jsm
resource://gre/modules/ContentPrefServiceChild.jsm
resource://gre/modules/ContentPrefServiceParent.jsm
resource://gre/modules/ContentPrefStore.jsm
resource://gre/modules/ContentPrefUtils.jsm
resource://gre/modules/CrashManager.jsm
resource://gre/modules/CrashMonitor.jsm
resource://gre/modules/CrashReports.jsm
resource://gre/modules/CrashSubmit.jsm
resource://gre/modules/CrashTestUtils.jsm
Traceback (most recent call last):
File "/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/toolkit/mozapps/installer/packager.py", line 407, in <module>
main()
File "/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/toolkit/mozapps/installer/packager.py", line 401, in main
args.source, gre_path, base)
File "/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/toolkit/mozapps/installer/packager.py", line 159, in precompile_cache
errors.fatal('Error while running startup cache precompilation')
File "/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/python/mozbuild/mozpack/errors.py", line 103, in fatal
self._handle(self.FATAL, msg)
File "/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/python/mozbuild/mozpack/errors.py", line 98, in _handle
raise ErrorMessage(msg)
mozpack.errors.ErrorMessage: Error: Error while running startup cache precompilation
make[2]: *** [stage-package] Error 1
make[1]: *** [postflight_all] Error 2
make: *** [build] Error 2
Comment 2 User image Treeherder Robot 2015-08-17 12:00:12 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22754
repository: comm-central
start_time: 2015-08-15T14:51:51
who: aleth[at]instantbird[dot]org
machine: bld-lion-r5-062
buildname: TB OS X 10.7 comm-central build
revision: ab6c50b76ed3

CalledProcessError: Command '['hg', 'log', '-r', u'ab6c50b76ed3909054735301c7ea32c9f5f753da', '--template', '{node}']' returned non-zero exit status 255
make[2]: *** [stage-package] Error 1
make[1]: *** [postflight_all] Error 2
make: *** [build] Error 2
Comment 3 User image Treeherder Robot 2015-08-17 12:00:13 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22758
repository: comm-central
start_time: 2015-08-16T03:02:42
who: aleth[at]instantbird[dot]org
machine: bld-lion-r5-088
buildname: TB OS X 10.7 comm-central nightly
revision: ab6c50b76ed3

OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-ntly-000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 4 User image Treeherder Robot 2015-08-17 12:00:14 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22753
repository: comm-central
start_time: 2015-08-15T14:51:51
who: aleth[at]instantbird[dot]org
machine: bld-lion-r5-080
buildname: TB OS X 10.7 64-bit comm-central leak test build
revision: ab6c50b76ed3

CalledProcessError: Command '['hg', 'log', '-r', u'ab6c50b76ed3909054735301c7ea32c9f5f753da', '--template', '{node}']' returned non-zero exit status 255
"%s: 0x%p FATAL ERROR\n", logname, lr, 0 );
OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 5 User image Treeherder Robot 2015-08-17 12:00:15 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22740
repository: comm-central
start_time: 2015-08-14T12:27:59
who: aleth[at]instantbird[dot]org
machine: bld-lion-r5-006
buildname: TB OS X 10.7 comm-central build
revision: 5502fceddab4

CalledProcessError: Command '['hg', 'log', '-r', u'5502fceddab4fa9d77889a19071b19b33f3ede8f', '--template', '{node}']' returned non-zero exit status 255
OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 6 User image Treeherder Robot 2015-08-17 12:00:16 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22763
repository: comm-central
start_time: 2015-08-17T03:02:29
who: aleth[at]instantbird[dot]org
machine: bld-lion-r5-062
buildname: TB OS X 10.7 comm-central nightly
revision: ab6c50b76ed3

make[2]: *** [stage-package] Error 1
make[1]: *** [postflight_all] Error 2
make: *** [build] Error 2
Comment 7 User image Treeherder Robot 2015-08-17 12:00:17 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22745
repository: comm-central
start_time: 2015-08-15T03:02:19
who: aleth[at]instantbird[dot]org
machine: bld-lion-r5-066
buildname: TB OS X 10.7 comm-central nightly
revision: 5502fceddab4

OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-ntly-000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 8 User image Treeherder Robot 2015-08-17 12:00:19 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22734
repository: comm-central
start_time: 2015-08-14T12:30:12
who: aleth[at]instantbird[dot]org
machine: bld-lion-r5-082
buildname: TB OS X 10.7 64-bit comm-central leak test build
revision: 5502fceddab4

"%s: 0x%p FATAL ERROR\n", logname, lr, 0 );
OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 9 User image aleth [:aleth] 2015-08-20 12:23:36 PDT
For debug builds, this step succeeds (albeit with some warnings), but then it fails a bit later on upload:

2015-08-19 15:39:36,892 - Copying DailyDebug.app.tar.gz to cache /builds/slave/tb-c-cen-m64-d-000000000000000/signing_cache/dmg/03ac8015f98a8cf1dc37de7076a6eedc342697f7
Traceback (most recent call last):
File "/tools/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/tools/python/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/python/mozbuild/mozbuild/action/make_dmg.py", line 37, in <module>
sys.exit(main(sys.argv[1:]))
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/python/mozbuild/mozbuild/action/make_dmg.py", line 32, in main
make_dmg(args[0], args[1])
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/python/mozbuild/mozbuild/action/make_dmg.py", line 15, in make_dmg
build = MozbuildObject.from_environment()
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/python/mozbuild/mozbuild/base.py", line 184, in from_environment
if not samepath(topobjdir, _config_topobjdir):
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/python/mozbuild/mozbuild/base.py", line 46, in samepath
return os.path.samefile(path1, path2)
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/objdir-tb/_virtualenv/lib/python2.7/posixpath.py", line 154, in samefile
s2 = os.stat(f2)
OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 10 User image Philipp Kewisch [:Fallen] 2015-08-23 05:01:25 PDT
This may be two different issues. For the debug failure, it is a regression after bug 1190522. From what I've been able to find out, I think it may have to do with how MozbuildObject.from_environment() detects the topsrcdir.

.../build/mozilla/python/mozbuild/mozbuild/action/make_dmg.py is the path that calls this function. The code at [1] breaks when it finds $dir/python/mozbuild/mozbuild/base.py in the ancestors of make_dmg.py's directory, which happens when it reaches .../build/mozilla. I didn't follow all paths here, but it's possible that resolve_mozconfig_topobjdir then just appends objdir-tb to the topsrcdir, hence we get .../build/mozilla/objdir-tb.

Not quite sure how we can fix this, I guess we need some better way to find the topsrcdir than just checking for a file. 


[1] http://mxr.mozilla.org/comm-central/source/mozilla/python/mozbuild/mozbuild/base.py#147
Comment 11 User image Philipp Kewisch [:Fallen] 2015-08-23 05:39:49 PDT
Looks like the "Error while running startup cache precompilation" error is gone, just what was happening on the debug builds now.
Comment 12 User image Philipp Kewisch [:Fallen] 2015-08-23 05:48:01 PDT
*** Bug 1197594 has been marked as a duplicate of this bug. ***
Comment 13 User image Philipp Kewisch [:Fallen] 2015-08-23 06:10:05 PDT
Ted, is there any other way that make_dmg.py could obtain a MozbuildObject? Or alternatively, do you have a good idea on how we could fix MozbuildObject.from_environment() so that it detects the right topsrcdir? We could certainly add a hack that checks for a c-c file in the parent directory, but obviously something less hacky would be great. See comment 10 for details.
Comment 14 User image Philipp Kewisch [:Fallen] 2015-08-25 11:52:27 PDT
Created attachment 8652481 [details] [diff] [review]
WiP - v1

This is what I am currently thinking. I didn't use a comm file as a marker to keep it more general, but I can add a check for, say, ../mailnews/moz.build.
Comment 15 User image Philipp Kewisch [:Fallen] 2015-08-25 12:13:05 PDT
Comment on attachment 8652481 [details] [diff] [review]
WiP - v1

ted told me on IRC he is not the right person to ask here. Gregory, can you take a look?
Comment 16 User image Treeherder Robot 2015-08-30 06:36:18 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22991
repository: comm-central
start_time: 2015-08-30T04:03:19
who: mozilla[at]kewis[dot]ch
machine: bld-lion-r5-072
buildname: TB OS X 10.7 64-bit comm-central leak test build
revision: 8fe4ce806f4b

"%s: 0x%p FATAL ERROR\n", logname, lr, 0 );
OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 17 User image Treeherder Robot 2015-08-30 06:36:42 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=22982
repository: comm-central
start_time: 2015-08-30T03:05:52
who: mozilla[at]kewis[dot]ch
machine: bld-lion-r5-060
buildname: TB OS X 10.7 comm-central nightly
revision: 8fe4ce806f4b

OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-ntly-000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 18 User image Treeherder Robot 2015-08-30 11:05:00 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=23003
repository: comm-central
start_time: 2015-08-30T06:30:18
who: mozilla[at]kewis[dot]ch
machine: bld-lion-r5-066
buildname: TB OS X 10.7 comm-central build
revision: 8fe4ce806f4b

OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 19 User image Treeherder Robot 2015-08-30 11:05:15 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=23004
repository: comm-central
start_time: 2015-08-30T06:29:45
who: mozilla[at]kewis[dot]ch
machine: bld-lion-r5-057
buildname: TB OS X 10.7 64-bit comm-central leak test build
revision: 8fe4ce806f4b

"%s: 0x%p FATAL ERROR\n", logname, lr, 0 );
OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 20 User image Gregory Szorc [:gps] (away until 2017-03-20) 2015-08-31 14:19:24 PDT
Comment on attachment 8652481 [details] [diff] [review]
WiP - v1

Review of attachment 8652481 [details] [diff] [review]:
-----------------------------------------------------------------

I'd rather not add this logic back in, as it and the associated complexity was removed in bug 1035099.

Some of the previous comments in this bug are misleading.

The topsrcdir/topobjdir is identified from cwd, not the location of the make_dmg.py file. If we run the make_dmg.py action ( $(call py_action,make_dmg,...) ) from an objdir directory we *should* have no problem identifying topsrcdir and topobjdir because they will be found from the mozinfo.json file.

What I suspect is happening (and I can't prove this because the original log exhibiting this issue no longer exists) is that comm-central's automation is not executing make_dmg.py with pwd underneath the objdir. I think if you change the make invocation to cd first, this error will magically go away.
Comment 21 User image Philipp Kewisch [:Fallen] 2015-09-01 03:49:59 PDT
Great, thanks for the hint! I'll take a look at the code to patch this.
Comment 22 User image Philipp Kewisch [:Fallen] 2015-09-02 17:41:53 PDT
Here is a more recent log (happens on all c-c mac builds)
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/comm-central-macosx64/1441220707/comm-central-macosx64-bm85-build1-build41.txt.gz

Looking at that, the make package call is already executed from the objdir. It may be that the sub-make does something with the path, though I don't think so. I'll take a closer look soon and do a try run to check the actual cwd.
Comment 23 User image aleth [:aleth] 2015-09-03 09:27:27 PDT
(In reply to Philipp Kewisch [:Fallen] from comment #22)
> Looking at that, the make package call is already executed from the objdir.
> It may be that the sub-make does something with the path, though I don't
> think so. I'll take a closer look soon and do a try run to check the actual
> cwd.

cd /builds/slave/tb-c-cen-m64-ntly-000000000000/build/objdir-tb/i386/dist && /builds/slave/tb-c-cen-m64-ntly-000000000000/build/objdir-tb/i386/_virtualenv/bin/python -m mozbuild.action.make_dmg 'universal/thunderbird' 'thunderbird-43.0a1.en-US.mac.dmg'

Looks like it's in the dist subdir of the objdir, actually.
Comment 24 User image Philipp Kewisch [:Fallen] 2015-09-04 05:05:08 PDT
I've been able to reproduce this locally now. Indeed the script is executed from the dist subdir of the objdir, but this will resolve to the real objdir when base.py checks ancestors.

I've checked the mozinfo.json generated in those directories, and it turns out that the topsrcdir defined there is actually .../build/mozilla/ which seems to come from config.status.m4's $srcdir variable. I'm not sure if this is expected to be the mozilla directory or not. It looks like we may have some control over this since we have our own aclocal.m4 which includes config.status.m4.

gps, given the script is already using the right mozinfo.json, do you have a new suggestion on how the right directories can be detected?
Comment 25 User image Gregory Szorc [:gps] (away until 2017-03-20) 2015-09-04 09:55:51 PDT
I defer to glandium on this, as he ripped out the special comm-central code.

I will add that the Python parts of the build system expect topsrcdir to refer to Mozilla's build system, not an app's. Changing mozinfo.json to refer to comm-central's srcdir will likely break things.
Comment 26 User image Philipp Kewisch [:Fallen] 2015-09-04 18:17:11 PDT
(In reply to Gregory Szorc [:gps] from comment #25)
> I will add that the Python parts of the build system expect topsrcdir to
> refer to Mozilla's build system, not an app's. Changing mozinfo.json to
> refer to comm-central's srcdir will likely break things.
I figured as much.

I could imagine "fixing" this by having build automation specify an absolute path for the objdir. I can take another look at this, last time I looked it didn't seem straightforward, but I'm sure it would somehow be possible.
Comment 27 User image aleth [:aleth] 2015-09-05 11:40:36 PDT
It's been mentioned on IRC, but for the record, this bug only occurs for universal builds.
Comment 28 User image aleth [:aleth] 2015-09-05 12:56:05 PDT
(In reply to aleth [:aleth] from comment #27)
> It's been mentioned on IRC, but for the record, this bug only occurs for
> universal builds.

Turns out it's not actually universal builds, but relative (rather than absolute) objdirs that cause the problem, as the relative path is appended to the (for c-c, incorrect) topsrcdir here: http://mxr.mozilla.org/comm-central/source/mozilla/python/mozbuild/mozbuild/base.py#210
Comment 29 User image Philipp Kewisch [:Fallen] 2015-09-06 03:24:30 PDT
Created attachment 8657553 [details] [diff] [review]
Fix it in buildbotcustom - v2

I have no easy way to test this, trying to set up a build master on dev-master2 did not work out for me. Seems the old scripts don't work any longer.

This patch changes buildbotcustom to use absolute objdirs in MOZ_OBJDIR. It should cover both code directly setting MOZ_OBJDIR and code that gets it from the environment. I'm not sure if I missed any other locations where MOZ_OBJDIR is used.

Let me know if I missed something and how I should test this patch.
Comment 30 User image Rail Aliiev [:rail] ⌚️ET 2015-09-08 11:08:32 PDT
Comment on attachment 8657553 [details] [diff] [review]
Fix it in buildbotcustom - v2

Review of attachment 8657553 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +457,5 @@
>                  self.signingServers, self.env.get('PYTHON26'))
>              self.env['MOZ_SIGN_CMD'] = WithProperties(self.signing_command)
>  
> +        # Make sure the objdir is specified with an absolute path
> +        if 'MOZ_OBJDIR' in self.env and not os.path.isabs(self.env['MOZ_OBJDIR']):

I wonder how os.path.isabs() behaves on Windows - we use "/" in paths here... It would be great if we can test this...
Comment 31 User image aleth [:aleth] 2015-09-08 13:10:57 PDT
(In reply to Rail Aliiev [:rail] from comment #30)
> Comment on attachment 8657553 [details] [diff] [review]
> Fix it in buildbotcustom - v2
> 
> Review of attachment 8657553 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: process/factory.py
> @@ +457,5 @@
> >                  self.signingServers, self.env.get('PYTHON26'))
> >              self.env['MOZ_SIGN_CMD'] = WithProperties(self.signing_command)
> >  
> > +        # Make sure the objdir is specified with an absolute path
> > +        if 'MOZ_OBJDIR' in self.env and not os.path.isabs(self.env['MOZ_OBJDIR']):
> 
> I wonder how os.path.isabs() behaves on Windows - we use "/" in paths
> here... It would be great if we can test this...

I don't have a Windows system to test with, but https://docs.python.org/2/library/os.path.html says:
"Returns true if path is an absolute pathname. On Unix, that means it begins with a slash, on Windows that it begins with a (back)slash after chopping off a potential drive letter."
Comment 32 User image Philipp Kewisch [:Fallen] 2015-09-08 14:46:24 PDT
:Aryx just tested this on windows with python 2.7 and os.path.isabs("/foo/bar") returns True.
Comment 33 User image Philipp Kewisch [:Fallen] 2015-09-09 06:05:06 PDT
https://hg.mozilla.org/build/buildbotcustom/rev/fb3167cad8da
Comment 34 User image Release Engineering SlaveAPI Service 2015-09-09 12:21:28 PDT
In production: https://hg.mozilla.org/build/buildbotcustom/rev/fb3167cad8da
Comment 35 User image Philipp Kewisch [:Fallen] 2015-09-09 15:11:30 PDT
Backed out for failures on at least mozilla-beta, comm-central doesn't look good either. Sorry about that!

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/136add2637de
remote:   https://hg.mozilla.org/build/buildbotcustom/rev/6ec74c025338
Comment 36 User image Nick Thomas [:nthomas] 2015-09-09 15:19:13 PDT
Backout deployed by a reconfig.
Comment 37 User image Treeherder Robot 2015-09-10 14:52:26 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=23516
repository: comm-central
start_time: 2015-09-10T11:25:19
who: mozilla[at]kewis[dot]ch
machine: bld-lion-r5-056
buildname: TB OS X 10.7 64-bit comm-central leak test build
revision: e5603a0e2e3c

CalledProcessError: Command '['hg', 'log', '-r', u'e5603a0e2e3c9d51e9e5a22c7354e0d753d75547', '--template', '{node}']' returned non-zero exit status 255
"%s: 0x%p FATAL ERROR\n", logname, lr, 0 );
Assertion failure: IsOnBackgroundThread(), at /builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/ipc/glue/BackgroundImpl.cpp:199
make[3]: *** [stage-package] Error 1
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 38 User image Treeherder Robot 2015-09-10 14:52:29 PDT
log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=23517
repository: comm-central
start_time: 2015-09-10T11:25:19
who: mozilla[at]kewis[dot]ch
machine: bld-lion-r5-084
buildname: TB OS X 10.7 comm-central build
revision: e5603a0e2e3c

CalledProcessError: Command '['hg', 'log', '-r', u'e5603a0e2e3c9d51e9e5a22c7354e0d753d75547', '--template', '{node}']' returned non-zero exit status 255
make[2]: *** [postflight_all] Error 1
make[1]: *** [build] Error 2
make: *** [build] Error 2
Comment 39 User image Philipp Kewisch [:Fallen] 2015-09-13 14:42:06 PDT
Created attachment 8660493 [details] [diff] [review]
Fix it in buildbotcustom - v3

Ok, here is a new version that also uses the basedir from the properties file. I was fooled by the fact that the variable absObjDir sounds like it is already an absolute path, but baseWorkDir is actually just "build". I've tested this patch on a windows and mac slave created on my local machines with the buildbot master on dev-master2. Here is what they set in the environment:

Windows: MOZ_OBJDIR=d:/mozilla/slave/tb-c-cen-w32-00000000000000000/build/objdir-tb
Mac: MOZ_OBJDIR=/builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb

While both builds did not go through until the last step, I think the patch is still correct. The windows build failed on make upload because my machine doesn't have the keys for dev-stage. I ran the remaining steps locally and it seems to work. The mac build failed somewhere during compile due to something with breakpad, I suspect this is due to an older xcode version.

If you think it is necessary, am happy to retry these builds using real loaners. I'll retry the mac build with the newer xcode, maybe I can get some keys for dev-stage tomorrow to do a complete windows build.

I also tried to run a mozilla-central build, but that stopped pretty quickly on the archiver client, I suspect my slave environment is missing something.

Logs for build folks:
http://dev-master2.bb.releng.use1.mozilla.com:8950/builders/TB%20OS%20X%2010.7%20comm-central%20build/builds/11
http://dev-master2.bb.releng.use1.mozilla.com:8950/builders/TB%20WINNT%205.2%20comm-central%20build/builds/8
Comment 40 User image Rail Aliiev [:rail] ⌚️ET 2015-09-14 08:29:14 PDT
Comment on attachment 8660493 [details] [diff] [review]
Fix it in buildbotcustom - v3

Review of attachment 8660493 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +3701,3 @@
>          # Make sure MOZ_PKG_PRETTYNAMES is set so that our source package is
>          # created in the expected place.
> +        self.env['MOZ_OBJDIR'] = WithProperties('%(basedir)s/' + self.absObjDir)

Hmm, prepending an absolute path (absObjDir) with something looks confusing... I won't be picky here, because this builder will be obsolete by bug 1178286.
Comment 41 User image Philipp Kewisch [:Fallen] 2015-09-14 08:44:13 PDT
(In reply to Rail Aliiev [:rail] from comment #40)
> Hmm, prepending an absolute path (absObjDir) with something looks
> confusing... I won't be picky here, because this builder will be obsolete by
> bug 1178286.

Yes, this is exactly what confused me in the first patch. The variable naming is taken from other similar places in that code.

pushed to default: https://hg.mozilla.org/build/buildbotcustom/rev/8c226d1b7a0b
Comment 42 User image Nick Thomas [:nthomas] 2015-09-14 22:16:51 PDT
Backed out at  http://hg.mozilla.org/build/buildbotcustom/rev/7e0879d5a25c for bustage in the source builder for Firefox 38.3.0esr, see http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/38.3.0esr-candidates/build1/logs/release-mozilla-esr38-firefox_source-bm73-build1-build2.txt.gz

Looks like there is an extra build/ in 
   MOZ_OBJDIR=/builds/slave/rel-m-esr38-firefox_source-000/build/mozilla-esr38/obj-firefox
compared to 38.2.1esr.
Comment 43 User image Philipp Kewisch [:Fallen] 2015-10-13 13:18:55 PDT
Created attachment 8673300 [details] [diff] [review]
Fix it in buildbotcustom - v4

Ok, next attempt. This is fix v3, but with a few added lines to fix the source builder. The source builder will now run with both MOZ_OBJDIR and PWD set to e.g. /builds/slave/tb-rel-c-beta-tb_source-000000/build/comm-beta/objdir-tb in the relevant steps. In comparison, this will add the extra /build/ to the path for the objdir for all steps, but this is consistent with the other builders.

I've tested this in staging just for the source builder and a local slave. For those of you that can access the build machines, see
http://dev-master2.bb.releng.use1.mozilla.com:8950/builders/release-comm-beta-thunderbird_source/builds/60

Interdiff valid for a month for convenience: https://diff.pastebin.mozilla.org/8849171
Comment 44 User image Rail Aliiev [:rail] ⌚️ET 2015-10-14 10:12:29 PDT
Comment on attachment 8673300 [details] [diff] [review]
Fix it in buildbotcustom - v4

Let's try again after we ship Firefox 41.0.2 (probably Thu/Fri)
Comment 45 User image Philipp Kewisch [:Fallen] 2015-10-16 14:22:00 PDT
Comment on attachment 8673300 [details] [diff] [review]
Fix it in buildbotcustom - v4

http://hg.mozilla.org/build/buildbotcustom/rev/2c7eb46074b8 (default)
Comment 46 User image aleth [:aleth] 2015-10-27 15:50:05 PDT
Still seeing

mozbuild.base.ObjdirMismatchException: Objdir mismatch: /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/i386 != /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb

Reopen or a new bug?
Comment 47 User image Philipp Kewisch [:Fallen] 2015-10-27 16:43:02 PDT
This should be bug 1214403.
Comment 48 User image aleth [:aleth] 2015-10-27 16:59:30 PDT
OK then.
Comment 49 User image aleth [:aleth] 2015-10-27 17:13:53 PDT
(In reply to aleth [:aleth] from comment #46)
> Still seeing
> 
> mozbuild.base.ObjdirMismatchException: Objdir mismatch:
> /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/i386 !=
> /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb

This happens on make_dmg though, so I don't understand how it's related to the unit tests.
Comment 50 User image Philipp Kewisch [:Fallen] 2015-10-27 17:25:23 PDT
Ah ok, then it may be something different. I don't recall getting this error on make_dmg when last testing it.
Comment 51 User image aleth [:aleth] 2015-10-30 08:10:11 PDT
(In reply to aleth [:aleth] from comment #49)
> (In reply to aleth [:aleth] from comment #46)
> > Still seeing
> > 
> > mozbuild.base.ObjdirMismatchException: Objdir mismatch:
> > /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/i386 !=
> > /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb
> 
> This happens on make_dmg though, so I don't understand how it's related to
> the unit tests.

See Bug 1220018.

Note You need to log in before you can comment on or make changes to this bug.