Closed Bug 1195442 Opened 9 years ago Closed 9 years ago

OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/objdir-tb'

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: aleth, Assigned: Fallen)

References

Details

Attachments

(1 file, 3 obsolete files)

Fix it in buildbotcustom - v4
4.57 KB, patch
rail
: review+
jlund
: checked-in+
Details | Diff | Splinter Review 
mozpack.errors.ErrorMessage: Error: Error while running startup cache precompilation
make[2]: *** [stage-package] Error 1

Looks like xpcshell is failing for some reason.
Local "mach package" works fine.
Full log e.g. https://treeherder.mozilla.org/logviewer.html#?job_id=22754&repo=comm-central

Executing /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/i386/../x86_64/dist/bin/xpcshell -g /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/x86_64/dist/Daily.app/Contents/Resources -a /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/x86_64/dist/Daily.app/Contents/Resources -f /builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/toolkit/mozapps/installer/precompile_cache.js -e precompile_startupcache("resource://gre/");
resource://gre/components/ActivityMessageConfigurator.js
resource://gre/components/ActivityProxy.js
resource://gre/components/ActivityRequestHandler.js
resource://gre/components/ActivityWrapper.js
resource://gre/components/AlarmsManager.js
resource://gre/components/AppsService.js
resource://gre/components/BrowserElementParent.js
resource://gre/components/CSSUnprefixingService.js
resource://gre/components/ChromeNotifications.js
resource://gre/components/ColorAnalyzer.js
resource://gre/components/ConsoleAPIStorage.js
resource://gre/components/ContactManager.js
resource://gre/components/ContentProcessSingleton.js
resource://gre/components/CrashService.js
resource://gre/components/DOMSecureElement.js
resource://gre/components/DataReportingService.js
resource://gre/components/DataStoreImpl.js
resource://gre/components/DownloadLegacy.js
resource://gre/components/DownloadsStartup.js
resource://gre/components/FeedProcessor.js
resource://gre/components/FormAutofillContentService.js
resource://gre/components/FormAutofillStartup.js
resource://gre/components/FormHistoryStartup.js
resource://gre/components/InterAppCommService.js
resource://gre/components/InterAppConnection.js
resource://gre/components/InterAppMessagePort.js
resource://gre/components/MainProcessSingleton.js
resource://gre/components/MozKeyboard.js
resource://gre/components/NetworkGeolocationProvider.js
resource://gre/components/NotificationStorage.js
resource://gre/components/PACGenerator.js
resource://gre/components/PageThumbsProtocol.js
resource://gre/components/PeerConnection.js
resource://gre/components/PermissionPromptService.js
resource://gre/components/PermissionSettings.js
resource://gre/components/PhoneNumberService.js
resource://gre/components/PlacesCategoriesStarter.js
resource://gre/components/PresentationDeviceInfoManager.js
resource://gre/components/PrivateBrowsingTrackingProtectionWhitelist.js
resource://gre/components/Push.js
resource://gre/components/PushClient.js
resource://gre/components/PushNotificationService.js
resource://gre/components/RequestSyncManager.js
resource://gre/components/RequestSyncScheduler.js
resource://gre/components/ResourceStatsManager.js
resource://gre/components/SettingsManager.js
resource://gre/components/SiteSpecificUserAgent.js
resource://gre/components/SlowScriptDebug.js
resource://gre/components/SystemMessageCache.js
resource://gre/components/SystemMessageInternal.js
resource://gre/components/SystemMessageManager.js
resource://gre/components/SystemUpdateManager.js
resource://gre/components/TCPPresentationServer.js
resource://gre/components/TCPServerSocket.js
resource://gre/components/TCPSocket.js
resource://gre/components/TCPSocketParentIntermediary.js
resource://gre/components/TelemetryStartup.js
resource://gre/components/TestInterfaceJS.js
resource://gre/components/TestInterfaceJSMaplike.js
resource://gre/components/TestStartupCacheTelemetry.js
resource://gre/components/UnifiedComplete.js
resource://gre/components/WebVTTParserWrapper.js
resource://gre/components/Webapps.js
resource://gre/components/XULStore.js
resource://gre/components/aboutRedirector.js
resource://gre/components/addonManager.js
resource://gre/components/amContentHandler.js
resource://gre/components/amInstallTrigger.js
resource://gre/components/amWebInstallListener.js
resource://gre/components/captivedetect.js
resource://gre/components/contentAreaDropListener.js
resource://gre/components/crypto-SDR.js
resource://gre/components/defaultShims.js
resource://gre/components/facebook.js
resource://gre/components/folderLookupService.js
resource://gre/components/glautocomp.js
resource://gre/components/gtalk.js
resource://gre/components/htmlMenuBuilder.js
resource://gre/components/httpd.js
resource://gre/components/imAccounts.js
resource://gre/components/imCommands.js
resource://gre/components/imContacts.js
resource://gre/components/imConversations.js
resource://gre/components/imCore.js
resource://gre/components/imIncomingServer.js
resource://gre/components/imProtocolInfo.js
resource://gre/components/irc.js
resource://gre/components/jsconsole-clhandler.js
resource://gre/components/jsmimeemitter.js
resource://gre/components/logger.js
resource://gre/components/mailContentHandler.js
resource://gre/components/mailGlue.js
resource://gre/components/marionettecomponent.js
resource://gre/components/mdn-service.js
resource://gre/components/messageWakeupService.js
resource://gre/components/mimeJSComponents.js
resource://gre/components/msgAsyncPrompter.js
resource://gre/components/msgOAuth2Module.js
resource://gre/components/multiprocessShims.js
resource://gre/components/newMailNotificationService.js
resource://gre/components/newsblog.js
resource://gre/components/nsAbAutoCompleteMyDomain.js
resource://gre/components/nsAbAutoCompleteSearch.js
resource://gre/components/nsAbLDAPAttributeMap.js
resource://gre/components/nsAbLDAPAutoCompleteSearch.js
resource://gre/components/nsActivity.js
resource://gre/components/nsActivityManager.js
resource://gre/components/nsActivityManagerUI.js
resource://gre/components/nsAsyncShutdown.js
resource://gre/components/nsBlocklistService.js
resource://gre/components/nsBlocklistServiceContent.js
resource://gre/components/nsBox.js
resource://gre/components/nsContentDispatchChooser.js
resource://gre/components/nsContentPrefService.js
resource://gre/components/nsCrashMonitor.js
resource://gre/components/nsDefaultCLH.js
resource://gre/components/nsDownloadManagerUI.js
resource://gre/components/nsFormAutoComplete.js
resource://gre/components/nsFormHistory.js
resource://gre/components/nsHandlerService.js
resource://gre/components/nsHelperAppDlg.js
resource://gre/components/nsHightail.js
resource://gre/components/nsINIProcessor.js
resource://gre/components/nsInputListAutoComplete.js
resource://gre/components/nsLDAPProtocolHandler.js
resource://gre/components/nsLivemarkService.js
resource://gre/components/nsLoginInfo.js
resource://gre/components/nsLoginManager.js
resource://gre/components/nsLoginManagerPrompter.js
resource://gre/components/nsMailDefaultHandler.js
resource://gre/components/nsMailNewsCommandLineHandler.js
resource://gre/components/nsMsgTraitService.js
resource://gre/components/nsNewsAutoCompleteSearch.js
resource://gre/components/nsPlacesAutoComplete.js
resource://gre/components/nsPlacesExpiration.js
resource://gre/components/nsPrompter.js
resource://gre/components/nsSMTPProtocolHandler.js
resource://gre/components/nsSearchService.js
resource://gre/components/nsSearchSuggestions.js
resource://gre/components/nsSetDefaultMail.js
resource://gre/components/nsTaggingService.js
resource://gre/components/nsTerminatorTelemetry.js
resource://gre/components/nsURLFormatter.js
resource://gre/components/nsUpdateService.js
resource://gre/components/nsUpdateServiceStub.js
resource://gre/components/nsUpdateTimerManager.js
resource://gre/components/nsUrlClassifierHashCompleter.js
resource://gre/components/nsUrlClassifierLib.js
resource://gre/components/nsUrlClassifierListManager.js
resource://gre/components/nsWebHandlerApp.js
resource://gre/components/odnoklassniki.js
resource://gre/components/offlineStartup.js
resource://gre/components/recording-cmdline.js
resource://gre/components/simpleServices.js
resource://gre/components/skype.js
resource://gre/components/smileProtocolHandler.js
resource://gre/components/smime-service.js
resource://gre/components/steelApplication.js
resource://gre/components/storage-json.js
resource://gre/components/twitter.js
resource://gre/components/txEXSLTRegExFunctions.js
resource://gre/components/xmpp.js
resource://gre/components/yahoo.js
resource://gre/modules/AboutReader.jsm
resource://gre/modules/ActivitiesService.jsm
resource://gre/modules/ActivitiesServiceFilter.jsm
resource://gre/modules/AddonManager.jsm
resource://gre/modules/AddonWatcher.jsm
resource://gre/modules/AlarmDB.jsm
resource://gre/modules/AlarmService.jsm
AlarmService: init()
AlarmService: _restoreAlarmsFromDb()
resource://gre/modules/AppConstants.jsm
resource://gre/modules/AppDownloadManager.jsm
resource://gre/modules/AppsServiceChild.jsm
resource://gre/modules/AppsUtils.jsm
resource://gre/modules/ArrayBufferUtils.jsm
resource://gre/modules/AsyncShutdown.jsm
resource://gre/modules/AsyncSpellCheckTestHelper.jsm
resource://gre/modules/AutoCompleteE10S.jsm
resource://gre/modules/BackgroundPageThumbs.jsm
resource://gre/modules/Battery.jsm
resource://gre/modules/BigInteger.jsm
resource://gre/modules/BinarySearch.jsm
resource://gre/modules/BookmarkHTMLUtils.jsm
resource://gre/modules/BookmarkJSONUtils.jsm
resource://gre/modules/Bookmarks.jsm
resource://gre/modules/BrowserElementPromptService.jsm
resource://gre/modules/BrowserUtils.jsm
resource://gre/modules/CertUtils.jsm
resource://gre/modules/CharsetMenu.jsm
resource://gre/modules/ChromeManifestParser.jsm
resource://gre/modules/ClientID.jsm
resource://gre/modules/ClusterLib.js
resource://gre/modules/ColorAnalyzer_worker.js
resource://gre/modules/ColorConversion.js
resource://gre/modules/CommonDialog.jsm
resource://gre/modules/CompatWarning.jsm
resource://gre/modules/ContactDB.jsm
resource://gre/modules/ContactService.jsm
resource://gre/modules/ContentPrefInstance.jsm
resource://gre/modules/ContentPrefService2.jsm
resource://gre/modules/ContentPrefServiceChild.jsm
resource://gre/modules/ContentPrefServiceParent.jsm
resource://gre/modules/ContentPrefStore.jsm
resource://gre/modules/ContentPrefUtils.jsm
resource://gre/modules/CrashManager.jsm
resource://gre/modules/CrashMonitor.jsm
resource://gre/modules/CrashReports.jsm
resource://gre/modules/CrashSubmit.jsm
resource://gre/modules/CrashTestUtils.jsm
Traceback (most recent call last):
File "/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/toolkit/mozapps/installer/packager.py", line 407, in <module>
main()
File "/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/toolkit/mozapps/installer/packager.py", line 401, in main
args.source, gre_path, base)
File "/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/toolkit/mozapps/installer/packager.py", line 159, in precompile_cache
errors.fatal('Error while running startup cache precompilation')
File "/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/python/mozbuild/mozpack/errors.py", line 103, in fatal
self._handle(self.FATAL, msg)
File "/builds/slave/tb-c-cen-m64-00000000000000000/build/mozilla/python/mozbuild/mozpack/errors.py", line 98, in _handle
raise ErrorMessage(msg)
mozpack.errors.ErrorMessage: Error: Error while running startup cache precompilation
make[2]: *** [stage-package] Error 1
make[1]: *** [postflight_all] Error 2
make: *** [build] Error 2
For debug builds, this step succeeds (albeit with some warnings), but then it fails a bit later on upload:

2015-08-19 15:39:36,892 - Copying DailyDebug.app.tar.gz to cache /builds/slave/tb-c-cen-m64-d-000000000000000/signing_cache/dmg/03ac8015f98a8cf1dc37de7076a6eedc342697f7
Traceback (most recent call last):
File "/tools/python/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/tools/python/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/python/mozbuild/mozbuild/action/make_dmg.py", line 37, in <module>
sys.exit(main(sys.argv[1:]))
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/python/mozbuild/mozbuild/action/make_dmg.py", line 32, in main
make_dmg(args[0], args[1])
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/python/mozbuild/mozbuild/action/make_dmg.py", line 15, in make_dmg
build = MozbuildObject.from_environment()
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/python/mozbuild/mozbuild/base.py", line 184, in from_environment
if not samepath(topobjdir, _config_topobjdir):
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/python/mozbuild/mozbuild/base.py", line 46, in samepath
return os.path.samefile(path1, path2)
File "/builds/slave/tb-c-cen-m64-d-000000000000000/build/objdir-tb/_virtualenv/lib/python2.7/posixpath.py", line 154, in samefile
s2 = os.stat(f2)
OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/objdir-tb'
make[3]: *** [make-package-internal] Error 255
make[2]: *** [make-package] Error 2
make[1]: *** [default] Error 2
make: *** [package] Error 2
This may be two different issues. For the debug failure, it is a regression after bug 1190522. From what I've been able to find out, I think it may have to do with how MozbuildObject.from_environment() detects the topsrcdir.

.../build/mozilla/python/mozbuild/mozbuild/action/make_dmg.py is the path that calls this function. The code at [1] breaks when it finds $dir/python/mozbuild/mozbuild/base.py in the ancestors of make_dmg.py's directory, which happens when it reaches .../build/mozilla. I didn't follow all paths here, but it's possible that resolve_mozconfig_topobjdir then just appends objdir-tb to the topsrcdir, hence we get .../build/mozilla/objdir-tb.

Not quite sure how we can fix this, I guess we need some better way to find the topsrcdir than just checking for a file. 


[1] http://mxr.mozilla.org/comm-central/source/mozilla/python/mozbuild/mozbuild/base.py#147
Looks like the "Error while running startup cache precompilation" error is gone, just what was happening on the debug builds now.
Summary: "Error while running startup cache precompilation" breaks c-c OS X builds → OSError: [Errno 2] No such file or directory: '/builds/slave/tb-c-cen-m64-d-000000000000000/build/mozilla/objdir-tb'
Ted, is there any other way that make_dmg.py could obtain a MozbuildObject? Or alternatively, do you have a good idea on how we could fix MozbuildObject.from_environment() so that it detects the right topsrcdir? We could certainly add a hack that checks for a c-c file in the parent directory, but obviously something less hacky would be great. See comment 10 for details.
Flags: needinfo?(ted)
Attached patch WiP - v1 (obsolete) — — Splinter Review 
This is what I am currently thinking. I didn't use a comm file as a marker to keep it more general, but I can add a check for, say, ../mailnews/moz.build.
Assignee: nobody → philipp
Status: NEW → ASSIGNED
Comment on attachment 8652481 [details] [diff] [review]
WiP - v1

ted told me on IRC he is not the right person to ask here. Gregory, can you take a look?
Flags: needinfo?(ted)
Attachment #8652481 - Flags: feedback?(gps)
Depends on: 1035099
Comment on attachment 8652481 [details] [diff] [review]
WiP - v1

Review of attachment 8652481 [details] [diff] [review]:
-----------------------------------------------------------------

I'd rather not add this logic back in, as it and the associated complexity was removed in bug 1035099.

Some of the previous comments in this bug are misleading.

The topsrcdir/topobjdir is identified from cwd, not the location of the make_dmg.py file. If we run the make_dmg.py action ( $(call py_action,make_dmg,...) ) from an objdir directory we *should* have no problem identifying topsrcdir and topobjdir because they will be found from the mozinfo.json file.

What I suspect is happening (and I can't prove this because the original log exhibiting this issue no longer exists) is that comm-central's automation is not executing make_dmg.py with pwd underneath the objdir. I think if you change the make invocation to cd first, this error will magically go away.
Attachment #8652481 - Flags: feedback?(gps)
Great, thanks for the hint! I'll take a look at the code to patch this.
Flags: needinfo?(philipp)
Here is a more recent log (happens on all c-c mac builds)
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/comm-central-macosx64/1441220707/comm-central-macosx64-bm85-build1-build41.txt.gz

Looking at that, the make package call is already executed from the objdir. It may be that the sub-make does something with the path, though I don't think so. I'll take a closer look soon and do a try run to check the actual cwd.
(In reply to Philipp Kewisch [:Fallen] from comment #22)
> Looking at that, the make package call is already executed from the objdir.
> It may be that the sub-make does something with the path, though I don't
> think so. I'll take a closer look soon and do a try run to check the actual
> cwd.

cd /builds/slave/tb-c-cen-m64-ntly-000000000000/build/objdir-tb/i386/dist && /builds/slave/tb-c-cen-m64-ntly-000000000000/build/objdir-tb/i386/_virtualenv/bin/python -m mozbuild.action.make_dmg 'universal/thunderbird' 'thunderbird-43.0a1.en-US.mac.dmg'

Looks like it's in the dist subdir of the objdir, actually.
I've been able to reproduce this locally now. Indeed the script is executed from the dist subdir of the objdir, but this will resolve to the real objdir when base.py checks ancestors.

I've checked the mozinfo.json generated in those directories, and it turns out that the topsrcdir defined there is actually .../build/mozilla/ which seems to come from config.status.m4's $srcdir variable. I'm not sure if this is expected to be the mozilla directory or not. It looks like we may have some control over this since we have our own aclocal.m4 which includes config.status.m4.

gps, given the script is already using the right mozinfo.json, do you have a new suggestion on how the right directories can be detected?
Flags: needinfo?(philipp) → needinfo?(gps)
I defer to glandium on this, as he ripped out the special comm-central code.

I will add that the Python parts of the build system expect topsrcdir to refer to Mozilla's build system, not an app's. Changing mozinfo.json to refer to comm-central's srcdir will likely break things.
Flags: needinfo?(gps) → needinfo?(mh+mozilla)
(In reply to Gregory Szorc [:gps] from comment #25)
> I will add that the Python parts of the build system expect topsrcdir to
> refer to Mozilla's build system, not an app's. Changing mozinfo.json to
> refer to comm-central's srcdir will likely break things.
I figured as much.

I could imagine "fixing" this by having build automation specify an absolute path for the objdir. I can take another look at this, last time I looked it didn't seem straightforward, but I'm sure it would somehow be possible.
It's been mentioned on IRC, but for the record, this bug only occurs for universal builds.
(In reply to aleth [:aleth] from comment #27)
> It's been mentioned on IRC, but for the record, this bug only occurs for
> universal builds.

Turns out it's not actually universal builds, but relative (rather than absolute) objdirs that cause the problem, as the relative path is appended to the (for c-c, incorrect) topsrcdir here: http://mxr.mozilla.org/comm-central/source/mozilla/python/mozbuild/mozbuild/base.py#210
Attached patch Fix it in buildbotcustom - v2 (obsolete) — — Splinter Review 
I have no easy way to test this, trying to set up a build master on dev-master2 did not work out for me. Seems the old scripts don't work any longer.

This patch changes buildbotcustom to use absolute objdirs in MOZ_OBJDIR. It should cover both code directly setting MOZ_OBJDIR and code that gets it from the environment. I'm not sure if I missed any other locations where MOZ_OBJDIR is used.

Let me know if I missed something and how I should test this patch.
Attachment #8652481 - Attachment is obsolete: true
Attachment #8657553 - Flags: review?(rail)
Flags: needinfo?(mh+mozilla)
Comment on attachment 8657553 [details] [diff] [review]
Fix it in buildbotcustom - v2

Review of attachment 8657553 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +457,5 @@
>                  self.signingServers, self.env.get('PYTHON26'))
>              self.env['MOZ_SIGN_CMD'] = WithProperties(self.signing_command)
>  
> +        # Make sure the objdir is specified with an absolute path
> +        if 'MOZ_OBJDIR' in self.env and not os.path.isabs(self.env['MOZ_OBJDIR']):

I wonder how os.path.isabs() behaves on Windows - we use "/" in paths here... It would be great if we can test this...
(In reply to Rail Aliiev [:rail] from comment #30)
> Comment on attachment 8657553 [details] [diff] [review]
> Fix it in buildbotcustom - v2
> 
> Review of attachment 8657553 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: process/factory.py
> @@ +457,5 @@
> >                  self.signingServers, self.env.get('PYTHON26'))
> >              self.env['MOZ_SIGN_CMD'] = WithProperties(self.signing_command)
> >  
> > +        # Make sure the objdir is specified with an absolute path
> > +        if 'MOZ_OBJDIR' in self.env and not os.path.isabs(self.env['MOZ_OBJDIR']):
> 
> I wonder how os.path.isabs() behaves on Windows - we use "/" in paths
> here... It would be great if we can test this...

I don't have a Windows system to test with, but https://docs.python.org/2/library/os.path.html says:
"Returns true if path is an absolute pathname. On Unix, that means it begins with a slash, on Windows that it begins with a (back)slash after chopping off a potential drive letter."
:Aryx just tested this on windows with python 2.7 and os.path.isabs("/foo/bar") returns True.
Attachment #8657553 - Flags: review?(rail) → review+
Keywords: checkin-needed
https://hg.mozilla.org/build/buildbotcustom/rev/fb3167cad8da
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Backed out for failures on at least mozilla-beta, comm-central doesn't look good either. Sorry about that!

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/136add2637de
remote:   https://hg.mozilla.org/build/buildbotcustom/rev/6ec74c025338
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Backout deployed by a reconfig.
Attachment #8657553 - Attachment is obsolete: true
Attachment #8657553 - Flags: checked-in-
Attached patch Fix it in buildbotcustom - v3 (obsolete) — — Splinter Review 
Ok, here is a new version that also uses the basedir from the properties file. I was fooled by the fact that the variable absObjDir sounds like it is already an absolute path, but baseWorkDir is actually just "build". I've tested this patch on a windows and mac slave created on my local machines with the buildbot master on dev-master2. Here is what they set in the environment:

Windows: MOZ_OBJDIR=d:/mozilla/slave/tb-c-cen-w32-00000000000000000/build/objdir-tb
Mac: MOZ_OBJDIR=/builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb

While both builds did not go through until the last step, I think the patch is still correct. The windows build failed on make upload because my machine doesn't have the keys for dev-stage. I ran the remaining steps locally and it seems to work. The mac build failed somewhere during compile due to something with breakpad, I suspect this is due to an older xcode version.

If you think it is necessary, am happy to retry these builds using real loaners. I'll retry the mac build with the newer xcode, maybe I can get some keys for dev-stage tomorrow to do a complete windows build.

I also tried to run a mozilla-central build, but that stopped pretty quickly on the archiver client, I suspect my slave environment is missing something.

Logs for build folks:
http://dev-master2.bb.releng.use1.mozilla.com:8950/builders/TB%20OS%20X%2010.7%20comm-central%20build/builds/11
http://dev-master2.bb.releng.use1.mozilla.com:8950/builders/TB%20WINNT%205.2%20comm-central%20build/builds/8
Attachment #8660493 - Flags: review?(rail)
Comment on attachment 8660493 [details] [diff] [review]
Fix it in buildbotcustom - v3

Review of attachment 8660493 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +3701,3 @@
>          # Make sure MOZ_PKG_PRETTYNAMES is set so that our source package is
>          # created in the expected place.
> +        self.env['MOZ_OBJDIR'] = WithProperties('%(basedir)s/' + self.absObjDir)

Hmm, prepending an absolute path (absObjDir) with something looks confusing... I won't be picky here, because this builder will be obsolete by bug 1178286.
Attachment #8660493 - Flags: review?(rail) → review+
(In reply to Rail Aliiev [:rail] from comment #40)
> Hmm, prepending an absolute path (absObjDir) with something looks
> confusing... I won't be picky here, because this builder will be obsolete by
> bug 1178286.

Yes, this is exactly what confused me in the first patch. The variable naming is taken from other similar places in that code.

pushed to default: https://hg.mozilla.org/build/buildbotcustom/rev/8c226d1b7a0b
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Backed out at  http://hg.mozilla.org/build/buildbotcustom/rev/7e0879d5a25c for bustage in the source builder for Firefox 38.3.0esr, see http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/38.3.0esr-candidates/build1/logs/release-mozilla-esr38-firefox_source-bm73-build1-build2.txt.gz

Looks like there is an extra build/ in 
   MOZ_OBJDIR=/builds/slave/rel-m-esr38-firefox_source-000/build/mozilla-esr38/obj-firefox
compared to 38.2.1esr.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Blocks: 1204381
Ok, next attempt. This is fix v3, but with a few added lines to fix the source builder. The source builder will now run with both MOZ_OBJDIR and PWD set to e.g. /builds/slave/tb-rel-c-beta-tb_source-000000/build/comm-beta/objdir-tb in the relevant steps. In comparison, this will add the extra /build/ to the path for the objdir for all steps, but this is consistent with the other builders.

I've tested this in staging just for the source builder and a local slave. For those of you that can access the build machines, see
http://dev-master2.bb.releng.use1.mozilla.com:8950/builders/release-comm-beta-thunderbird_source/builds/60

Interdiff valid for a month for convenience: https://diff.pastebin.mozilla.org/8849171
Attachment #8660493 - Attachment is obsolete: true
Attachment #8673300 - Flags: review?(rail)
Comment on attachment 8673300 [details] [diff] [review]
Fix it in buildbotcustom - v4

Let's try again after we ship Firefox 41.0.2 (probably Thu/Fri)
Attachment #8673300 - Flags: review?(rail) → review+
Attachment #8673300 - Flags: checked-in+
Severity: normal → blocker
Still seeing

mozbuild.base.ObjdirMismatchException: Objdir mismatch: /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/i386 != /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb

Reopen or a new bug?
This should be bug 1214403.
OK then.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
(In reply to aleth [:aleth] from comment #46)
> Still seeing
> 
> mozbuild.base.ObjdirMismatchException: Objdir mismatch:
> /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/i386 !=
> /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb

This happens on make_dmg though, so I don't understand how it's related to the unit tests.
Ah ok, then it may be something different. I don't recall getting this error on make_dmg when last testing it.
(In reply to aleth [:aleth] from comment #49)
> (In reply to aleth [:aleth] from comment #46)
> > Still seeing
> > 
> > mozbuild.base.ObjdirMismatchException: Objdir mismatch:
> > /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb/i386 !=
> > /builds/slave/tb-c-cen-m64-00000000000000000/build/objdir-tb
> 
> This happens on make_dmg though, so I don't understand how it's related to
> the unit tests.

See Bug 1220018.
Blocks: 1214403
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: