917753 - make clean doesn't remove xpidl files

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Landry Breuil (:gaston)]

Since a little while my openbsd clobber builds randomly fail with missing nsISupports.h headers in the path, so that might be a dependency problem ? Usually removing the objdir fixes it but that's not a proper fix. Dunno what's supposed to generate nsISupports.h ?


eg++ -o nsReadableUtils.o -c  -fvisibility=hidden -DMOZ_GLUE_IN_PROGRAM -DMOZILLA_INTERNAL_API -DNO_NSPR_10_SUPPORT -DIMPL_LIBXUL  -I/home/buildslave/mozilla-central-sparc64/build/xpcom/string/src -I. -I../../../dist/include  -I/data/obj/buildslave/m-c/dist/include/nspr -I/data/obj/buildslave/m-c/dist/include/nss      -fPIC  -I/usr/X11R6/include -DMOZILLA_CLIENT -include ../../../mozilla-config.h -MD -MP -MF .deps/nsReadableUtils.o.pp -I/usr/X11R6/include -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -fno-exceptions -fno-strict-aliasing -fno-rtti -fno-exceptions -fno-math-errno -std=gnu++0x -pthread -pipe  -DNDEBUG -DTRIMMED -g -O -fomit-frame-pointer  /home/buildslave/mozilla-central-sparc64/build/xpcom/string/src/nsReadableUtils.cpp
nsString.o
In file included from ../../../dist/include/nsCycleCollectionNoteChild.h:12:0,
                 from ../../../dist/include/nsTArray.h:18,
                 from /home/buildslave/mozilla-central-sparc64/build/xpcom/string/src/nsReadableUtils.cpp:10:
../../../dist/include/nsCycleCollectionTraversalCallback.h:9:25: fatal error: nsISupports.h: No such file or directory
compilation terminated.

This is happening regardless of arch & compiler used.

Comment 1

[Landry Breuil (:gaston)]

http://buildbot.rhaalovely.net/builders/mozilla-central-amd64/builds/874 is an occurence from some days ago.. i think it was one of the first hit.
http://buildbot.rhaalovely.net/builders/mozilla-central-sparc64/builds/554 is another from yesterday.

It also affects c-c, see http://buildbot.rhaalovely.net/builders/comm-central-amd64/builds/855

Sometimes, it's on a missing nsIAtom.h:

eg++ -o nsSubstring.o -c -I../../../dist/stl_wrappers -I../../../dist/system_wrappers -include /home/buildslave/comm-central-i386/build/mozilla/config/gcc_hidden.h -DMOZ_GLUE_IN_PROGRAM -DMOZILLA_INTERNAL_API -DNO_NSPR_10_SUPPORT -DIMPL_LIBXUL  -I/home/buildslave/comm-central-i386/build/mozilla/xpcom/string/src -I. -I../../../dist/include  -I/var/buildslave/comm-central-i386/build/obj-i386-unknown-openbsd5.4/mozilla/dist/include/nspr -I/var/buildslave/comm-central-i386/build/obj-i386-unknown-openbsd5.4/mozilla/dist/include/nss      -fPIC  -I/usr/X11R6/include -DMOZILLA_CLIENT -include ../../../mozilla-config.h -MD -MP -MF .deps/nsSubstring.o.pp -I/usr/X11R6/include -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -Wcast-align -fno-exceptions -fno-strict-aliasing -fno-rtti -fno-exceptions -fno-math-errno -std=gnu++0x -pthread -pipe  -DNDEBUG -DTRIMMED -O -fomit-frame-pointer  /home/buildslave/comm-central-i386/build/mozilla/xpcom/string/src/nsSubstring.cpp
In file included from /home/buildslave/comm-central-i386/build/mozilla/xpcom/string/src/nsSubstring.cpp:25:0:
../../../dist/include/nsStaticAtom.h:9:21: fatal error: nsIAtom.h: No such file or directory
compilation terminated.

(from http://buildbot.rhaalovely.net/builders/comm-central-i386/builds/407)

Comment 2

[Landry Breuil (:gaston)]

http://buildbot.rhaalovely.net/builders/mozilla-central-amd64/builds/880 is also an occurence of a missing nsIAtom.h

Comment 3

[Landry Breuil (:gaston)]

More failures... removed the objdirs this time. The usual process is hg pull -u, gmake -f client.mk, gmake package, gmake clean. In case nsI* are sometimes not generated (missing dep ?)

http://buildbot.rhaalovely.net/builders/mozilla-central-sparc64/builds/556
http://buildbot.rhaalovely.net/builders/mozilla-central-amd64/builds/886
http://buildbot.rhaalovely.net/builders/comm-central-amd64/builds/861

Comment 4

[Landry Breuil (:gaston)]

Sadly, this also affects aurora:
http://buildbot.rhaalovely.net/builders/mozilla-aurora-amd64/builds/871

ccing some hopefully helpful folks for hints on the possible cause of this :)

Comment 5

[Andrew McCreight [:mccr8]]

Maybe Ehsan has a better idea of what might have regressed it.  It sounds like something the recent flurry of work on #include elimination could have broken.

Comment 6

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Sounds more like the exporting work

Comment 7

[(no longer active)]

Yeah this is almost definitely a race condition in the build system.

Comment 8

[(no longer active)]

(In reply to comment #5)
> Maybe Ehsan has a better idea of what might have regressed it.  It sounds like
> something the recent flurry of work on #include elimination could have broken.

Note that could not have possibly happened, since this bug seems to be about the files not being there on the filesystem, it seems.

Comment 9

[Landry Breuil (:gaston)]

(In reply to :Ehsan Akhgari (needinfo? me!) from comment #8)
> (In reply to comment #5)
> > Maybe Ehsan has a better idea of what might have regressed it.  It sounds like
> > something the recent flurry of work on #include elimination could have broken.
> 
> Note that could not have possibly happened, since this bug seems to be about
> the files not being there on the filesystem, it seems.

Yeah, when the failure shows up, the offending file is not in the objdir at all, ie not generated.

Comment 10

[Gregory Szorc [:gps]]

You are seeing this on current mozilla-central? I could understand nsISupports.h being out of date in dist/includes, but for it to not be present at all is, uh, very surprising.

What do you see in objdir/config/makefiles/xpidl?

Comment 11

[Landry Breuil (:gaston)]

My other builders are failing hard atm because of #919111 but for aurora from yesterday:
http://buildbot.rhaalovely.net/builders/mozilla-aurora-amd64/builds/873/steps/build/logs/stdio

This is what i have there:

/usr/obj/buildslave-m-a/config/makefiles/xpidl/Makefile

/usr/obj/buildslave-m-a/config/makefiles/xpidl/xpt:
WorkerTest.xpt                  dom_base.xpt                    dom_svg.xpt                     intl.xpt                        necko_websocket.xpt             update.xpt
accessibility.xpt               dom_camera.xpt                  dom_system.xpt                  jar.xpt                         necko_wyciwyg.xpt               uriloader.xpt
alerts.xpt                      dom_canvas.xpt                  dom_telephony.xpt               jsdebugger.xpt                  parentalcontrols.xpt            url-classifier.xpt
appshell.xpt                    dom_contacts.xpt                dom_time.xpt                    jsdownloads.xpt                 pipboot.xpt                     urlformatter.xpt
appstartup.xpt                  dom_core.xpt                    dom_traversal.xpt               jsdservice.xpt                  pipnss.xpt                      webBrowser_core.xpt
autocomplete.xpt                dom_css.xpt                     dom_webspeechrecognition.xpt    jsinspector.xpt                 pippki.xpt                      webbrowserpersist.xpt
autoconfig.xpt                  dom_devicestorage.xpt           dom_webspeechsynth.xpt          layout_base.xpt                 places.xpt                      widget.xpt
browser-feeds.xpt               dom_events.xpt                  dom_xbl.xpt                     layout_printing.xpt             plugin.xpt                      windowds.xpt
browsercompsbase.xpt            dom_file.xpt                    dom_xpath.xpt                   layout_xul.xpt                  pref.xpt                        windowwatcher.xpt
caps.xpt                        dom_gamepad.xpt                 dom_xul.xpt                     layout_xul_tree.xpt             prefetch.xpt                    xpcom_base.xpt
chrome.xpt                      dom_geolocation.xpt             downloads.xpt                   locale.xpt                      profile.xpt                     xpcom_components.xpt
commandhandler.xpt              dom_html.xpt                    editor.xpt                      loginmgr.xpt                    rdf.xpt                         xpcom_ds.xpt
commandlines.xpt                dom_indexeddb.xpt               embed_base.xpt                  lwbrk.xpt                       satchel.xpt                     xpcom_io.xpt
composer.xpt                    dom_json.xpt                    extensions.xpt                  migration.xpt                   saxparser.xpt                   xpcom_system.xpt
content_base.xpt                dom_media.xpt                   exthandler.xpt                  mimetype.xpt                    services-crypto-component.xpt   xpcom_threads.xpt
content_canvas.xpt              dom_messages.xpt                exthelper.xpt                   mozfind.xpt                     sessionstore.xpt                xpcom_xpti.xpt
content_events.xpt              dom_mobilemessage.xpt           fastfind.xpt                    necko.xpt                       shellservice.xpt                xpcomsample.xpt
content_html.xpt                dom_network.xpt                 feeds.xpt                       necko_about.xpt                 shistory.xpt                    xpconnect.xpt
content_htmldoc.xpt             dom_notification.xpt            filepicker.xpt                  necko_cache.xpt                 spellchecker.xpt                xpctest.xpt
content_xslt.xpt                dom_offline.xpt                 find.xpt                        necko_cookie.xpt                startupcache.xpt                xul.xpt
cookie.xpt                      dom_permissionsettings.xpt      fuel.xpt                        necko_dns.xpt                   storage.xpt                     xulapp.xpt
directory.xpt                   dom_power.xpt                   gfx.xpt                         necko_file.xpt                  telemetry.xpt                   xuldoc.xpt
diskspacewatcher.xpt            dom_quota.xpt                   hal.xpt                         necko_ftp.xpt                   test_necko.xpt                  xultmpl.xpt
docshell.xpt                    dom_range.xpt                   html5.xpt                       necko_http.xpt                  toolkitprofile.xpt              zipwriter.xpt
dom.xpt                         dom_settings.xpt                htmlparser.xpt                  necko_ipc.xpt                   toolkitremote.xpt
dom_activities.xpt              dom_sidebar.xpt                 identity.xpt                    necko_res.xpt                   txmgr.xpt
dom_alarm.xpt                   dom_smil.xpt                    imgicon.xpt                     necko_socket.xpt                txtsvc.xpt
dom_apps.xpt                    dom_storage.xpt                 imglib2.xpt                     necko_strconv.xpt               uconv.xpt
dom_audiochannel.xpt            dom_stylesheets.xpt             inspector.xpt                   necko_viewsource.xpt            unicharutil.xpt


And the list of nsI* headers in dist/include:
[21:18] dawn:/usr/obj/buildslave-m-a/dist/include/ $ls nsI*
nsIAccessibilityService.h@            nsIDTD.h@                             nsIJSNativeInitializer.h@             nsIPrivateTextEvent.h@                nsIStyleRuleProcessor.h@
nsIAllocator.h@                       nsIDateTimeFormat.h@                  nsILanguageAtomService.h@             nsIPrivateTextRange.h@                nsIStyleSheet.h@
nsIAnonymousContentCreator.h@         nsIDeviceContextSpec.h@               nsILayoutDebugger.h@                  nsIRDFContentSink.h@                  nsIStyleSheetLinkingElement.h@
nsIAppStartupNotifier.h@              nsIDocument.h@                        nsILayoutHistoryState.h@              nsIRadioGroupContainer.h@             nsISupportsBase.h@
nsIAttribute.h@                       nsIDocumentInlines.h@                 nsILineBreaker.h@                     nsIRadioVisitor.h@                    nsISupportsImpl.h@
nsIAudioChannelAgent.h                nsIDocumentObserver.h@                nsILineIterator.h@                    nsIReflowCallback.h@                  nsISupportsObsolete.h@
nsICSSDeclaration.h@                  nsIDocumentTransformer.h@             nsILinkHandler.h@                     nsIRollupListener.h@                  nsISupportsUtils.h@
nsICSSLoaderObserver.h@               nsIFileStorage.h@                     nsIListControlFrame.h@                nsISMILAttr.h@                        nsITableCellLayout.h@
nsICSSPseudoComparator.h@             nsIForm.h@                            nsILocalStore.h@                      nsISMILType.h@                        nsITextControlElement.h@
nsICSSRuleList.h@                     nsIFormControl.h@                     nsIMutationObserver.h@                nsIScriptContext.h@                   nsITextControlFrame.h@
nsICSSStyleRuleDOMWrapper.h@          nsIFormControlFrame.h@                nsINIParser.h@                        nsIScriptElement.h@                   nsITextService.h@
nsICachedFileDescriptorListener.h@    nsIFormProcessor.h@                   nsINameSpaceManager.h@                nsIScriptExternalNameSet.h@           nsITextServicesDocument.h@
nsICanvasElementExternal.h@           nsIFragmentContentSink.h@             nsINativeKeyBindings.h@               nsIScriptGlobalObject.h@              nsITheme.h@
nsICanvasRenderingContextInternal.h@  nsIFrame.h@                           nsINode.h@                            nsIScriptGlobalObjectOwner.h@         nsITokenizer.h@
nsICaseConversion.h@                  nsIFrameTraversal.h@                  nsINodeInfo.h@                        nsIScriptNameSpaceManager.h@          nsIUGenCategory.h@
nsICharsetDetectionObserver.h@        nsIFrameUtil.h@                       nsINodeList.h@                        nsIScriptObjectPrincipal.h@           nsIUnicodeDecoder.h@
nsICharsetDetector.h@                 nsIGlobalObject.h@                    nsIOS2Locale.h@                       nsIScriptTimeoutHandler.h@            nsIUnicodeEncoder.h@
nsIClassInfoImpl.h@                   nsIGridPart.h@                        nsIObjectFrame.h@                     nsIScrollPositionListener.h@          nsIWeakReferenceUtils.h@
nsIComboboxControlFrame.h@            nsIHTMLCollection.h@                  nsIOfflineStorage.h@                  nsIScrollableFrame.h@                 nsIWebShellServices.h@
nsIConstraintValidation.h@            nsIHTMLContentSink.h@                 nsIPageSequenceFrame.h@               nsIScrollbarMediator.h@               nsIWidget.h@
nsIContent.h@                         nsIHTMLDocument.h@                    nsIParser.h@                          nsIScrollbarOwner.h@                  nsIWidgetListener.h@
nsIContentIterator.h@                 nsIID.h@                              nsIParserService.h@                   nsISelectControlFrame.h@              nsIWordBreaker.h@
nsIContentSerializer.h@               nsIIPCSerializableInputStream.h@      nsIPercentHeightObserver.h@           nsISizeOf.h@                          nsIXMLContentSink.h@
nsIContentSink.h@                     nsIIPCSerializableURI.h@              nsIPlatformCharset.h@                 nsISpellChecker.h@                    nsIXULDocument.h@
nsID.h@                               nsIImageToPixbuf.h@                   nsIPluginWidget.h@                    nsIStatefulFrame.h@                   nsImageLoadingContent.h@
nsIDOMClassInfo.h@                    nsIInterfaceRequestorUtils.h@         nsIPresShell.h@                       nsIStringCharsetDetector.h@           nsInterfaceHashtable.h@
nsIDOMScriptObjectFactory.h@          nsIJSEventListener.h@                 nsIPrintDialogService.h@              nsIStyleRule.h@                       nsInterfaceRequestorAgg.h@

Comment 12

[Gregory Szorc [:gps]]

I believe there was a bug in the build rules impacting Windows. But that shouldn't have impacted any platform with symlinks.

Are you doing any kind of manual clobber, such as deleting bits of dist/include? We don't have explicit rules to install XPIDL-deleted dist/include/.h files if they are deleted (we assume the build system behaves a certain way, which it should as long as you are building through client.mk). 

Do you have bug 908977 / ba4acc701c7e in your tree? That should have fixed lingering build dependency issues.

Comment 13

[Landry Breuil (:gaston)]

(In reply to Gregory Szorc [:gps] from comment #12)
> I believe there was a bug in the build rules impacting Windows. But that
> shouldn't have impacted any platform with symlinks.
> 
> Are you doing any kind of manual clobber, such as deleting bits of
> dist/include? We don't have explicit rules to install XPIDL-deleted
> dist/include/.h files if they are deleted (we assume the build system
> behaves a certain way, which it should as long as you are building through
> client.mk). 

The buildslave does this every 24h (you can see the steps on http://buildbot.rhaalovely.net/builders/mozilla-aurora-amd64/builds/873):
- hg qpop -a (in case i have patches to fix build issues in my queue)
- hg pull -u
- hg qpush -a
- gmake -f client.mk configure
- gmake -f client.mk
- gmake -C objdir package
- gmake -f client.mk clean

Of course, all the steps are done only if the previous succeeds. That means the clean step is done only if everything went fine, so that i can have a look at the objdir. In all the previous occurences of that bug, manually rm -Rf'ing the objdir 'fixed' the issue for a day or two, then it usually happens again. SO no, i dont manually purge the objdir, it's an all or nothing thing.

> Do you have bug 908977 / ba4acc701c7e in your tree? That should have fixed
> lingering build dependency issues.

The random failures started happening something like 15 days ago, and now happens on aurora & central, so yes i have that changeset.

Comment 14

[Gregory Szorc [:gps]]

|make clean| will blow out dist/. However, it won't delete the .xpt files from config/makefiles/xpidl. As a result, the build system thinks the .h in dist/include are up to date and doesn't reinstall them. We should delete the .xpt files during |make clean|.

Comment 15

[Landry Breuil (:gaston)]

(In reply to Gregory Szorc [:gps] from comment #14)
> |make clean| will blow out dist/. However, it won't delete the .xpt files
> from config/makefiles/xpidl. As a result, the build system thinks the .h in
> dist/include are up to date and doesn't reinstall them. We should delete the
> .xpt files during |make clean|.

so you mean i should rather replace my last step by make clobber ?
what recently changed in that area that could trigger that new failure ?

Comment 16

[Gregory Szorc [:gps]]

(In reply to Landry Breuil (:gaston) from comment #15)
> (In reply to Gregory Szorc [:gps] from comment #14)
> > |make clean| will blow out dist/. However, it won't delete the .xpt files
> > from config/makefiles/xpidl. As a result, the build system thinks the .h in
> > dist/include are up to date and doesn't reinstall them. We should delete the
> > .xpt files during |make clean|.
> 
> so you mean i should rather replace my last step by make clobber ?
> what recently changed in that area that could trigger that new failure ?

I think |make clobber| is busted. Use |mach clobber| or rm -rf the objdir.

We should fix |make clean|. This is fallout from bug 850380.

Comment 17

[Landry Breuil (:gaston)]

Another interesting different one:
http://buildbot.rhaalovely.net/builders/mozilla-central-amd64/builds/891

In file included from /var/buildslave-mozilla/mozilla-central-amd64/build/xpcom/base/AvailableMemoryTracker.cpp:22:
../../dist/include/mozilla/Preferences.h:13:10: fatal error: 'nsIPrefService.h' file not found
#include "nsIPrefService.h"
         ^
1 error generated.
gmake[5]: *** [AvailableMemoryTracker.o] Error 1
gmake[5]: *** Waiting for unfinished jobs....
In file included from /var/buildslave-mozilla/mozilla-central-amd64/build/xpcom/base/CycleCollectedJSRuntime.cpp:60:
In file included from ../../dist/include/mozilla/dom/BindingUtils.h:14:
In file included from ../../dist/include/mozilla/dom/CallbackObject.h:32:
../../dist/include/xpcpublic.h:17:10: fatal error: 'nsIURI.h' file not found
#include "nsIURI.h"

Comment 18

[Landry Breuil (:gaston)]

Funky new one too.. 
http://buildbot.rhaalovely.net/builders/mozilla-central-sparc64/builds/561

In file included from /home/buildslave/mozilla-central-sparc64/build/xpcom/base/AvailableMemoryTracker.cpp:22:0:
../../dist/include/mozilla/Preferences.h:13:28: fatal error: nsIPrefService.h: No such file or directory
compilation terminated.

I'll remove all objdirs and switch to mach clobber on my builders until the issue is fixed.

Comment 19

[Landry Breuil (:gaston)]

This bug still affects my c-c builder where i cant really use mach clobber...

Even doing gmake -f client.mk clean + rm -Rf objdir/dist* after a successful build, the next one fails.

http://buildbot.rhaalovely.net/builders/comm-central-amd64/builds/880

Comment 20

[Gregory Szorc [:gps]]

Attachment: Add XPIDL files to clean target

I hate having to redefine the targets like this, but it's the simplest
patch. At some point we should merge the two xpidl Makefile.in together.

Comment 21

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/19978aab0190

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/19978aab0190