1471685 - xul.dll link fails with "NSModules are not adjacent"

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

While doing a clean build from central commit 9c7bb8874337 my link fails with:

0:11.35 LINK : xul.dll not found or not built by the last incremental link; performing full link
 1:49.95    Creating library xul.lib and object xul.exp
 1:54.81 LINK : warning LNK4199: /DELAYLOAD:msdmo.dll ignored; no imports found from msdmo.dll
 2:45.83 5b07cac 4 __start_kPStaticModules
 2:45.83 5b07db0 4 SandboxSettingsModule_NSModule
 2:45.83 5b07db4 4 nsPrefModule_NSModule
 2:45.83 5b07db8 4 nsUConvModule_NSModule
 2:45.83 5b07dbc 4 nsI18nModule_NSModule
 2:45.83 5b07dc0 4 nsDNSServiceDiscoveryModule_NSModule
 2:45.83 5b07dc4 4 necko_NSModule
 2:45.83 5b07dc8 4 nsAuthModule_NSModule
 2:45.84 5b07dcc 4 nsChardetModule_NSModule
 2:45.84 5b07dd0 4 nsJarModule_NSModule
 2:45.84 5b07dd4 4 ZipWriterModule_NSModule
 2:45.84 5b07dd8 4 mozStorageModule_NSModule
 2:45.84 5b07ddc 4 nsCookieModule_NSModule
 2:45.84 5b07de0 4 nsPermissionsModule_NSModule
 2:45.84 5b07de4 4 nsParserModule_NSModule
 2:45.84 5b07de8 4 nsGfxModule_NSModule
 2:45.84 5b07dec 4 nsImageLib2Module_NSModule
 2:45.84 5b07df0 4 nsIconDecoderModule_NSModule
 2:45.84 5b07df4 4 FontTableURIProtocolHandler_NSModule
 2:45.84 5b07df8 4 BlobURLProtocolHandler_NSModule
 2:45.84 5b07e00 4 fakesynth_NSModule
 2:45.84 5b07e04 4 synthsapi_NSModule
 2:45.84 5b07e08 4 peerconnection_NSModule
 2:45.84 5b07e0c 4 nsPluginModule_NSModule
 2:45.84 5b07e10 4 PaymentRequestModule_NSModule
 2:45.84 5b07e14 4 PresentationDeviceProviderModule_NSModule
 2:45.84 5b07e18 4 nsContentProcessWidgetModule_NSModule
 2:45.84 5b07e1c 4 nsWidgetModule_NSModule
 2:45.84 5b07e20 4 nsTransactionManagerModule_NSModule
 2:45.84 5b07e24 4 nsComposerModule_NSModule
 2:45.84 5b07e28 4 nsLayoutModule_NSModule
 2:45.85 5b07e2c 4 nsLayoutDebugModule_NSModule
 2:45.85 5b07e30 4 docshell_provider_NSModule
 2:45.85 5b07e34 4 appshell_NSModule
 2:45.85 5b07e38 4 nsUniversalCharDetModule_NSModule
 2:45.85 5b07e3c 4 nsProfilerModule_NSModule
 2:45.85 5b07e40 4 mozSpellCheckerModule_NSModule
 2:45.85 5b07e44 4 LocalCertServiceModule_NSModule
 2:45.85 5b07e48 4 NSS_NSModule
 2:45.85 5b07e4c 4 PKI_NSModule
 2:45.85 5b07e50 4 Browser_Embedding_Module_NSModule
 2:45.86 5b07e54 4 CommandLineModule_NSModule
 2:45.86 5b07e58 4 DiskSpaceWatcherModule_NSModule
 2:45.86 5b07e5c 4 nsMediaSnifferModule_NSModule
 2:45.86 5b07e60 4 mozMozIntlHelperModule_NSModule
 2:45.86 5b07e64 4 jsperf_NSModule
 2:45.86 5b07e68 4 nsPlacesModule_NSModule
 2:45.86 5b07e6c 4 jsreflect_NSModule
 2:45.86 5b07e70 4 nsTelemetryModule_NSModule
 2:45.86 5b07e74 4 nsPageThumbsModule_NSModule
 2:45.86 5b07e78 4 jsctypes_NSModule
 2:45.86 5b07e7c 4 tkAutoCompleteModule_NSModule
 2:45.86 5b07e80 4 satchel_NSModule
 2:45.86 5b07e84 4 nsToolkitCompsModule_NSModule
 2:45.86 5b07e88 4 Apprunner_NSModule
 2:45.86 5b07e8c 4 embedcomponents_NSModule
 2:45.86 5b07e90 4 nsWindowsProxyModule_NSModule
 2:45.87 5b07e94 4 nsAutoConfigModule_NSModule
 2:45.87 5b07e98 4 jsinspector_NSModule
 2:45.87 5b07e9c 4 identity_NSModule
 2:45.87 5b07ea0 4 jsdebugger_NSModule
 2:45.87 5b07ea4 4 nsBrowserCompsModule_NSModule
 2:45.87 5b07fd8 4 __stop_kPStaticModules
2:45.87 TEST-UNEXPECTED-FAIL | check_nsmodules | xul.dll | NSModules are not adjacent

This is the first build using that changeset, and previous builds on different changesets with the same mozconfig have worked.

I'm building with VS2017 (15.6.3) on Windows 10. mozconfig:

ac_add_options --enable-warnings-as-errors
ac_add_options --enable-debug
ac_add_options --enable-optimize
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../mozilla-builds/obj-ff-dbg-opt
ac_add_options --enable-eme=widevine
ac_add_options --enable-profiling

Comment 1

[Nathan Froyd [:froydnj]]

As bug 1471665 points out, this only seems to affect --enable-debug builds.

Comment 3

[Jorg K (CEST = GMT+2)]

Log of the console in bug 1470127 in attachment 8988241 [details]. Not seen on Thunderbird server builds, only local debug builds.

Comment 4

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Same config with additional clang config does not have the issue:

export CC="clang-cl.exe"
export CXX="clang-cl.exe"
export LINKER="lld-link.exe"

Let me know if there's anything I can do to assist with diagnosis or otherwise.

Comment 5

[Mike Hommey [:glandium]]

This is puzzling, because Windows build is the thing that changed the least in bug 1470127...

Comment 6

[Nathan Froyd [:froydnj]]

(In reply to Bryce Van Dyk (:bryce) from comment #0)
> I'm building with VS2017 (15.6.3) on Windows 10. mozconfig:

I thought we only permitted the most recent patch release of 15.6 (15.6.6).  Do the problems go away if you upgrade to 15.6.6?

Automation is certainly using the latest patch release, so perhaps that's the key difference?

Comment 7

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #5)
> This is puzzling, because Windows build is the thing that changed the least
> in bug 1470127...

Actually... bug 1470127 didn't change anything, only made checks less permissive. Can you check whether this is caused for you by bug 1470127 or bug 1471132? If the former, then that means your builds were only working by chance in the past.

Comment 8

[Jorg K (CEST = GMT+2)]

(In reply to Nathan Froyd [:froydnj] from comment #6)
> I thought we only permitted the most recent patch release of 15.6 (15.6.6). 
> Do the problems go away if you upgrade to 15.6.6?
Is that possible? I get offered 15.7.4. What will break with that version ;-(

Comment 9

[Nathan Froyd [:froydnj]]

(In reply to Jorg K (GMT+2) from comment #8)
> (In reply to Nathan Froyd [:froydnj] from comment #6)
> > I thought we only permitted the most recent patch release of 15.6 (15.6.6). 
> > Do the problems go away if you upgrade to 15.6.6?
> Is that possible? I get offered 15.7.4. What will break with that version ;-(

You have to manually install 15.6.6; I can't recall where the installer is offhand.  15.7+ should be rejected by configure for breaking everything.

Comment 10

[Xidorn Quan [:xidorn] UTC+10]

I'm using VS 15.6.7, and seeing the same issue as well.

Comment 11

[Xidorn Quan [:xidorn] UTC+10]

I tried comment 7, and it seems backing out bug 1471132 doesn't fix this issue, neither does backing out bug 1470127.

Comment 12

[Mike Hommey [:glandium]]

(In reply to Xidorn Quan [:xidorn] UTC+10 from comment #11)
> I tried comment 7, and it seems backing out bug 1471132 doesn't fix this
> issue.

This part implies your build was in a dangerous form before, but the build system wasn't reporting it, which it now is.

> neither does backing out bug 1470127.

Backing out bug 1470127 only doesn't make much sense, although it shouldn't actually hit the "NSModules are not ordered appropriately" error which you said on irc you are hitting. This also suggests your original error is not exactly the same as others, and the exact output from the build error you get with an unmodified tree would be interesting to have.

I'm more interested in a xul.dll and xul.pdb from a build with both bugs backed out, though.

Comment 13

[Mike Hommey [:glandium]]

Thanks to the files xidorn sent me, I could narrow this down to incremental linking because the cause for the problem. For some reason, the linker inserts padding when merging sections with incremental linking enabled. Incremental linking is enabled by default on debug builds, and explicitly disabled on automation, which explains why this works fine on automation.

To me, this is linker bug.

A workaround is to use LDFLAGS=-INCREMENTAL:NO.

Comment 14

[Mike Hommey [:glandium]]

Attachment: Bug 1471685 - Work around MSVC linker adding padding during incremental builds.

Review commit: https://reviewboard.mozilla.org/r/253710/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/253710/

Comment 15

[Xidorn Quan [:xidorn] UTC+10]

I can confirm that adding
> export LDFLAGS=-INCREMENTAL:NO
to mozconfig fixes this issue.

Comment 16

[Mike Hommey [:glandium]]

Comment on attachment 8988411 [details]
Bug 1471685 - Work around MSVC linker adding padding during incremental builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/253710/diff/1-2/

Comment 17

[Xidorn Quan [:xidorn] UTC+10]

And the latest commit also fixes this issue.

Comment 18

[Xidorn Quan [:xidorn] UTC+10]

I mean, the latest patch.

Comment 19

[Nathan Froyd [:froydnj]]

Comment on attachment 8988411 [details]
Bug 1471685 - Work around MSVC linker adding padding during incremental builds.

https://reviewboard.mozilla.org/r/253710/#review260352

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a57560dbc9ac
Work around MSVC linker adding padding during incremental builds. r=froydnj

Comment 21

[Mozilla Autoland]

We're sorry - something has gone wrong while rewriting or rebasing your commits. The commits being pushed no longer match what was requested. Please file a bug.

Comment 22

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/a57560dbc9ac

Comment 23

[Tom Ritter [:tjr]]

Attachment: Bug 1471685 - Work around MSVC linker adding padding during incremental builds. r=froydnj (esr60)

Comment 24

[Tom Ritter [:tjr]]

Comment on attachment 9035856 [details] [diff] [review]
Bug 1471685 - Work around MSVC linker adding padding during incremental builds. r=froydnj (esr60)

This is one of series of patches I am requesting uplift to esr60. Please don't uplift any if the entire series won't go. The whole series will need to go in one push.

This try run (applied on tip-of-esr60 as of an hour ago; and beginning with 'Bug 1491901 - move MK*SHLIB to moz.configure') represents the patch series. It must be applied in that order: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0659d6e265f3b624ad6fbac0c9cd7ce246094596 (If this try run doesn't complete successfully, I will investigate and figure out why)

The uplift request form is the same for all of the patch series; see https://bugzilla.mozilla.org/show_bug.cgi?id=1491901#c10

Comment 25

[Julien Cristau [:jcristau]]

Comment on attachment 9035856 [details] [diff] [review]
Bug 1471685 - Work around MSVC linker adding padding during incremental builds. r=froydnj (esr60)

approved for 60.5esr

Comment 26

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/f13c9d320dc7