Building using cross-platform scripts vs. xcodebuild yields wide performance disparities

UNCONFIRMED
Unassigned

Status

Tamarin
Build Config
P4
normal
UNCONFIRMED
8 years ago
7 years ago

People

(Reporter: William Maddox, Unassigned)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

unspecified
Future
x86
Mac OS X
Dependency tree / graph
Bug Flags:
flashplayer-injection -
flashplayer-qrb +
flashplayer-bug +

Details

Attachments

(2 attachments)

(Reporter)

Description

8 years ago
Created attachment 455248 [details]
Performance suite run of TR built with configure/make compared with TR build with xcodebuild

I have observed wide performance disparities on MacOS X when building using the cross-platform scripts vs. using xcodebuild.  The attachment shows the results of such a run.  The executables under test were built as follows:

mkdir objdir
cd objdir
python ../configure.py --enable-shell
make

xcodebuild ARCHS=i386 -project platform/mac/avmshell/avmshell.xcodeproj -configuration Release

It is likely that the default compiler options in the cross-platform scripts are suboptimal.
(Reporter)

Comment 1

8 years ago
Not only does xcodebuild seem to give better performance overall, it also introduces additional variability in the results.  While investigating bug 565489, I compared the effect of a patch with the baseline using executables built both with the cross-platform scripts and xcodebuild.

With executables built using xcodebuild, we get, for example, the following:

asmicro/closedvar-write-1.as     3771 3742.0   4847 4876.8   29.7  30.3 ++
asmicro/closedvar-write-2.as     4422 4379.8   3633 3644.8  -17.5 -22.7 --

With executables built using configure/make, we get the these results:

asmicro/closedvar-write-1.as     4789 4698.6   4938 4942.0    3.4  2.9 + 
asmicro/closedvar-write-2.as     3652 3643.4   3606 3610.6   -1.0 -3.7 - 

It is inconceivable that the disparity observed in the first case is actually due to the (first order effect of) the patch.

I have previously observed significant sensitivity of benchmark results to incidental changes in code placement in the executables, which is quite likely what we are seeing here.  Based on the observations so far, I would draw the tentative conclusion that builds for performance benchmarking should be made with the cross-platform infrastructure.  Clearly, we need to figure out what is going on with xcodebuild -- both why we see better performance overall but also greater sensitivity to perturbations in the code.
(Reporter)

Comment 2

8 years ago
Created attachment 455258 [details]
Effect of patch for bug 565489 when built with configure/make and xcodebuild

Performance benchmark results for the following patch:
https://bug565489.bugzilla.mozilla.org/attachment.cgi?id=455231

Results shown for builds with configure/make and with xcodebuild.
MacOS X (10.6.3), xcode 3.2.1

Developer Information:

  Version:	3.2 (10M2003)
  Location:	/Developer
  Applications:
  Xcode:	3.2.1 (1613)
  Interface Builder:	3.2.1 (740)
  Instruments:	2.0.1 (1096)
  Dashcode:	3.0 (328)
  SDKs:
  Mac OS X:
  10.4:	(8S2167)
  10.5:	(9J61)
  10.6:	(10M2003)

Comment 3

8 years ago
i've also seen wide discrepancies with the same exact source code, Makefiles, with two clones using the commandline build. (on a clock-locked linux machine) Something's up, and to isolate that possible effect, are the xcode and commandline builds above done from the same physical repository?
(In reply to comment #3)
> i've also seen wide discrepancies with the same exact source code, Makefiles,
> with two clones using the commandline build. (on a clock-locked linux machine)
> Something's up, and to isolate that possible effect, are the xcode and
> commandline builds above done from the same physical repository?

Just to second Ed's remarks here: on a different project (Larceny), I would sometimes see (wildly) different behaviors from two checkouts of the repository but otherwise identical builds.  Once when it was really important to track down a particular bug, I tracked it down to the length of the paths I was using to invoke the executable and to refer to its subsidiary components, like its heap image.  Once I put the checkouts into analogous directory structures, I got identical behavior.

So even if you are using the same repository, you may want to also ensure that the build object subdirectories holding the executables and such are located at analogous places in the directory substructure.

(One might go overboard here, and start wondering if the hash codes of the paths matter...)

Comment 5

8 years ago
QE: Can you take a crack at isolating the issue(s) at play here?  I would expect the same default compiler flags to be used in both build systems, even if this is not a big factor here.
Flags: flashplayer-qrb+
Priority: -- → P4
Target Milestone: --- → flash10.2

Updated

8 years ago
Assignee: nobody → trbaker

Updated

8 years ago
Assignee: trbaker → cpeyer
Status: NEW → ASSIGNED

Updated

7 years ago
Depends on: 641051
Flags: flashplayer-injection-
Flags: flashplayer-bug+

Comment 6

7 years ago
May not need to investigate this if we adopt gyp and drop the xplat system.
Assignee: cpeyer → nobody
Status: ASSIGNED → UNCONFIRMED
Component: JIT Compiler (NanoJIT) → Build Config
Ever confirmed: false
QA Contact: nanojit → build-config
Target Milestone: Q3 11 - Serrano → Future

Updated

7 years ago
Blocks: 645018
You need to log in before you can comment on or make changes to this bug.