1273981 - make the mac tier-2 cross-build taskcluster jobs work again

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Nathan Froyd [:froydnj]]

Bug 1246743 consciously broke these; fixing them should be a matter of straightening out the Mac cross sysroots to not use old libc++ headers.  This is probably easy with the move to 10.9 support, bug 1270217.

CC philor so he can use this bug for nefarious disabling purposes.

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Hid the TC OSX build jobs in the "TC OSX builds" exclusion.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

We should turn these jobs off for now. Leaving them permafailing for over a month with no active plans to fix them is a waste of resources. If/when we start caring about these jobs again, turning them back on is always a single changeset away.

Comment 4

[Selena Deckelmann :selenamarie :selena]

Ted - do you have any insight into the failures here?

Comment 5

[(not currently active) Ted Mielczarek]

They were broken when froydnj switched on libc++ in bug 1246743. I was under the impression that it was going to get fixed in the near future (and just broken temporarily so as to not block landing that change), but clearly that hasn't been the case.

Comment 6

[(not currently active) Ted Mielczarek]

If we need to, we can add `--enable-macos-target=10.9` in the cross mozconfig if that fixes things, until bug 1270217 is fixable:
https://dxr.mozilla.org/mozilla-central/source/build/macosx/cross-mozconfig.common

Comment 7

[Ralph Giles (:rillian)]

I tried this, but it looks like more modification of the environment and/or compiler options is required.

 22:36:29     INFO -  checking whether the C compiler (/usr/bin/ccache /home/worker/workspace/build/src/clang/bin/clang -target x86_64-apple-darwin10 -mlinker-version=136 -B /home/worker/workspace/build/src/cctools/bin -isysroot /home/worker/workspace/build/src/MacOSX10.7.sdk -std=gnu99  -Wl,-syslibroot,/home/worker/workspace/build/src/MacOSX10.7.sdk -Wl,-dead_strip) works... no
 22:36:29     INFO -  configure: error: installation or configuration problem: C compiler cannot create executables.
 22:36:29     INFO -  DEBUG: <truncated - see config.log for full output>
 22:36:29     INFO -  DEBUG: clang-3.8: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated
 22:36:29     INFO -  DEBUG: clang-3.8: warning: treating 'cpp-output' input as 'c++-cpp-output' when in C++ mode, this behavior is deprecated
 22:36:29     INFO -  DEBUG: configure:1985: checking for gcc
 22:36:29     INFO -  DEBUG: configure:2098: checking whether the C compiler (/usr/bin/ccache /home/worker/workspace/build/src/clang/bin/clang -target x86_64-apple-darwin10 -mlinker-version=136 -B /home/worker/workspace/build/src/cctools/bin -isysroot /home/worker/workspace/build/src/MacOSX10.7.sdk -std=gnu99  -Wl,-syslibroot,/home/worker/workspace/build/src/MacOSX10.7.sdk -Wl,-dead_strip) works
 22:36:29     INFO -  DEBUG: configure:2114: /usr/bin/ccache /home/worker/workspace/build/src/clang/bin/clang -target x86_64-apple-darwin10 -mlinker-version=136 -B /home/worker/workspace/build/src/cctools/bin -isysroot /home/worker/workspace/build/src/MacOSX10.7.sdk -std=gnu99 -o conftest   -Wl,-syslibroot,/home/worker/workspace/build/src/MacOSX10.7.sdk -Wl,-dead_strip conftest.c  1>&5
 22:36:29     INFO -  DEBUG: configure:2111:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
 22:36:29     INFO -  DEBUG: main(){return(0);}
 22:36:29     INFO -  DEBUG: ^
 22:36:29     INFO -  DEBUG: 1 warning generated.
 22:36:29     INFO -  DEBUG: ld: warning: directory not found for option '-L/home/worker/workspace/src/obj-firefox'
 22:36:29     INFO -  DEBUG: ld: warning: directory not found for option '-L/home/worker/workspace/src/gcc/lib64'
 22:36:29     INFO -  DEBUG: ld: entry point (start) undefined.  Usually in crt1.o for architecture x86_64
 22:36:29 INFO - DEBUG: clang-3.8: error: linker command failed with exit code 1 (use -v to see invocation)

Comment 8

[Wander Lairson Costa]

I tried to fix this by using SDK 10.9. After some roller coaster trip, figured it out binutils no longer worked with this new SDK. Ted suggested rebuilding it with [0]. Turns out the script was outdated. I updated it and added a taskcluster job [1]. But it never built ld and gas, because they are unsupported for darwin target [2], and from my git log discoveries in binutils-gdb repo, they never were. So, my question is, how did we get ld and gas in our current build of cctools?

[0] https://dxr.mozilla.org/mozilla-central/source/taskcluster/scripts/misc/build-cctools.sh
[1] https://github.com/walac/gecko-dev/commit/22a974c2d069ee3e33abaa1b98f9ebb158d64a51
[2] https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=configure;h=a5f4fc5aa74efb55e6d2574b169080c9ddcc2b81;hb=HEAD#l3710

Comment 9

[(not currently active) Ted Mielczarek]

cctools isn't binutils--it's Apple's equivalent. I'm not sure why you changed the build scripts to build binutils instead of cctools?

Comment 10

[(not currently active) Ted Mielczarek]

If that crosstool-ng fork is outdated we might need to try a different source, like https://github.com/tpoechtrager/cctools-port

Comment 11

[Nathan Froyd [:froydnj]]

(In reply to Wander Lairson Costa [:wcosta] from comment #8)
> So, my question is, how did we get ld and gas in our current build of cctools?

As Ted has noted, cctools != binutils.

Comment 12

[Wander Lairson Costa]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> cctools isn't binutils--it's Apple's equivalent. I'm not sure why you
> changed the build scripts to build binutils instead of cctools?

Well, the original repo doesn't exist anymore, then I googled for crosstool-ng and found github.com/crosstool-ng/crosstool-ng, which uses binutils. In summary, it was a chain of unfortunate misunderstandings that eventually led to a big *facepalm* moment.

Comment 13

[Wander Lairson Costa]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #10)
> If that crosstool-ng fork is outdated we might need to try a different
> source, like https://github.com/tpoechtrager/cctools-port

Ok, just for the records, this a different fork, build-cctools.sh must be changed for new set of scripts, right?

Comment 14

[(not currently active) Ted Mielczarek]

Correct, you'd be rewriting build-cctools.sh to use whatever build process that repo has.

FYI, the original repo I used seems to still be there:
https://github.com/diorcety/crosstool-ng

It just hasn't had any commits in 2 years, which is not super promising.

Comment 15

[Wander Lairson Costa]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #14)
> Correct, you'd be rewriting build-cctools.sh to use whatever build process
> that repo has.
> 
> FYI, the original repo I used seems to still be there:
> https://github.com/diorcety/crosstool-ng
> 

Wtf?!?! The repo was 404 when I tried to build cctools.

> It just hasn't had any commits in 2 years, which is not super promising.

Anyway, better migrating to something that's receiving maintenance anyway. I could build it in a Ubuntu box, migrating the script to build in desktop-builder.

Comment 16

[Mike Hommey [:glandium]]

(In reply to Wander Lairson Costa [:wcosta] from comment #15)
> Anyway, better migrating to something that's receiving maintenance anyway. I
> could build it in a Ubuntu box, migrating the script to build in
> desktop-builder.

That has the risk of taking you on a journey to fix stuff, because of the unexpected differences in the toolchain. It's better to get those builds running again and *then* switch to something that is up-to-date than the opposite. Especially for things like the linker. I'm pretty sure things will break by not building with ld64.

Comment 17

[Wander Lairson Costa]

(In reply to Mike Hommey [:glandium] from comment #16)
> That has the risk of taking you on a journey to fix stuff, because of the
> unexpected differences in the toolchain. It's better to get those builds
> running again and *then* switch to something that is up-to-date than the
> opposite. Especially for things like the linker. I'm pretty sure things will
> break by not building with ld64.

Right on my face. The new cctools-port needs at least gcc 4.7, so I am using tooltool clang, but I am having linking problems with libpthread https://tools.taskcluster.net/task-inspector/#CWYugApFTuS5oS-FLIB2yA/0

Comment 18

[Wander Lairson Costa]

Update: just rebuilding cctools didn't work [0]. I suspect we need a new version of cctools. There are C++ compile errors as well than didn't happen before, but one problem at a time. Neither crosstool repos build cleanly in Cent OS. I am diving into this issue. 


[0]https://tools.taskcluster.net/task-inspector/#X_yVq4atSgqr0DxOTfNrxg/

Comment 19

[Wander Lairson Costa]

Ok, I could build cctools-port, but I still have linking problems with libc++. I setup a little test source code (all the tests were performed in a one-click-loaner docker container):

#include <iostream>
int main() {
        std::cout << "Hello World" << std::endl;
}

It compiles successfully:

workspace/build/src/clang/bin/clang++ -c -o test.o --target=x86_64-apple-darwin10 -isysroot workspace/build/src/MacOSX10.9.sdk -B workspace/build/src/cctools/bin -stdlib=libc++ -v test.C

But when I try to link:

# workspace/build/src/cctools/bin/x86_64-apple-darwin10-ld -dynamic -arch x86_64 -macosx_version_min 10.6.0 -syslibroot workspace/build/src/MacOSX10.9.sdk -o a.out -lcrt1.10.6.o test.o -lc++ -lSystem -v
264.3.102
configured to support archs: armv4t armv5 armv6 armv7 armv7f armv7k armv7s armv6m armv7m armv7em armv8 arm64 arm64v8 i386 x86_64 x86_64h (tvOS)
Library search paths:
        workspace/build/src/MacOSX10.9.sdk/usr/lib
Framework search paths:
        workspace/build/src/MacOSX10.9.sdk/System/Library/Frameworks/
Undefined symbols for architecture x86_64:
  "__ZNSolsEPFRSoS_E", referenced from:
      _main in test.o
  "__ZNSt8ios_base4InitC1Ev", referenced from:
      ___cxx_global_var_init in test.o
  "__ZNSt8ios_base4InitD1Ev", referenced from:
      ___cxx_global_var_init in test.o
  "__ZSt4cout", referenced from:
      _main in test.o
  "__ZSt4endlIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_", referenced from:
      _main in test.o
  "__ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc", referenced from:
      _main in test.o
ld: symbol(s) not found for architecture x86_64

It cannot resolve for symbols in libc++. The reason for this is that the mangled names generated by the compiler are different from that in libc++. Take, for example, ios_base4InitC1Ev, in libc++ the symbols are mangled differently:

# cctools/bin/x86_64-apple-darwin10-nm -g MacOSX10.9.sdk/usr/lib/libc++.1.dylib  | grep ios_base | grep Init
0000000000013ee2 T __ZNSt3__18ios_base4InitC1Ev
0000000000013f2a T __ZNSt3__18ios_base4InitC2Ev
0000000000013eec T __ZNSt3__18ios_base4InitD1Ev
0000000000014314 T __ZNSt3__18ios_base4InitD2Ev
00014c22 T __ZNSt3__18ios_base4InitC1Ev
00014c84 T __ZNSt3__18ios_base4InitC2Ev
00014c32 T __ZNSt3__18ios_base4InitD1Ev
00015044 T __ZNSt3__18ios_base4InitD2Ev

Looking at Xcode 6, which ships SDK 10.9, the clang version is 6.0, which is based on official 3.5svn:

wcosta@Wanders-MBP:bin $ ./clang --version
Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
wcosta@Wanders-MBP:bin $ pwd
/Volumes/Xcode/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

In principle I suspect this has to do with different clang versions between shipped SDK and our build machines. But, why doesn't it happen in BB with SDK 10.7? Ideas? Just for the record, I have this problem with the linker since I switched to SDK 10.9.

https://tools.taskcluster.net/task-inspector/#Kp58-srXRiqv-pbCKpkpHQ/0

Comment 20

[[e-mail]]

I can get a build with clang built by clang team. But not with clang built by mozilla which seems do not support libc++

Comment 21

[Wander Lairson Costa]

(In reply to [e-mail] from comment #20)
> I can get a build with clang built by clang team. But not with clang built
> by mozilla which seems do not support libc++

Are you doing cross builds in linux host and macosx target or compiling natively in the target environment?

Comment 22

[Nathan Froyd [:froydnj]]

My guess at the issue is that the clang you are using has its own set of libc++ headers and those are what's getting used when you compile with the 10.9 SDK, but not the 10.7 SDK, as the SDK is checked for headers first, then whatever comes along with the compiler.  Last time I checked, the libc++ headers aren't being distributed in newer SDKs, but instead come packaged with XCode.  You can do something like:

find $OSX_10_7_SDK -name iostream
find $OSX_10_9_SDK -name iostream

and the two commands ought to return something very different.

The name mangling bits used by the compiler really really shouldn't vary between compiler versions, so if you are seeing mismatches between the names the compiler is generating and the names required by the libraries you are linking against, then the problem is not the compiler, but whatever is being input to the compiler to generate those names.  In this case, that would be the libc++ headers.

Comment 23

[Wander Lairson Costa]

(In reply to Nathan Froyd [:froydnj] from comment #22)
> My guess at the issue is that the clang you are using has its own set of
> libc++ headers and those are what's getting used when you compile with the
> 10.9 SDK, but not the 10.7 SDK, as the SDK is checked for headers first,
> then whatever comes along with the compiler.

That would be surprising, because I had to make a symlink inside MacOSX10.9.sdk/usr/include/c++ folder:
# ll
total 8
drwxr-xr-x 7 worker worker 4096 Apr 23  2015 4.2.1
lrwxrwxrwx 1 worker worker   69 Jul 27 13:11 v1 -> /home/worker/workspace/build/src/MacOSX10.9.sdk/usr/include/c++/4.2.1

If I don't, clang doesn't find standard C++ headers when I use -stdlib=libc++. That folder comes along with 10.7 sdk, but not with 10.9.

Comment 24

[Nathan Froyd [:froydnj]]

(In reply to Wander Lairson Costa [:wcosta] from comment #23)
> (In reply to Nathan Froyd [:froydnj] from comment #22)
> > My guess at the issue is that the clang you are using has its own set of
> > libc++ headers and those are what's getting used when you compile with the
> > 10.9 SDK, but not the 10.7 SDK, as the SDK is checked for headers first,
> > then whatever comes along with the compiler.
> 
> That would be surprising, because I had to make a symlink inside
> MacOSX10.9.sdk/usr/include/c++ folder:
> # ll
> total 8
> drwxr-xr-x 7 worker worker 4096 Apr 23  2015 4.2.1
> lrwxrwxrwx 1 worker worker   69 Jul 27 13:11 v1 ->
> /home/worker/workspace/build/src/MacOSX10.9.sdk/usr/include/c++/4.2.1
> 
> If I don't, clang doesn't find standard C++ headers when I use
> -stdlib=libc++. That folder comes along with 10.7 sdk, but not with 10.9.

You've created a symlink from where clang is looking for the libc++ headers to the libstdc++ headers.  Those two things are very different.  If the v1 folder isn't in the 10.9 SDK, that means the libc++ headers don't come with the 10.9 SDK.

Comment 25

[Wander Lairson Costa]

(In reply to Nathan Froyd [:froydnj] from comment #24)
> (In reply to Wander Lairson Costa [:wcosta] from comment #23)
> > (In reply to Nathan Froyd [:froydnj] from comment #22)
> > > My guess at the issue is that the clang you are using has its own set of
> > > libc++ headers and those are what's getting used when you compile with the
> > > 10.9 SDK, but not the 10.7 SDK, as the SDK is checked for headers first,
> > > then whatever comes along with the compiler.
> > 
> > That would be surprising, because I had to make a symlink inside
> > MacOSX10.9.sdk/usr/include/c++ folder:
> > # ll
> > total 8
> > drwxr-xr-x 7 worker worker 4096 Apr 23  2015 4.2.1
> > lrwxrwxrwx 1 worker worker   69 Jul 27 13:11 v1 ->
> > /home/worker/workspace/build/src/MacOSX10.9.sdk/usr/include/c++/4.2.1
> > 
> > If I don't, clang doesn't find standard C++ headers when I use
> > -stdlib=libc++. That folder comes along with 10.7 sdk, but not with 10.9.
> 
> You've created a symlink from where clang is looking for the libc++ headers
> to the libstdc++ headers.  Those two things are very different.  If the v1
> folder isn't in the 10.9 SDK, that means the libc++ headers don't come with
> the 10.9 SDK.

Oh, you are right, thanks. Indeed, SDK 10.9 doesn't ship libc++ headers, they are shipped with clang toolchain. I think I need to rebuild clang and include libc++?

Comment 26

[Nathan Froyd [:froydnj]]

(In reply to Wander Lairson Costa [:wcosta] from comment #25)
> Oh, you are right, thanks. Indeed, SDK 10.9 doesn't ship libc++ headers,
> they are shipped with clang toolchain. I think I need to rebuild clang and
> include libc++?

That should work, yes.  I'm not sure what libc++'s binary compat guarantees are; you do run the risk of having newer headers that would expect to find certain symbols in libc++.dylib, but those symbols don't exist in the libc++.dylib that you're linking to.  But including libc++ would get you further down the path, I bet.

Comment 27

[Wander Lairson Costa]

Attachment: Add libc++ to clang. r=ted

We need to rebuild clang with libc++ to get compatible headers for cross
builds.

Comment 28

[Wander Lairson Costa]

Attachment: Fix macosx64 cross builds. r=ted

Use clang with libc++.

Comment 29

[Wander Lairson Costa]

Comment on attachment 8775380 [details] [diff] [review]
Add libc++ to clang. r=ted

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b5a31439a18f

Comment 30

[Wander Lairson Costa]

Comment on attachment 8775381 [details] [diff] [review]
Fix macosx64 cross builds. r=ted

https://treeherder.allizom.org/#/jobs?repo=try&revision=afa09ee20335&selectedJob=25182398

Comment 31

[Mike Hommey [:glandium]]

Comment on attachment 8775381 [details] [diff] [review]
Fix macosx64 cross builds. r=ted

Review of attachment 8775381 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/config/tooltool-manifests/macosx64/cross-releng.manifest
@@ +1,3 @@
>  [
>  {
> +"version": "clang 3.8.0",

please keep the version of libgcc. Please don't change clang on those builds only, but on *all* linux-host builds. We don't want to be using different versions of clang. Also, if you modify the build-clang script in a way that impacts other platforms, you'll have to build clang on those platforms and switch them too. We don't want discrepancy between build-clang and the clang actually used.

Comment 32

[Wander Lairson Costa]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2d1ff289db64

Comment 33

[Wander Lairson Costa]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c2fd0717fc23

Comment 34

[Wander Lairson Costa]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b32e4ae676f8

Comment 35

[Wander Lairson Costa]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cdc150a9730f

Comment 36

[Wander Lairson Costa]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1bfdfd5c0104

Comment 38

[Wander Lairson Costa]

Attachment: part 1: Add libc++ to clang. r=glandium

We need to rebuild clang with libc++ to get compatible headers for cross
builds. libc++abi is a dependency of libc++, as the build instructions
says [0].

[0] http://libcxx.llvm.org/docs/BuildingLibcxx.html

Comment 39

[Wander Lairson Costa]

Attachment: part 2: Update clang for linux hosts. r=ted

Update clang for the built version shipping libc++. This is primarly
intended to fix Mac OSX cross builds, but for a matter of consistency,
we update it for all clang builds done in a Linux host.

Comment 40

[Mike Hommey [:glandium]]

Comment on attachment 8775716 [details] [diff] [review]
part 2: Update clang for linux hosts. r=ted

Review of attachment 8775716 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/config/tooltool-manifests/linux32/clang.manifest
@@ +1,5 @@
>  [
>  {
>  "version": "clang 3.8.0, libgcc 4.8.5",
> +"size": 139101340,
> +"digest": "27c34cfaf5c0ecf25f73dc8370aca6f54857c631ab5628d66c8ed3e8aae81cd984442909d7516985d35c8471ab82cd2224c622a0a280b843edfef79f72f02603",

This actually doesn't match any of the clang.tar.xz from the clang tasks on your last 6 try pushes.

Comment 41

[Wander Lairson Costa]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=be3a5940176f

Comment 42

[Wander Lairson Costa]

Attachment: part 2: Update clang for linux hosts. r=ted

Update clang for the built version shipping libc++. This is primarly
intended to fix Mac OSX cross builds, but for a matter of consistency,
we update it for all clang builds done in a Linux host.

Comment 43

[Wander Lairson Costa]

(In reply to Mike Hommey [:glandium] from comment #40)
> Comment on attachment 8775716 [details] [diff] [review]
> part 2: Update clang for linux hosts. r=ted
> 
> Review of attachment 8775716 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: browser/config/tooltool-manifests/linux32/clang.manifest
> @@ +1,5 @@
> >  [
> >  {
> >  "version": "clang 3.8.0, libgcc 4.8.5",
> > +"size": 139101340,
> > +"digest": "27c34cfaf5c0ecf25f73dc8370aca6f54857c631ab5628d66c8ed3e8aae81cd984442909d7516985d35c8471ab82cd2224c622a0a280b843edfef79f72f02603",
> 
> This actually doesn't match any of the clang.tar.xz from the clang tasks on
> your last 6 try pushes.

This is because I used an older build when I didn't format commit messages properly. Anyway, I have updated the patch to include the latest clang from the series of 6 try pushes.

Comment 44

[Wander Lairson Costa]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c75deec80f9ad5b3e5e260c35d2efe1be8a194b3
Bug 1273981 part 1: Add libc++ to clang. r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/1c0dc20b3472fc85065e1811c39d867c7077c62d
Bug 1273981 part 2: Update clang for linux hosts. r=ted

Comment 45

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/c75deec80f9a
https://hg.mozilla.org/mozilla-central/rev/1c0dc20b3472

Comment 46

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/81424f27115f
https://hg.mozilla.org/releases/mozilla-aurora/rev/f44def10d980

Comment 47

[Ryan VanderMeulen [:RyanVM]]

For 49, I'm just going to take the cross-build tooltool manifest change to un-break those jobs while leaving the rest alone (since otherwise things get intertwined with bug 1278718).

Comment 48

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/3966f3217387

Comment 49

[Ryan VanderMeulen [:RyanVM]]

Turns out Beta has some funkiness around setting --target causing opt builds to fail. On 50+, things work as expected (maybe due to bug 1288313). After discussion with Ted, we felt that the best option for 49 was to hack around it instead of messing around with uplifts. I verified that the patch below only produces a behavior change for Taskcluster-based OSX cross-builds (not the regular buildbot ones we actually ship) and will therefore land it a=NPOTB with IRC r+ from Ted.

https://hg.mozilla.org/releases/mozilla-beta/rev/50aa7f83a669

I've also unhidden OSX tc(B) across all trees \m/.