All users were logged out of Bugzilla on October 13th, 2018

make the mac tier-2 cross-build taskcluster jobs work again

RESOLVED FIXED in Firefox 49

Status

RESOLVED FIXED
2 years ago
8 months ago

People

(Reporter: froydnj, Assigned: wcosta)

Tracking

Trunk
mozilla51
Dependency tree / graph

Firefox Tracking Flags

(firefox49 fixed, firefox50 fixed, firefox51 fixed)

Details

Attachments

(2 attachments, 3 obsolete attachments)

(Reporter)

Description

2 years ago
Bug 1246743 consciously broke these; fixing them should be a matter of straightening out the Mac cross sysroots to not use old libc++ headers.  This is probably easy with the move to 10.9 support, bug 1270217.

CC philor so he can use this bug for nefarious disabling purposes.
Flags: needinfo?(philringnalda)
Hid the TC OSX build jobs in the "TC OSX builds" exclusion.
Flags: needinfo?(philringnalda)
Duplicate of this bug: 1278510
(Assignee)

Updated

2 years ago
Blocks: 1244150
(Assignee)

Updated

2 years ago
Blocks: 1274980
We should turn these jobs off for now. Leaving them permafailing for over a month with no active plans to fix them is a waste of resources. If/when we start caring about these jobs again, turning them back on is always a single changeset away.
Ted - do you have any insight into the failures here?
Flags: needinfo?(ted)
They were broken when froydnj switched on libc++ in bug 1246743. I was under the impression that it was going to get fixed in the near future (and just broken temporarily so as to not block landing that change), but clearly that hasn't been the case.
Flags: needinfo?(ted)
(Assignee)

Updated

2 years ago
Assignee: nobody → wcosta
Status: NEW → ASSIGNED
If we need to, we can add `--enable-macos-target=10.9` in the cross mozconfig if that fixes things, until bug 1270217 is fixable:
https://dxr.mozilla.org/mozilla-central/source/build/macosx/cross-mozconfig.common
I tried this, but it looks like more modification of the environment and/or compiler options is required.

 22:36:29     INFO -  checking whether the C compiler (/usr/bin/ccache /home/worker/workspace/build/src/clang/bin/clang -target x86_64-apple-darwin10 -mlinker-version=136 -B /home/worker/workspace/build/src/cctools/bin -isysroot /home/worker/workspace/build/src/MacOSX10.7.sdk -std=gnu99  -Wl,-syslibroot,/home/worker/workspace/build/src/MacOSX10.7.sdk -Wl,-dead_strip) works... no
 22:36:29     INFO -  configure: error: installation or configuration problem: C compiler cannot create executables.
 22:36:29     INFO -  DEBUG: <truncated - see config.log for full output>
 22:36:29     INFO -  DEBUG: clang-3.8: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated
 22:36:29     INFO -  DEBUG: clang-3.8: warning: treating 'cpp-output' input as 'c++-cpp-output' when in C++ mode, this behavior is deprecated
 22:36:29     INFO -  DEBUG: configure:1985: checking for gcc
 22:36:29     INFO -  DEBUG: configure:2098: checking whether the C compiler (/usr/bin/ccache /home/worker/workspace/build/src/clang/bin/clang -target x86_64-apple-darwin10 -mlinker-version=136 -B /home/worker/workspace/build/src/cctools/bin -isysroot /home/worker/workspace/build/src/MacOSX10.7.sdk -std=gnu99  -Wl,-syslibroot,/home/worker/workspace/build/src/MacOSX10.7.sdk -Wl,-dead_strip) works
 22:36:29     INFO -  DEBUG: configure:2114: /usr/bin/ccache /home/worker/workspace/build/src/clang/bin/clang -target x86_64-apple-darwin10 -mlinker-version=136 -B /home/worker/workspace/build/src/cctools/bin -isysroot /home/worker/workspace/build/src/MacOSX10.7.sdk -std=gnu99 -o conftest   -Wl,-syslibroot,/home/worker/workspace/build/src/MacOSX10.7.sdk -Wl,-dead_strip conftest.c  1>&5
 22:36:29     INFO -  DEBUG: configure:2111:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
 22:36:29     INFO -  DEBUG: main(){return(0);}
 22:36:29     INFO -  DEBUG: ^
 22:36:29     INFO -  DEBUG: 1 warning generated.
 22:36:29     INFO -  DEBUG: ld: warning: directory not found for option '-L/home/worker/workspace/src/obj-firefox'
 22:36:29     INFO -  DEBUG: ld: warning: directory not found for option '-L/home/worker/workspace/src/gcc/lib64'
 22:36:29     INFO -  DEBUG: ld: entry point (start) undefined.  Usually in crt1.o for architecture x86_64
 22:36:29 INFO - DEBUG: clang-3.8: error: linker command failed with exit code 1 (use -v to see invocation)
(Assignee)

Comment 8

2 years ago
I tried to fix this by using SDK 10.9. After some roller coaster trip, figured it out binutils no longer worked with this new SDK. Ted suggested rebuilding it with [0]. Turns out the script was outdated. I updated it and added a taskcluster job [1]. But it never built ld and gas, because they are unsupported for darwin target [2], and from my git log discoveries in binutils-gdb repo, they never were. So, my question is, how did we get ld and gas in our current build of cctools?

[0] https://dxr.mozilla.org/mozilla-central/source/taskcluster/scripts/misc/build-cctools.sh
[1] https://github.com/walac/gecko-dev/commit/22a974c2d069ee3e33abaa1b98f9ebb158d64a51
[2] https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=configure;h=a5f4fc5aa74efb55e6d2574b169080c9ddcc2b81;hb=HEAD#l3710
Flags: needinfo?(ted)
Flags: needinfo?(nfroyd)
cctools isn't binutils--it's Apple's equivalent. I'm not sure why you changed the build scripts to build binutils instead of cctools?
Flags: needinfo?(ted)
If that crosstool-ng fork is outdated we might need to try a different source, like https://github.com/tpoechtrager/cctools-port
(Reporter)

Comment 11

2 years ago
(In reply to Wander Lairson Costa [:wcosta] from comment #8)
> So, my question is, how did we get ld and gas in our current build of cctools?

As Ted has noted, cctools != binutils.
Flags: needinfo?(nfroyd)
(Assignee)

Comment 12

2 years ago
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> cctools isn't binutils--it's Apple's equivalent. I'm not sure why you
> changed the build scripts to build binutils instead of cctools?

Well, the original repo doesn't exist anymore, then I googled for crosstool-ng and found github.com/crosstool-ng/crosstool-ng, which uses binutils. In summary, it was a chain of unfortunate misunderstandings that eventually led to a big *facepalm* moment.
(Assignee)

Comment 13

2 years ago
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #10)
> If that crosstool-ng fork is outdated we might need to try a different
> source, like https://github.com/tpoechtrager/cctools-port

Ok, just for the records, this a different fork, build-cctools.sh must be changed for new set of scripts, right?
Flags: needinfo?(ted)
Correct, you'd be rewriting build-cctools.sh to use whatever build process that repo has.

FYI, the original repo I used seems to still be there:
https://github.com/diorcety/crosstool-ng

It just hasn't had any commits in 2 years, which is not super promising.
Flags: needinfo?(ted)
(Assignee)

Comment 15

2 years ago
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #14)
> Correct, you'd be rewriting build-cctools.sh to use whatever build process
> that repo has.
> 
> FYI, the original repo I used seems to still be there:
> https://github.com/diorcety/crosstool-ng
> 

Wtf?!?! The repo was 404 when I tried to build cctools.

> It just hasn't had any commits in 2 years, which is not super promising.

Anyway, better migrating to something that's receiving maintenance anyway. I could build it in a Ubuntu box, migrating the script to build in desktop-builder.
(In reply to Wander Lairson Costa [:wcosta] from comment #15)
> Anyway, better migrating to something that's receiving maintenance anyway. I
> could build it in a Ubuntu box, migrating the script to build in
> desktop-builder.

That has the risk of taking you on a journey to fix stuff, because of the unexpected differences in the toolchain. It's better to get those builds running again and *then* switch to something that is up-to-date than the opposite. Especially for things like the linker. I'm pretty sure things will break by not building with ld64.
(Assignee)

Comment 17

2 years ago
(In reply to Mike Hommey [:glandium] from comment #16)
> That has the risk of taking you on a journey to fix stuff, because of the
> unexpected differences in the toolchain. It's better to get those builds
> running again and *then* switch to something that is up-to-date than the
> opposite. Especially for things like the linker. I'm pretty sure things will
> break by not building with ld64.

Right on my face. The new cctools-port needs at least gcc 4.7, so I am using tooltool clang, but I am having linking problems with libpthread https://tools.taskcluster.net/task-inspector/#CWYugApFTuS5oS-FLIB2yA/0
(Assignee)

Comment 18

2 years ago
Update: just rebuilding cctools didn't work [0]. I suspect we need a new version of cctools. There are C++ compile errors as well than didn't happen before, but one problem at a time. Neither crosstool repos build cleanly in Cent OS. I am diving into this issue. 


[0]https://tools.taskcluster.net/task-inspector/#X_yVq4atSgqr0DxOTfNrxg/
(Assignee)

Comment 19

2 years ago
Ok, I could build cctools-port, but I still have linking problems with libc++. I setup a little test source code (all the tests were performed in a one-click-loaner docker container):

#include <iostream>
int main() {
        std::cout << "Hello World" << std::endl;
}

It compiles successfully:

workspace/build/src/clang/bin/clang++ -c -o test.o --target=x86_64-apple-darwin10 -isysroot workspace/build/src/MacOSX10.9.sdk -B workspace/build/src/cctools/bin -stdlib=libc++ -v test.C

But when I try to link:

# workspace/build/src/cctools/bin/x86_64-apple-darwin10-ld -dynamic -arch x86_64 -macosx_version_min 10.6.0 -syslibroot workspace/build/src/MacOSX10.9.sdk -o a.out -lcrt1.10.6.o test.o -lc++ -lSystem -v
264.3.102
configured to support archs: armv4t armv5 armv6 armv7 armv7f armv7k armv7s armv6m armv7m armv7em armv8 arm64 arm64v8 i386 x86_64 x86_64h (tvOS)
Library search paths:
        workspace/build/src/MacOSX10.9.sdk/usr/lib
Framework search paths:
        workspace/build/src/MacOSX10.9.sdk/System/Library/Frameworks/
Undefined symbols for architecture x86_64:
  "__ZNSolsEPFRSoS_E", referenced from:
      _main in test.o
  "__ZNSt8ios_base4InitC1Ev", referenced from:
      ___cxx_global_var_init in test.o
  "__ZNSt8ios_base4InitD1Ev", referenced from:
      ___cxx_global_var_init in test.o
  "__ZSt4cout", referenced from:
      _main in test.o
  "__ZSt4endlIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_", referenced from:
      _main in test.o
  "__ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc", referenced from:
      _main in test.o
ld: symbol(s) not found for architecture x86_64

It cannot resolve for symbols in libc++. The reason for this is that the mangled names generated by the compiler are different from that in libc++. Take, for example, ios_base4InitC1Ev, in libc++ the symbols are mangled differently:

# cctools/bin/x86_64-apple-darwin10-nm -g MacOSX10.9.sdk/usr/lib/libc++.1.dylib  | grep ios_base | grep Init
0000000000013ee2 T __ZNSt3__18ios_base4InitC1Ev
0000000000013f2a T __ZNSt3__18ios_base4InitC2Ev
0000000000013eec T __ZNSt3__18ios_base4InitD1Ev
0000000000014314 T __ZNSt3__18ios_base4InitD2Ev
00014c22 T __ZNSt3__18ios_base4InitC1Ev
00014c84 T __ZNSt3__18ios_base4InitC2Ev
00014c32 T __ZNSt3__18ios_base4InitD1Ev
00015044 T __ZNSt3__18ios_base4InitD2Ev

Looking at Xcode 6, which ships SDK 10.9, the clang version is 6.0, which is based on official 3.5svn:

wcosta@Wanders-MBP:bin $ ./clang --version
Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
wcosta@Wanders-MBP:bin $ pwd
/Volumes/Xcode/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

In principle I suspect this has to do with different clang versions between shipped SDK and our build machines. But, why doesn't it happen in BB with SDK 10.7? Ideas? Just for the record, I have this problem with the linker since I switched to SDK 10.9.

https://tools.taskcluster.net/task-inspector/#Kp58-srXRiqv-pbCKpkpHQ/0
Flags: needinfo?(ted)
Flags: needinfo?(nfroyd)

Comment 20

2 years ago
I can get a build with clang built by clang team. But not with clang built by mozilla which seems do not support libc++
(Assignee)

Comment 21

2 years ago
(In reply to zhoubcfan@163.com from comment #20)
> I can get a build with clang built by clang team. But not with clang built
> by mozilla which seems do not support libc++

Are you doing cross builds in linux host and macosx target or compiling natively in the target environment?
(Reporter)

Comment 22

2 years ago
My guess at the issue is that the clang you are using has its own set of libc++ headers and those are what's getting used when you compile with the 10.9 SDK, but not the 10.7 SDK, as the SDK is checked for headers first, then whatever comes along with the compiler.  Last time I checked, the libc++ headers aren't being distributed in newer SDKs, but instead come packaged with XCode.  You can do something like:

find $OSX_10_7_SDK -name iostream
find $OSX_10_9_SDK -name iostream

and the two commands ought to return something very different.

The name mangling bits used by the compiler really really shouldn't vary between compiler versions, so if you are seeing mismatches between the names the compiler is generating and the names required by the libraries you are linking against, then the problem is not the compiler, but whatever is being input to the compiler to generate those names.  In this case, that would be the libc++ headers.
Flags: needinfo?(nfroyd)
(Assignee)

Comment 23

2 years ago
(In reply to Nathan Froyd [:froydnj] from comment #22)
> My guess at the issue is that the clang you are using has its own set of
> libc++ headers and those are what's getting used when you compile with the
> 10.9 SDK, but not the 10.7 SDK, as the SDK is checked for headers first,
> then whatever comes along with the compiler.

That would be surprising, because I had to make a symlink inside MacOSX10.9.sdk/usr/include/c++ folder:
# ll
total 8
drwxr-xr-x 7 worker worker 4096 Apr 23  2015 4.2.1
lrwxrwxrwx 1 worker worker   69 Jul 27 13:11 v1 -> /home/worker/workspace/build/src/MacOSX10.9.sdk/usr/include/c++/4.2.1

If I don't, clang doesn't find standard C++ headers when I use -stdlib=libc++. That folder comes along with 10.7 sdk, but not with 10.9.
(Reporter)

Comment 24

2 years ago
(In reply to Wander Lairson Costa [:wcosta] from comment #23)
> (In reply to Nathan Froyd [:froydnj] from comment #22)
> > My guess at the issue is that the clang you are using has its own set of
> > libc++ headers and those are what's getting used when you compile with the
> > 10.9 SDK, but not the 10.7 SDK, as the SDK is checked for headers first,
> > then whatever comes along with the compiler.
> 
> That would be surprising, because I had to make a symlink inside
> MacOSX10.9.sdk/usr/include/c++ folder:
> # ll
> total 8
> drwxr-xr-x 7 worker worker 4096 Apr 23  2015 4.2.1
> lrwxrwxrwx 1 worker worker   69 Jul 27 13:11 v1 ->
> /home/worker/workspace/build/src/MacOSX10.9.sdk/usr/include/c++/4.2.1
> 
> If I don't, clang doesn't find standard C++ headers when I use
> -stdlib=libc++. That folder comes along with 10.7 sdk, but not with 10.9.

You've created a symlink from where clang is looking for the libc++ headers to the libstdc++ headers.  Those two things are very different.  If the v1 folder isn't in the 10.9 SDK, that means the libc++ headers don't come with the 10.9 SDK.
(Assignee)

Comment 25

2 years ago
(In reply to Nathan Froyd [:froydnj] from comment #24)
> (In reply to Wander Lairson Costa [:wcosta] from comment #23)
> > (In reply to Nathan Froyd [:froydnj] from comment #22)
> > > My guess at the issue is that the clang you are using has its own set of
> > > libc++ headers and those are what's getting used when you compile with the
> > > 10.9 SDK, but not the 10.7 SDK, as the SDK is checked for headers first,
> > > then whatever comes along with the compiler.
> > 
> > That would be surprising, because I had to make a symlink inside
> > MacOSX10.9.sdk/usr/include/c++ folder:
> > # ll
> > total 8
> > drwxr-xr-x 7 worker worker 4096 Apr 23  2015 4.2.1
> > lrwxrwxrwx 1 worker worker   69 Jul 27 13:11 v1 ->
> > /home/worker/workspace/build/src/MacOSX10.9.sdk/usr/include/c++/4.2.1
> > 
> > If I don't, clang doesn't find standard C++ headers when I use
> > -stdlib=libc++. That folder comes along with 10.7 sdk, but not with 10.9.
> 
> You've created a symlink from where clang is looking for the libc++ headers
> to the libstdc++ headers.  Those two things are very different.  If the v1
> folder isn't in the 10.9 SDK, that means the libc++ headers don't come with
> the 10.9 SDK.

Oh, you are right, thanks. Indeed, SDK 10.9 doesn't ship libc++ headers, they are shipped with clang toolchain. I think I need to rebuild clang and include libc++?
(Reporter)

Comment 26

2 years ago
(In reply to Wander Lairson Costa [:wcosta] from comment #25)
> Oh, you are right, thanks. Indeed, SDK 10.9 doesn't ship libc++ headers,
> they are shipped with clang toolchain. I think I need to rebuild clang and
> include libc++?

That should work, yes.  I'm not sure what libc++'s binary compat guarantees are; you do run the risk of having newer headers that would expect to find certain symbols in libc++.dylib, but those symbols don't exist in the libc++.dylib that you're linking to.  But including libc++ would get you further down the path, I bet.
(Assignee)

Comment 27

2 years ago
Created attachment 8775380 [details] [diff] [review]
Add libc++ to clang. r=ted

We need to rebuild clang with libc++ to get compatible headers for cross
builds.
(Assignee)

Comment 28

2 years ago
Created attachment 8775381 [details] [diff] [review]
Fix macosx64 cross builds. r=ted

Use clang with libc++.
(Assignee)

Comment 29

2 years ago
Comment on attachment 8775380 [details] [diff] [review]
Add libc++ to clang. r=ted

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b5a31439a18f
Flags: needinfo?(ted)
Attachment #8775380 - Flags: review?(ted)
Comment on attachment 8775381 [details] [diff] [review]
Fix macosx64 cross builds. r=ted

Review of attachment 8775381 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/config/tooltool-manifests/macosx64/cross-releng.manifest
@@ +1,3 @@
>  [
>  {
> +"version": "clang 3.8.0",

please keep the version of libgcc. Please don't change clang on those builds only, but on *all* linux-host builds. We don't want to be using different versions of clang. Also, if you modify the build-clang script in a way that impacts other platforms, you'll have to build clang on those platforms and switch them too. We don't want discrepancy between build-clang and the clang actually used.
Attachment #8775381 - Flags: review?(ted)
Duplicate of this bug: 1247306
Attachment #8775380 - Flags: review?(ted) → review+
(Assignee)

Updated

2 years ago
Attachment #8775380 - Attachment is obsolete: true
(Assignee)

Updated

2 years ago
Attachment #8775381 - Attachment is obsolete: true
(Assignee)

Comment 38

2 years ago
Created attachment 8775715 [details] [diff] [review]
part 1: Add libc++ to clang. r=glandium

We need to rebuild clang with libc++ to get compatible headers for cross
builds. libc++abi is a dependency of libc++, as the build instructions
says [0].

[0] http://libcxx.llvm.org/docs/BuildingLibcxx.html
(Assignee)

Comment 39

2 years ago
Created attachment 8775716 [details] [diff] [review]
part 2: Update clang for linux hosts. r=ted

Update clang for the built version shipping libc++. This is primarly
intended to fix Mac OSX cross builds, but for a matter of consistency,
we update it for all clang builds done in a Linux host.
(Assignee)

Updated

2 years ago
Attachment #8775715 - Flags: review?(mh+mozilla)
(Assignee)

Updated

2 years ago
Attachment #8775716 - Flags: review?(ted)
Attachment #8775716 - Flags: feedback?(mh+mozilla)
Attachment #8775716 - Flags: review?(ted) → review+
Attachment #8775715 - Flags: review?(mh+mozilla) → review+
Attachment #8775716 - Flags: feedback?(mh+mozilla) → feedback+
Comment on attachment 8775716 [details] [diff] [review]
part 2: Update clang for linux hosts. r=ted

Review of attachment 8775716 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/config/tooltool-manifests/linux32/clang.manifest
@@ +1,5 @@
>  [
>  {
>  "version": "clang 3.8.0, libgcc 4.8.5",
> +"size": 139101340,
> +"digest": "27c34cfaf5c0ecf25f73dc8370aca6f54857c631ab5628d66c8ed3e8aae81cd984442909d7516985d35c8471ab82cd2224c622a0a280b843edfef79f72f02603",

This actually doesn't match any of the clang.tar.xz from the clang tasks on your last 6 try pushes.
Attachment #8775716 - Flags: feedback+ → feedback-
(Assignee)

Updated

2 years ago
Attachment #8775716 - Attachment is obsolete: true
(Assignee)

Comment 42

2 years ago
Created attachment 8776991 [details] [diff] [review]
part 2: Update clang for linux hosts. r=ted

Update clang for the built version shipping libc++. This is primarly
intended to fix Mac OSX cross builds, but for a matter of consistency,
we update it for all clang builds done in a Linux host.
(Assignee)

Updated

2 years ago
Attachment #8776991 - Flags: review?(mh+mozilla)
(Assignee)

Comment 43

2 years ago
(In reply to Mike Hommey [:glandium] from comment #40)
> Comment on attachment 8775716 [details] [diff] [review]
> part 2: Update clang for linux hosts. r=ted
> 
> Review of attachment 8775716 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: browser/config/tooltool-manifests/linux32/clang.manifest
> @@ +1,5 @@
> >  [
> >  {
> >  "version": "clang 3.8.0, libgcc 4.8.5",
> > +"size": 139101340,
> > +"digest": "27c34cfaf5c0ecf25f73dc8370aca6f54857c631ab5628d66c8ed3e8aae81cd984442909d7516985d35c8471ab82cd2224c622a0a280b843edfef79f72f02603",
> 
> This actually doesn't match any of the clang.tar.xz from the clang tasks on
> your last 6 try pushes.

This is because I used an older build when I didn't format commit messages properly. Anyway, I have updated the patch to include the latest clang from the series of 6 try pushes.
Attachment #8776991 - Flags: review?(mh+mozilla) → review+

Comment 45

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/c75deec80f9a
https://hg.mozilla.org/mozilla-central/rev/1c0dc20b3472
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
status-firefox51: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla51
For 49, I'm just going to take the cross-build tooltool manifest change to un-break those jobs while leaving the rest alone (since otherwise things get intertwined with bug 1278718).

Comment 48

2 years ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/3966f3217387
status-firefox49: affected → fixed
Turns out Beta has some funkiness around setting --target causing opt builds to fail. On 50+, things work as expected (maybe due to bug 1288313). After discussion with Ted, we felt that the best option for 49 was to hack around it instead of messing around with uplifts. I verified that the patch below only produces a behavior change for Taskcluster-based OSX cross-builds (not the regular buildbot ones we actually ship) and will therefore land it a=NPOTB with IRC r+ from Ted.

https://hg.mozilla.org/releases/mozilla-beta/rev/50aa7f83a669

I've also unhidden OSX tc(B) across all trees \m/.

Updated

8 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.