Closed Bug 1300208 Opened 3 years ago Closed 3 years ago

Allow specifying multiple rust crates to mozbuild within the same FINAL_LIBRARY

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox52 fixed)

RESOLVED FIXED
mozilla52
Tracking Status
firefox52 --- fixed

People

(Reporter: Nika, Assigned: Nika)

References

Details

Attachments

(1 file, 11 obsolete files)

14.22 KB, patch
froydnj
: review+
Details | Diff | Splinter Review
This also handles rust libraries across fake static library objects.

This works by asking cargo to build rlibs instead of .a files. These rlib files are propagated through mozbuild like a STATIC_LIBRARY. When expandlibs_exec.py is asked to expand any libraries, if passed a --prelink-rust argument, it will pass these rlibs to rustc, generating a static library which is then provided to the calling program.

This fixes the linking issues blocking bug 1295762.
MozReview-Commit-ID: IIjV4Kg7wOv
Attachment #8787783 - Flags: review?(nfroyd)
NOTE: The build currently fails on try because I'm forgetting to pass the target triple to rustc when it links the implicit metacrate. This should be an easy fix next monday.
Why do we need this? Can't you just reference all the crates you need via toolkit/library/rust/Cargo.toml?
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #3)
> Why do we need this? Can't you just reference all the crates you need via
> toolkit/library/rust/Cargo.toml?

We actually have 2 different crate roots right now in our tree, `toolkit/library/rust/Cargo.toml` and `toolkit/library/gtest/rust/Cargo.toml`. In the future, we might have crates which aren't built into libxul, but instead are part of spidermonkey or NSS.

This change allows us to treat crate roots like a source in our tree, and not have to worry about making sure that we only have a single crate within each FINAL_LIBRARY (which can be tricky, as if FINAL_LIBRARY is a STATIC_LIB, and we also want to include rust in a program which links that STATIC_LIB, there is no easy solution right now, and this patch handles that case). 

This forward-proofs us for those situations, and makes the problems I was encountering with duplicate symbols when linking libxul-gtest due to the crate being used in libgkrust and libgkrust-gtest when working on bug 1295762 go away.
I fixed up the tests to pass, made sure to pass the right targets, and made sure that it works on win32. This is the fixed up patch which passes try with my nsstring patches applied on top of it:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ed7c01b56342f1a0e8b70f37f624dd5b98fc8872
Attachment #8787879 - Flags: review?(nfroyd)
Attachment #8787783 - Attachment is obsolete: true
Attachment #8787783 - Flags: review?(nfroyd)
Comment on attachment 8787879 [details] [diff] [review]
Allow specifying multiple rust crates to mozbuild within the same FINAL_LIBRARY

Review of attachment 8787879 [details] [diff] [review]:
-----------------------------------------------------------------

This is super-clever.  To be clear, the link line for xul-gtest now gets:

expandlibs_exec.py ... libgkrust.rlib ... libgkrust-test.rlib

and we link those two things together, so any duplicate crates get filtered out by the rust linker?

Also, for avoidance of doubt, libxul_s.a contains Rust code, correct?

::: config/config.mk
@@ +514,5 @@
>  MDDEPDIR := .deps
>  
>  EXPAND_LIBS_EXEC = $(PYTHON) $(MOZILLA_DIR)/config/expandlibs_exec.py
> +ifdef MOZ_RUST
> +EXPAND_LIBS_EXEC += --prelink-rust "$(RUSTC) -C panic=abort --target $(RUST_TARGET)"

This should just be '--prelink-rust', and expandlibs_exec.py can assemble the command line itself, so that we don't have to bother with argument splitting and whatnot.  Any configuration variables that you need can be retrieved from config/expandlibs_config.py after appropriate modification.

::: config/expandlibs_exec.py
@@ +266,5 @@
> +        tmp = tempfile.mkdtemp(dir=os.curdir)
> +        self.tmp.append(tmp)
> +
> +        # XXX: We might want to put this somewhere in the tree instead of in a
> +        # tempfile, but that would require more build system integration

I think we don't want to do this.  Let's say somebody modifies some C++ code and rebuilds.  With these changes, and assuming I understand the code here, we're going to relink all the Rust code in libxul, even if none of it has changed.  That is undesirable, and will only get more painful as we acquire more Rust code.
Attachment #8787879 - Flags: review?(nfroyd) → feedback+
(In reply to Nathan Froyd [:froydnj] from comment #6)
> Comment on attachment 8787879 [details] [diff] [review]
> Allow specifying multiple rust crates to mozbuild within the same
> FINAL_LIBRARY
> 
> Review of attachment 8787879 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> This is super-clever.  To be clear, the link line for xul-gtest now gets:
> 
> expandlibs_exec.py ... libgkrust.rlib ... libgkrust-test.rlib
> 
> and we link those two things together, so any duplicate crates get filtered
> out by the rust linker?

Yup! We now propagate .rlib files around like .o files, and at link time merge them with the rust linker, so that duplicate crates get filtered out by the rust linker, and we get only one copy of the 

> Also, for avoidance of doubt, libxul_s.a contains Rust code, correct?

There is only a libxul_s.a.desc, and it contains `libgkrust.rlib` in it. If we actually created a real libxul_s.a, it would contain that rust code in it, but we don't.

> ::: config/config.mk
> @@ +514,5 @@
> >  MDDEPDIR := .deps
> >  
> >  EXPAND_LIBS_EXEC = $(PYTHON) $(MOZILLA_DIR)/config/expandlibs_exec.py
> > +ifdef MOZ_RUST
> > +EXPAND_LIBS_EXEC += --prelink-rust "$(RUSTC) -C panic=abort --target $(RUST_TARGET)"
> 
> This should just be '--prelink-rust', and expandlibs_exec.py can assemble
> the command line itself, so that we don't have to bother with argument
> splitting and whatnot.  Any configuration variables that you need can be
> retrieved from config/expandlibs_config.py after appropriate modification.

Good point. I'll make those changes, and post a new version of this patch.

> ::: config/expandlibs_exec.py
> @@ +266,5 @@
> > +        tmp = tempfile.mkdtemp(dir=os.curdir)
> > +        self.tmp.append(tmp)
> > +
> > +        # XXX: We might want to put this somewhere in the tree instead of in a
> > +        # tempfile, but that would require more build system integration
> 
> I think we don't want to do this.  Let's say somebody modifies some C++ code
> and rebuilds.  With these changes, and assuming I understand the code here,
> we're going to relink all the Rust code in libxul, even if none of it has
> changed.  That is undesirable, and will only get more painful as we acquire
> more Rust code.

Right now when linking C++ code in gecko, we always link in a single step all of our C++ object files. We do fake intermediate links by creating the .a.desc files, but those don't actually perform any linking. By linking the rust crates together at the same time that we do other links, we actually make our rust build system act more consistently with our C++ one.

In our old system we "partially linked" ahead of time by linking the rlibs together into a .a earlier on, now we just do it when we would invoke our normal link step. I doubt that this will significantly hurt build times, and I think it actually simplifies our build system conceptually by being more consistent.

In addition, if we wanted to link this staticlib earlier on, it would require some fairly significant changes to our system, because the linking strategy requires us to look through our .a.desc faux-static-lib abstraction to discover all of the .rlib files, and that normally isn't done until this point when we invoke the linker. We don't have the information required to create the staticlib until this point.

This comment is more related to the fact that our .a and .rs file which we generate are currently going into a temporary directory instead of into the objdir, which is unfortunate, as it means that they are hard to inspect if they cause a build failure. I could parse the linker argument list for a output file argument, and put the temporary files in the same directory, but this is easier, and these objects are not really interesting temporaries.

(If I sound confusing or you have other questions ping me on irc - it might be easier to explain what's going on here there).
This is an updated patch which fixes your review comments, and now caches the intermediate link result in the outdir next to the linker final library as part of the build process.

Currently doing a try run to make sure that it works on all platforms. I'll mark you for review once I've confirmed that.
Attachment #8787879 - Attachment is obsolete: true
Fixed builds without --enable-rust
Attachment #8788701 - Flags: review?(nfroyd)
Attachment #8788650 - Attachment is obsolete: true
Comment on attachment 8788701 [details] [diff] [review]
Allow specifying multiple rust crates to mozbuild within the same FINAL_LIBRARY

Review of attachment 8788701 [details] [diff] [review]:
-----------------------------------------------------------------

I like the idea, but I am torn about hiding actual build parameters (e.g. optimization flags, panic=abort) in this file, which duplicates information found in the Cargo.toml files and adds another place for people to have to modify (and remember to modify!) when/if we need to change Rust compilation settings.  (People have already expressed dislike that debug builds compile with opt-level=1, and I imagine that dislike would only increase if they found they had to modify multiple places to make that go away.)  Suspending review temporarily while getting glandium's feedback.

glandium, WDYT here?  Ideally, I think we'd express this in moz.build, but the way we link libxul/libxul-gtest and Rust's want-to-see-the-world approach to linking don't really play nicely together.  Perhaps this is the least awful solution?

::: config/expandlibs_exec.py
@@ +244,5 @@
>          f.close()
>          self.tmp.append(tmp)
>          self.append(option % tmp)
>  
> +    # Discover what the output file from the underlying call will by by looking

Nit: "...will be by looking..."

@@ +259,5 @@
> +        return path
> +
> +    def prelinkRust(self, verbose):
> +        # Split the arguments into the rlibs, other arguments which came before
> +        # the first rlibs, and other arguments which came after the first rlibs.

You mean "before the first rlib" and "after the first rlib" (singular) here, right?
Attachment #8788701 - Flags: review?(nfroyd) → feedback?(mh+mozilla)
(In reply to Nathan Froyd [:froydnj] from comment #10)
> I like the idea, but I am torn about hiding actual build parameters (e.g.
> optimization flags, panic=abort) in this file, which duplicates information
> found in the Cargo.toml files and adds another place for people to have to
> modify (and remember to modify!) when/if we need to change Rust compilation
> settings.  (People have already expressed dislike that debug builds compile
> with opt-level=1, and I imagine that dislike would only increase if they
> found they had to modify multiple places to make that go away.)  Suspending
> review temporarily while getting glandium's feedback.
> 
> glandium, WDYT here?  Ideally, I think we'd express this in moz.build, but
> the way we link libxul/libxul-gtest and Rust's want-to-see-the-world
> approach to linking don't really play nicely together.  Perhaps this is the
> least awful solution?

As I said to nathan on IRC, I have a prototype for a change to our build system which generates Cargo.toml files with mozbuild, allowing for --disable-optimize to affect rust code as well as C++ code. This would fix many of the problems with having to change multiple places in the build system.

I'll link the bug here when I polish it up enough to share.
MozReview-Commit-ID: IIjV4Kg7wOv
Attachment #8788995 - Flags: feedback?(mh+mozilla)
Attachment #8788701 - Attachment is obsolete: true
Attachment #8788701 - Flags: feedback?(mh+mozilla)
(In reply to Michael Layzell [:mystor] from comment #11)
> As I said to nathan on IRC, I have a prototype for a change to our build
> system which generates Cargo.toml files with mozbuild, allowing for
> --disable-optimize to affect rust code as well as C++ code. This would fix
> many of the problems with having to change multiple places in the build
> system.
> 
> I'll link the bug here when I polish it up enough to share.

The patch is in bug 1300835, and applies on top of this one.
This is a solution to the problem of duplicated build options for rust files. It causes the prelink step to read the settings from gkrust's Cargo.toml file.
Attachment #8790940 - Flags: review?(nfroyd)
MozReview-Commit-ID: GX0FJjDVA5q
Attachment #8790941 - Flags: review?(nfroyd)
Comment on attachment 8788995 [details] [diff] [review]
Allow specifying multiple rust crates to mozbuild within the same FINAL_LIBRARY

Review of attachment 8788995 [details] [diff] [review]:
-----------------------------------------------------------------

There's way too much rust compiler knowledge encoded in there. This essentially goes in the opposite direction of the recent move to cargo. Relatedly, I don't want expandlibs to be have more features than it already has (because eventually I want to kill it ; adding features to it makes that harder).

My understanding was that we were going to build with a meta crate linking them all together with cargo? Did I misinterpret something? Nathan?
Attachment #8788995 - Flags: feedback?(mh+mozilla) → feedback-
(In reply to Michael Layzell [:mystor] from comment #7)
> In addition, if we wanted to link this staticlib earlier on, it would
> require some fairly significant changes to our system, because the linking
> strategy requires us to look through our .a.desc faux-static-lib abstraction
> to discover all of the .rlib files, and that normally isn't done until this
> point when we invoke the linker. We don't have the information required to
> create the staticlib until this point.

We don't and we do. We don't when make is recursing the build. But we do at the moment we generate the build backend.
(In reply to Mike Hommey [:glandium] from comment #16)
> My understanding was that we were going to build with a meta crate linking
> them all together with cargo? Did I misinterpret something? Nathan?

We were and we are doing that.  The problem comes when we have Rust parts of libxul (libgkrust) and the Rust parts of libxul-gtest (libgkrust-gtest) both wanting to include the same crate, $CRATE.  We compile libgkrust into a linkable .a file, where symbols from $CRATE are visible; that libgkrust.a is included in the description file for libxul.  We then compile libgkrust_gtest into a linkable .a file, where symbols from $CRATE are also visible.  When we go to link libxul-gtest, then, we are attempting to link libgkrust.a and libgkrust_gtest.a, both of which have definitions from $CRATE, and the linker complains.

This only happens in opt builds.  In debug builds, $CRATE appears as the same.o file in libgkrust.a and libgkrust_gtest.a, and the linker is clever enough that it takes the definitions from a single .o and ignores the identical, duplicate definitions.  In opt builds, however, we LTO libgkrust, and this changes how the linker sticks things together, because $CRATE now lives in a single, large, gkrust.0.o file in libgkrust.a.  Whatever the linker did in debug builds, it no longer does, and we get duplicate definition errors.  (In both builds, the Rust functions in question live in identically-named sections, fwiw.)

Note too that the above problem is not just for crates we write ourselves (nsstring in bug 1295762); we also see link errors for things in the libc crate (!), though why the particular things from libc are being kept around as visible symbols when we don't actually use those things eludes me.

This wasn't a problem before because our gtest Rust code was extremely minimal.  But with increasing amounts of Rust code coming into the tree, it is not reasonable to restrict how we code things in Rust + gtest.

Does that explanation make sense?
Flags: needinfo?(mh+mozilla)
Comment on attachment 8790941 [details] [diff] [review]
Part 3: Ensure that panic = 'abort' is specified for all toplevel Cargo.tomls

Review of attachment 8790941 [details] [diff] [review]:
-----------------------------------------------------------------

This is fine as far as it goes, but ted's vendoring patches that just landed are going to bitrot this patch, so I'd like to see a revised version rebased on top of that.
Attachment #8790941 - Flags: review?(nfroyd) → feedback+
(In reply to Mike Hommey [:glandium] from comment #17)
> (In reply to Michael Layzell [:mystor] from comment #7)
> > In addition, if we wanted to link this staticlib earlier on, it would
> > require some fairly significant changes to our system, because the linking
> > strategy requires us to look through our .a.desc faux-static-lib abstraction
> > to discover all of the .rlib files, and that normally isn't done until this
> > point when we invoke the linker. We don't have the information required to
> > create the staticlib until this point.
> 
> We don't and we do. We don't when make is recursing the build. But we do at
> the moment we generate the build backend.

If we wanted to take advantage of that information, I feel like we would have to do all of the work to remove expandlibs in addition. We can add that as a follow-up to this bug (I don't think that this would be particularly hard to move when the infrastructure for removing expandlibs is removed), but I think we should be able to land this and get rust builds building before we do all of the work to fix that.
MozReview-Commit-ID: IIjV4Kg7wOv
Attachment #8791284 - Flags: review?(nfroyd)
Attachment #8788995 - Attachment is obsolete: true
Attachment #8790940 - Attachment is obsolete: true
Attachment #8790941 - Attachment is obsolete: true
Attachment #8790940 - Flags: review?(nfroyd)
MozReview-Commit-ID: JDEc8ZEWlgj
Attachment #8791285 - Flags: review?(nfroyd)
MozReview-Commit-ID: GX0FJjDVA5q
Attachment #8791286 - Flags: review?(nfroyd)
I rebased on inbound, and have attached the rebased patches above.
(In reply to Michael Layzell [:mystor] from comment #20)
> If we wanted to take advantage of that information, I feel like we would
> have to do all of the work to remove expandlibs in addition.

I don't think so.
Comment on attachment 8791284 [details] [diff] [review]
Part 1: Allow specifying multiple rust crates to mozbuild within the same FINAL_LIBRARY

Review of attachment 8791284 [details] [diff] [review]:
-----------------------------------------------------------------

My interpretation of comment 25 is that we can know, at build-backend time, all the rlibs that are going into a library.  So we can adopt bug 1163224's approach to write out a .rs file that builds the final staticlib, and have that linked into libxul.  expandlibs will ignore the .rlib files, and we'll shoehorn our magic library into FINAL_LIBRARY the same way that bug 1163224 did.  That strikes me as a more workable approach than trying to shoehorn everything into expandlibs.  It's not great (it would be nicer if Cargo did all this stuff), but it does put dependency checking, temporary .rs files, etc. in the build system, where it belongs.  I don't see a way to make Rust's compilation model cooperate with expandlibs beyond the ones we've discussed and discarded.

We might have to write a RUSTC wrapper than reads in profile settings from toolkit/library/rust/Cargo.toml to get -C opt-level et al, though.
Attachment #8791284 - Flags: review?(nfroyd)
Comment on attachment 8791285 [details] [diff] [review]
Part 2: Read rust build options from gkrust's Cargo.toml when prelinking rust

Review of attachment 8791285 [details] [diff] [review]:
-----------------------------------------------------------------

Canceling because we suggested a different approach in part 1.
Attachment #8791285 - Flags: review?(nfroyd)
Comment on attachment 8791286 [details] [diff] [review]
Part 3: Ensure that panic = 'abort' is specified for all toplevel Cargo.tomls

Review of attachment 8791286 [details] [diff] [review]:
-----------------------------------------------------------------

I think this can be applied independently of the other patches.  Want to spin it out into a separate bug?
Attachment #8791286 - Flags: review?(nfroyd) → review+
I don't do the expandlibs thing anymore in this patch - Hopefully it works out better?
Attachment #8791826 - Flags: review?(nfroyd)
Attachment #8791284 - Attachment is obsolete: true
Attachment #8791285 - Attachment is obsolete: true
Attachment #8791286 - Attachment is obsolete: true
Comment on attachment 8791826 [details] [diff] [review]
Part 1: Allow specifying multiple rust crates to mozbuild within the same FINAL_LIBRARY

Glandium, is this a better patch than the original? I tried to not depend on expandlibs, only adding some code to make sure that we don't accidentally include another library's rust prelink in .a.desc files.
Attachment #8791826 - Flags: feedback?(mh+mozilla)
Comment on attachment 8791826 [details] [diff] [review]
Part 1: Allow specifying multiple rust crates to mozbuild within the same FINAL_LIBRARY

Review of attachment 8791826 [details] [diff] [review]:
-----------------------------------------------------------------

This is definitely better.  I'm sad about the expandlibs change, and would have thought that shoehorning the prelink library in on  the side the way it was done pre-Cargo would have worked OK.  Did that approach not pan out?

::: config/expandlibs_gen.py
@@ +22,5 @@
>                  raise Exception("File not found: %s" % arg)
>          elif os.path.splitext(arg)[1] == conf.LIB_SUFFIX:
> +            # We want to skip static libraries with the name foo-rs-prelink
> +            # as they are individually linked for every final library, and
> +            # thus should not be included in the descriptor file

This is sort of unfortunate, but maybe that's the way things have to be.

@@ +29,5 @@
>              if os.path.exists(arg) or os.path.exists(arg + conf.LIBS_DESC_SUFFIX):
>                  desc['LIBS'].append(os.path.abspath(arg))
>              else:
>                  raise Exception("File not found: %s" % arg)
> +

Please remove this.

::: config/rules.mk
@@ +961,5 @@
> +endif
> +
> +$(RUST_PRELINK): $(RUST_PRELINK_DEPS)
> +	$(REPORT_BUILD)
> +	echo $(RUST_PRELINK_SRC) | $(RUSTC) -o $@ --crate-type staticlib $(RUST_PRELINK_FLAGS) -

Surely we need to provide --target=$(RUST_TARGET) here as well?

::: python/mozbuild/mozbuild/backend/recursivemake.py
@@ +1235,5 @@
>  
> +        def findAllRlibs(obj):
> +            if isinstance(obj, RustLibrary):
> +                return [obj]
> +            if isinstance(obj, StaticLibrary) and not obj.no_expand_lib:

Why is this no_expand_lib check here?

@@ +1246,5 @@
> +        # Check if we have any rust libraries to prelink and include in our final library.
> +        # If we do, write out the RUST_PRELINK information
> +        rlibs = []
> +        for l in obj.linked_libraries:
> +            rlibs += findAllRlibs(l)

There has to be a better way to write this function:

def find_rlibs(obj):
  if isinstance(obj, (SharedLibrary, StaticLibrary)):
    for l in obj.linked_libraries:
      for r in find_rlibs(l):
        yield r
  elif isinstance(obj, RustLibrary):
    yield obj

rlibs = list(find_rlibs(obj))

assuming that we can avoid the no_expand_lib check.  But even adding the no_expand_lib check, I think doing it this way is nicer.

@@ +1247,5 @@
> +        # If we do, write out the RUST_PRELINK information
> +        rlibs = []
> +        for l in obj.linked_libraries:
> +            rlibs += findAllRlibs(l)
> +        if len(rlibs) > 0:

You can just say |if rlibs:| here.

@@ +1261,5 @@
> +            backend_file.write('RUST_PRELINK_DEPS :=\n')
> +            for rlib in rlibs:
> +                rlib_relpath = pretty_relpath(rlib)
> +                backend_file.write('RUST_PRELINK_SRC += "extern crate %s;"\n'
> +                                   % rlib.basename.replace('-', '_'))

This should be written out as a separate file, and compiled thusly in rules.mk.

::: python/mozbuild/mozbuild/frontend/data.py
@@ +488,5 @@
>              rust_build_kind,
>              self.lib_name,
>          )
> +        self.deps_path = '%s/%s/deps' % (
> +            context.config.substs['RUST_TARGET'],

Can you set this first, and then make self.import_name use self.deps_path?

::: toolkit/library/rust/Cargo.toml
@@ +33,5 @@
>  debug = true
>  rpath = false
>  lto = true
>  debug-assertions = false
> +codegen-units = 1

codegen-units is ignored when lto=true, so we can remove this.
Attachment #8791826 - Flags: review?(nfroyd) → feedback+
I updated to respond to your comments.

I couldn't use exactly your implementation. It is very important that when discovering rlibs I only look through static libraries which will use expandlibs, otherwise I would link in rlibs into my module which are already linked into a static or dynamic library which I will also link to. For example, if I used your implementation I would link the rust code directly into every test which builds against libxul, as it would look through the dynamic library, and discover libgkrust. That's the purpose of the no_expand_lib check.

I think other than that I have changed everything you asked about.
Attachment #8792150 - Flags: review?(nfroyd)
Attachment #8791826 - Attachment is obsolete: true
Attachment #8791826 - Flags: feedback?(mh+mozilla)
Comment on attachment 8792150 [details] [diff] [review]
Part 1: Allow specifying multiple rust crates to mozbuild within the same FINAL_LIBRARY

Review of attachment 8792150 [details] [diff] [review]:
-----------------------------------------------------------------

I think this makes sense.  I tried rewriting find_libs to unify the two loops, but nothing super-nice came out of it.

::: python/mozbuild/mozbuild/backend/recursivemake.py
@@ -1234,5 @@
>                                     % (relpath, lib.import_name))
>  
> -        # We have to link any Rust libraries after all intermediate static
> -        # libraries have been listed to ensure that the Rust libraries are
> -        # searched after the C/C++ objects that might reference Rust symbols.

I think we should keep this comment here.  (Just this part, not the below, which is completely bogus and it's a good thing it's going away.)

@@ +1247,5 @@
> +        if isinstance(obj, (SharedLibrary, StaticLibrary)):
> +            for l in obj.linked_libraries:
> +                rlibs += find_rlibs(l)
> +        if rlibs:
> +            print "RLIBS", rlibs

We can delete this line.
Attachment #8792150 - Flags: review?(nfroyd) → review+
Flags: needinfo?(mh+mozilla)
Pushed by michael@thelayzells.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/362852b89f4e
Allow specifying multiple rust crates to mozbuild within the same FINAL_LIBRARY, r=froydnj
https://hg.mozilla.org/mozilla-central/rev/362852b89f4e
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.