1351559 - Upload symbols archive containing files for the js shell

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Currently, in order to get symbols for the js shell, we need to download the full archive containing the DWARFs for the full Firefox browser. It might be more ideal to have a smaller archive containing just the files needed for the js shell.

See bug 1348229.

Comment 1

[Steve Fink [:sfink] [:s:]]

Attachment: Adjust job to be useful as a downloadable fuzzing build

This does not at all do what the bug asks for, but it gives you what you need.

As it turns out, we don't need to package up the symbols at all. The SM(f) (sm-fuzzing) build is not stripped and includes debug symbols. So the only thing you need is the llvm-symbolizer binary. This patch copies it over from clang.

To use this,

1. download https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.firefox.sm-fuzzing-linux64/artifacts/public/build/llvm-symbolizer.gz if you don't have it already.

2. gunzip llvm-symbolizer.gz; chmod +x llvm-symbolizer

3. Download https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.firefox.sm-fuzzing-linux64/artifacts/public/build/js

4. chmod +x js

To run it, use

  ASAN_SYMBOLIZER_PATH=$(pwd)/llvm-symbolizer ./js <...args...>

so for example, you should see a stack if you do

  ASAN_SYMBOLIZER_PATH=$(pwd)/llvm-symbolizer ./js -e 'crash()'

Comment 2

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Is there a way to know the m-c revision hash of this "latest" build? (i.e. download another .txt file or something)

Comment 3

[Steve Fink [:sfink] [:s:]]

Hm. I don't have a *good* way.

One way would be to grab https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.firefox.sm-fuzzing-linux64 which will give you a simple JSON blob like

{
  "namespace": "gecko.v2.mozilla-central.latest.firefox.sm-fuzzing-linux64",
  "taskId": "LjEFLkkuRhmgFQpLmCibcQ",
  "rank": 1513374744,
  "data": {},
  "expires": "2018-12-15T21:54:08.960Z"
}

Grab the taskId from there, then fetch (for example) https://queue.taskcluster.net/v1/task/LjEFLkkuRhmgFQpLmCibcQ

That gives another JSON blob that lists the revision ID in a couple of places. It's in response.route, which is a list, and one of the elements will be something like "index.gecko.v2.mozilla-central.revision.4398768baa23a1d1888116a484817e20f2701dcc.firefox.sm-fuzzing-linux64". Or payload.env.GECKO_HEAD_REV is set to just the revision (4398...). Or get it from 'metadata', which includes stuff like

 "metadata": {
    "source": "https://hg.mozilla.org/mozilla-central/file/4398768baa23a1d1888116a484817e20f2701dcc/taskcluster/ci/spidermonkey",
    "name": "spidermonkey-sm-fuzzing-linux64/opt"
  }

None of these answers are great, because in theory there's a race condition between fetching the js binary and fetching the initial JSON blob.

Comment 4

[Jon Coppeard (:jonco)]

Comment on attachment 8937233 [details] [diff] [review]
Adjust job to be useful as a downloadable fuzzing build

Review of attachment 8937233 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine by my knowledge in this area is pretty limited TBH.

::: taskcluster/scripts/builder/build-sm.sh
@@ +18,5 @@
>  cp -rL $SRCDIR/obj-spider/dist/bin/{js,jsapi-tests,js-gdb.py} $UPLOAD_DIR
>  
> +# Fuzzing users would really like to have llvm-symbolizer available in the same
> +# directory as the built output.
> +gzip -c $TOOLTOOL_CHECKOUT/clang/bin/llvm-symbolizer > $UPLOAD_DIR/llvm-symbolizer.gz || true

What's the "|| true" for?

Comment 5

[Steve Fink [:sfink] [:s:]]

(In reply to Jon Coppeard (:jonco) from comment #4)
> Comment on attachment 8937233 [details] [diff] [review]
> Adjust job to be useful as a downloadable fuzzing build
> 
> Review of attachment 8937233 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks fine by my knowledge in this area is pretty limited TBH.
> 
> ::: taskcluster/scripts/builder/build-sm.sh
> @@ +18,5 @@
> >  cp -rL $SRCDIR/obj-spider/dist/bin/{js,jsapi-tests,js-gdb.py} $UPLOAD_DIR
> >  
> > +# Fuzzing users would really like to have llvm-symbolizer available in the same
> > +# directory as the built output.
> > +gzip -c $TOOLTOOL_CHECKOUT/clang/bin/llvm-symbolizer > $UPLOAD_DIR/llvm-symbolizer.gz || true
> 
> What's the "|| true" for?

Um. [looks embarrassed]. So I expect this command to fail for some builds, namely the gcc-using ones. If this shell script used |set -e|, the failure would make the script abort, which would be no good.

But you might notice that this script does *not* use set -e. However, it does

    source $(dirname $0)/sm-tooltool-config.sh

Recently nbp had an SM(..) job fail that he was having trouble reproducing locally, and he noticed that the failed jobs were not uploading their binaries. I had thought through that, and intentionally tried to set it up so that it *would* upload binaries in all cases. The problem turned out to be the above 'source' line; sm-tooltool-config.sh uses |set -e|, and that was infecting anything that sourced it. :-( I have now fixed the problem, which means the || true is not actually needed here, but I'm paranoid and so my default is to make callers robust, at least for stupid little things like this change that I certainly don't want to mess up the overall job even if I screw up with sourcing a set -e script again.

So... it's not a *good* reason, but it's the reason. Read it as "note: it is expected that this job might fail sometimes, and we don't really care." I could even argue that having it this way would allow me to add a set -e to some or all of this script, if it turned out to make sense later. (Right now, it does not.)

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ca3090e98b00
Make fuzzing builds match firefox more closely, r=jonco

Comment 7

[Jon Coppeard (:jonco)]

(In reply to Steve Fink [:sfink] [:s:] from comment #5)
Ah, thanks for the explanation.  I looked for |set -e| but couldn't see it.

Comment 8

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/ca3090e98b00