570248 - Create new unittest suite: "jetpack-SDK" to verify tip-of-jetpack works

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

The objective here is to automatically run a set of jetpack tests against a vanilla firefox browser that has jetpack installed and enabled. 

The plan is to set this up as a new test suite which does the following:
 - grab existing m-c build
 - grab (from hg?) the latest jetpack
 - enable that jetpack in the firefox install 
 - grab (from hg?) and run jetpack unittest suite
 - close down and report results

Probably the easiest way to do this is grab an existing unittest suite, copy it, and then rip out and replace the parts you dont need. That way you get the base environment setup, the "grab and install existing build", and "close down and report results" steps for free.


Once this suite works once manually, close this bug, and we'll track the adding-to-automation work in the blocked bugs.


(There's a few variations on this theme, but lets walk-before-running; lets do tip-of-jetpack suite and get it running on mozilla-central. We can then see which versions of jetpack we need to test on which release and project branches)

Comment 1

[Brian Warner [:warner :bwarner]]

Some extra notes:

 - there are two test suites inside the Jetpack SDK. "cfx testcfx" exercises the python components, and does not require a mozlla application (firefox, xulrunner, thunderbird, etc). "cfx testall" exercises the JS portions, and *does* require an application
 - the "cfx testall" suite should be told which application to use, by e.g. passing "-b /path/to/firefox"
 - when regular developers use the SDK, they start by "activating" the SDK by sourcing a shell fragment which sets up their $PATH and environment. The "cfx" tool is then run from within this environment. As a result, it's a bit tricky to get the buildbot to run the test suite. For the prototype buildbot I threw together, I wrote a helper script which does the shell-script-juggling.
 - the combined test suites currently take about 11 seconds to execute

I'm not sure what would be the best way to "grab the latest jetpack [release]", but I imagine we're not the first project to discover a need for this. Maybe a special hg tag for "latest-release"? Or do an http fetch of some little page which tells us what the latest release tag should be?

Comment 2

[Brian Warner [:warner :bwarner]]

Attachment: master.cfg for my prototype buildmaster

Here's the master.cfg from my little prototype buildbot.

Comment 3

[Brian Warner [:warner :bwarner]]

Attachment: this is the 'run-cfx' helper script

the 'run-cfx' helper script is used to overcome the funky 'source bin/activate' / 'cfx testall' sequenced used to invoke the jetpack SDK test suite. It also uses xvfb-run to help run a child firefox process on a headless (virtualized) linux box.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

This bug is to track taking the existing jetpack test suite, and wrapper it so it can be invoked by RelEng infrastructure just like any other mozilla unittest suite.

Using a wrapper like this will allow all the usual "what port to use" "where to send logs" etc to work as usual. Not sure how easy/hard this will be; it might be easier to figure out which specific settings are needed and re-implement them from scratch. 

Leaving this bug here in MozillaLabs:JetpackSDK until ctalbert, joduinn, bwarner meet, do some experiments and figure out how to tackle this.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

bwarner, ctalbert: what needs to be done here? And I guess as importantly, who need to do it?

Comment 6

[Brian Warner [:warner :bwarner]]

I'm not sure. I talked briefly to lsblakk about the other bug, and IIRC she said a new branch wasn't necessary, and I think that ctalbert has the pieces he needs. I'll track down ctalbert in person today and see what I can do to help out.

Comment 7

[cmtalbert]

I'm not clear what I'm supposed to do here.  Running the jetpack SDK tests is as easy as the steps up there in the buildbot script that Brian attached.  I checked them out and started playing with them last week.  I think that looks really straightforward to wire into the existing RelEng buildslaves, so I'm not sure what we're expecting me (or someone on my team) to do.

Comment 8

[cmtalbert]

Sorry for the spam.  I should say, we're happy to do it, once we figure out what *it* is. :)

Comment 9

[Brian Warner [:warner :bwarner]]

Clint and I spoke to jhford, and it sounds like the next step is to get lsblakk to edit some buildbot configs. I'll stop by in the morning when she gets back and see what needs to be done.

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

buildbot command - run from inside the jetpack-sdk directory should work (in Xvfb) as:

/bin/sh -c "(source bin/activate; cfx testall)" 

or:

/bin/sh -c "(source bin/activate; cfx testall -a firefox)"

to test firefox specifically - but might want to make it possible to put in other applications

Comment 11

[Lukas Blakk [:lsblakk] use ?needinfo]

/bin/sh -c "(source bin/activate; cfx testall -a firefox -b $pathToExecutable)" alternately if the path is not /usr/bin/firefox (or not in your $PATH)

Comment 12

[Lukas Blakk [:lsblakk] use ?needinfo]

Closing since the suite now exists, made a note in bug 570251 regarding the scripts and the commands mentioned in comment #10 and comment #11

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

After meeting with Brian Warner and then John O'Duinn to discuss I realized what was missing from this process - we still need a wrapper for this test suite so that it works with our existing unittest suite automation.

This script is great and runs the test from the command line in a buildbot instance but it doesn't run in the way that the current suites do - via python script run on packaged build/test downloads - so this suite is still incomplete in terms of being able to hand it to Releng for automation into mozilla-central's test suites.

What would be great is to be able to have this suite run in such a way like our current crashtest, reftest, etc where we can simply enable it in our configs:

http://hg.mozilla.org/build/buildbot-configs/file/2a700c313c27/mozilla/config.py#l44

and run it from a python script in the build factory like our current packaged tests:

http://hg.mozilla.org/build/buildbotcustom/file/cdaaed009fde/steps/unittest.py#l321

I'm moving this to Testing for this stage of the process - when it's complete it can be passed back to Releng for activating the suite on mozilla-central and mozilla-1.9.2.

Comment 14

[Lukas Blakk [:lsblakk] use ?needinfo]

Update: Spoke with Clint to confirm what is needed here so that we have a unittest buildbot step that can be turned on per branch as needed for running the jetpack tests. Comparable to http://hg.mozilla.org/build/buildbotcustom/file/cdaaed009fde/steps/unittest.py#l321

Comment 15

[cmtalbert]

Attachment: Jetpack Test Step v1

So, here's a first stab at a quick and dirty integration.  This takes the jetpack test suite as-is and wires it into the buildbot steps that our buildbot system expects.  It makes use of the run-cfx script that Brian provided and should get the output from a normal jetpack run.

I haven't tested this at all yet, but just wanted to get something together and see if I'm on the right track.

Comment 16

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 457694 [details] [diff] [review]
Jetpack Test Step v1

Great start, thanks for jumping on it so fast.

I did a little poking around and it will be interesting handling this unittest suite's urls since the default assumptions in our factory right now are that there is a tests package http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l6181 to download and unpack as well as possibly a symbols download.

So it looks like we can go ahead with the idea of sending two files (urls) in: a firefox path (latest, release) and the jetpack 'build' (hg, latest) and I will check with more experienced buildbot folks about whether the best thing is to add a "jetpack_enabled" flag to branch we want this suite run on in order to maneuver around the symbols and unpacking steps in order to get to the Run Them! here: http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l6312 where we will use the packaged test step you're creating.

Comment 17

[Lukas Blakk [:lsblakk] use ?needinfo]

How's the testing on this coming along?  Any progress or ETA?

Comment 18

[cmtalbert]

Attachment: Some tweaks from testing

Sorry this took so long. So I found a couple of issues with the code, and fixed those.  I could not manage to get a sendchange to properly fire because I couldn't get a slave configured properly for some reason.  I did mock out a bunch of buildbot objects and managed to call a the commands the jetpack unitest step exposes.  I think this is going to work, but I think we won't really know until we put it in a true staging environment.

One thing it depends on:
* That the jetpack bundle is unpacked and that this step is executed from within that bundle.
* It also depends that the run-cfx script is checked in to the root of the jetpack sdk tree and that it is executable.

This is what happens when the jetpack test is run:
1. The jetpack package is unpacked to a directory, let's call it jetpack-sdk
2. The uniteststep is run with jetpack-sdk as its cwd
3. This uniteststep will run run-cfx script (which is available at jetpack-sdk/run-cfx)
4. The test runs, printing output to stdout
5. The builder captures that stdout log and it goes in the logs variable of the buildstep object
6. The jetpackStep's createSummary is called which greps through the collected log text and determines the total number of passes/fails and prints that as a tinderbox result.
7. If there is a Traceback, that is also flaged and the step in that case will return a warning to its caller.

Comment 19

[cmtalbert]

Attachment: runcfx patch

Here's the runcfx patch that puts it in misc and guards against trying to run xvfb on platforms that don't have it.  One more question (which shows my lack of linux knowledge) is xvfb-run some script you had on the buildbot box you were running?  I thought the command was only "xvfb"  Should we also check in the xvfb-run script too?

Comment 20

[Brian Warner [:warner :bwarner]]

Comment on attachment 467607 [details] [diff] [review]
runcfx patch

looks ok. My only comment would be that the use of xvfb-run is not really optional: firefox at least, and probably xulrunner too, won't run without an X context. So either xvfb-run should be used all the time, or the script should take an argument to say if you want it to use xvfb-run, rather than having the script silently skip using it when it's not installed (causing a confusing error later).

Oh, and "/usr/bin/xvfb-run" is a standard tool, part of the "xvfb" package (at least on debian). If you're using some other linux distribution, maybe they put it in some other package?

Comment 21

[cmtalbert]

(In reply to comment #20)
> Comment on attachment 467607 [details] [diff] [review]
> runcfx patch
> 
> looks ok. My only comment would be that the use of xvfb-run is not really
> optional: firefox at least, and probably xulrunner too, won't run without an X
> context. So either xvfb-run should be used all the time, or the script should
> take an argument to say if you want it to use xvfb-run, rather than having the
> script silently skip using it when it's not installed (causing a confusing
> error later).
> 
> Oh, and "/usr/bin/xvfb-run" is a standard tool, part of the "xvfb" package (at
> least on debian). If you're using some other linux distribution, maybe they put
> it in some other package?
My thought was that on systems where we need to run xvfb (i.e. Fedora linux) then that will be available. But for non-X11 based systems like windows and mac, this would just cause a script error.

Lukas, can you make sure that Xvfb-run exists on the Fedora talos slave reference platform?  I know that the Fedora slaves already run in a XVFB environment, so starting that up may actually be done higher up the chain in buildbot, and we may not need to mess with it at all in this script.  I'll go ahead with this as is, and depending on what Lukas's sleuthing uncovers about the Fedora buildbot slaves, we'll change it accordingly.

Comment 22

[cmtalbert]

Landed runcfx script: changeset f7b3157e9088

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

> Lukas, can you make sure that Xvfb-run exists on the Fedora talos slave
> reference platform?  I know that the Fedora slaves already run in a XVFB
> environment, so starting that up may actually be done higher up the chain in
> buildbot, and we may not need to mess with it at all in this script.  I'll go
> ahead with this as is, and depending on what Lukas's sleuthing uncovers about
> the Fedora buildbot slaves, we'll change it accordingly.

just checked out talos-r3-fed-001 staging box and /usr/bin/xvfb-run is present and accounted for.

Comment 24

[cmtalbert]

(In reply to comment #23)
> > Lukas, can you make sure that Xvfb-run exists on the Fedora talos slave
> > reference platform?  I know that the Fedora slaves already run in a XVFB
> > environment, so starting that up may actually be done higher up the chain in
> > buildbot, and we may not need to mess with it at all in this script.  I'll go
> > ahead with this as is, and depending on what Lukas's sleuthing uncovers about
> > the Fedora buildbot slaves, we'll change it accordingly.
> 
> just checked out talos-r3-fed-001 staging box and /usr/bin/xvfb-run is present
> and accounted for.
Sweet.  Any word on running this code in staging yet?

Comment 25

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: [wip] running the jetpack suite in staging

hey clint - so this is what i added to your existing work in order to actually run the test suite.  it's obviously a hack but i wanted you to see what commands i had to do in order to even get something going.  

a few hiccups in making this work that might be useful to put in a jetpack/runjetpack.py or something instead of in self.command: 

* getting the jetpack-sdk downloaded based on the .txt file (which is awesome btw), unpacked, and then navigating into the dir when its changeset is different each time 
* calling source bin/activate
* being able to locate the binary during tests (to be in jetpack-sdk-$changeset we are two dirs down from where the other unittest suites are called from)

here is sample output from the run so you can see what it looks like right now:

http://pastebin.mozilla.org/789779

Comment 26

[Lukas Blakk [:lsblakk] use ?needinfo]

Welcome to the Jetpack SDK. Run 'cfx docs' for assistance.Testing cfx...............................................................----------------------------------------------------------------------Ran 60 tests in 3.924sOKTesting reading-data...Traceback (most recent call last):  File "/Users/cltbld/talos-slave/mozilla-central_snowleopard_test-jetpack/build/jetpack/jetpack-sdk-9bfe1631d20c/bin/cfx", line 6, in <module>    cuddlefish.run()  File "/Users/cltbld/talos-slave/mozilla-central_snowleopard_test-jetpack/build/jetpack/jetpack-sdk-9bfe1631d20c/python-lib/cuddlefish/__init__.py", line 341, in run    test_all(env_root, defaults=options.__dict__)  File "/Users/cltbld/talos-slave/mozilla-central_snowleopard_test-jetpack/build/jetpack/jetpack-sdk-9bfe1631d20c/python-lib/cuddlefish/__init__.py", line 215, in test_all    test_all_examples(env_root, defaults)  File "/Users/cltbld/talos-slave/mozilla-central_snowleopard_test-jetpack/build/jetpack/jetpack-sdk-9bfe1631d20c/python-lib/cuddlefish/__init__.py", line 250, in test_all_examples    env_root=env_root)  File "/Users/cltbld/talos-slave/mozilla-central_snowleopard_test-jetpack/build/jetpack/jetpack-sdk-9bfe1631d20c/python-lib/cuddlefish/__init__.py", line 602, in run    addons=options.addons)  File "/Users/cltbld/talos-slave/mozilla-central_snowleopard_test-jetpack/build/jetpack/jetpack-sdk-9bfe1631d20c/python-lib/cuddlefish/runner.py", line 291, in run_app    kp_kwargs=popen_kwargs)  File "/Users/cltbld/talos-slave/mozilla-central_snowleopard_test-jetpack/build/jetpack/jetpack-sdk-9bfe1631d20c/python-lib/mozrunner/__init__.py", line 339, in __init__    raise Exception("Binary path does not exist "+self.binary)Exception: Binary path does not exist /Users/cltbld/talos-slave/mozilla-central_snowleopard_test-jetpack/build/Minefield.app/Contents/MacOS/firefox-bin/Contents/MacOS/firefox-binI tested the binary, it starts up. That is the absolute path, and it exists.  Not sure what is going on here - Clint or Brian?  Ideas on what could be the issue?

Comment 27

[Lukas Blakk [:lsblakk] use ?needinfo]

talked to clint in irc and he mentioned that this behaviour is a known bug (bug 583142) -- he believes jetpack needs to update the mozrunner install so I've filed bug 597503 to get this fixed.

Comment 28

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: [tested] jetpack suite enabled in factory.py and unittest.py

Hey Clint - so this runs in staging, I will attach the log and the shell script used to call the jetpack suite in a second.  Can you experiment with the logging?  I had to take out your original logging code because it lead to exceptions. This version just finished with warnings but also provides no useful print info.  Hopefully the logs from the most recent run will help.

Comment 29

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: complete log from run_jetpack.sh step

Here's the stdio output for help in log summary section.

Comment 30

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: [tested] run_jetpack.sh for tools/buildfarm/utils

tested this in staging mozilla-tests1

Comment 31

[cmtalbert]

Comment on attachment 476895 [details] [diff] [review]
[tested] jetpack suite enabled in factory.py and unittest.py

This looks good to me.  I'm not sure what you mean about the log issue though.  I think the s = "(\d+) of (\d+) tests passed" should match the "summary line" of the output.  It might need some .* either prepended or postpended to it.  Let me double check that.  I think all the jetpack folks wanted right now were those three numbers: s.group(1) == tests passing s.group(2) == total tests, s.group(2) - s.group(1) = tests failing.

Comment 32

[cmtalbert]

So I dumped the log into a text file...
>>> f = open("foo.txt", "r")
>>> import re
>>> r = re.compile("(\d+) of (\d+) tests passed")
>>> for l in f:
...   if (r.search(l)):
...     s = r.search(l)
...     print "s.group(1) is " + str(s.group(1))
...     print "s.group(2) is " + str(s.group(2))
... 
s.group(1) is 3
s.group(2) is 3
s.group(1) is 3226
s.group(2) is 3324
>>> 
So, it looks like the regex is correct.  It is matching twice though, which might be causing the problem.  So evidently somewhere in the log there is a statement of the form "3 of 3 tests passed".  Are those 3 tests included in the 3,324?  Can we change the jetpack log format to give us a definitive summary output that will only match our "summary regex" once?  Perhaps something like what mochitest, reftest etc do?  Myk, do you guys have any attachment to that summary output?

Comment 33

[Lukas Blakk [:lsblakk] use ?needinfo]

I think we need

+        if re.search(r"^Traceback", cmd.logs["stdio"].getText(), re.MULTILINE):
+            return WARNINGS

to be in the log summary, so that we can get error output instead of having the buildbot step just error out.

Comment 34

[cmtalbert]

(In reply to comment #33)
> I think we need
> 
> +        if re.search(r"^Traceback", cmd.logs["stdio"].getText(),
> re.MULTILINE):
> +            return WARNINGS
> 
> to be in the log summary, so that we can get error output instead of having the
> buildbot step just error out.
That's a good idea.  I wasn't thinking about all the tracebacks.  I agree.  I also think we should have a definitive summary for passed/failed/total as well.

Comment 35

[Brian Warner [:warner :bwarner]]

Getting a total-number-of-tests-passed/failed is non-trivial, unfortunately, since 'cfx testall' is basically calling three independent test programs in succession, with the exit code being the only piece of structured data coming out. We can work on it, but it probably won't happen too quickly.

Comment 36

[Chris AtLee [:catlee]]

Comment on attachment 476897 [details] [diff] [review]
[tested] run_jetpack.sh for tools/buildfarm/utils

>diff --git a/buildfarm/utils/run_jetpack.sh b/buildfarm/utils/run_jetpack.sh
>new file mode 100755
>--- /dev/null
>+++ b/buildfarm/utils/run_jetpack.sh
>@@ -0,0 +1,24 @@
>+#!/bin/bash
>+set -e
>+JETPACK_TARBALL="jetpack.tar.bz2"
>+BASE_PATH=$(pwd)
>+APP_PATH=$(find $BASE_PATH -maxdepth 1 -type d -name '*.app')

We should error out if APP_PATH is empty

>+if [ ! -d "./jetpack" ]; then
>+  echo "No jetpack directory present!  Cannot run test suite."
>+  exit 1
>+fi;
>+
>+# Set up for running jetpack
>+cd "./jetpack"
>+# Download jetpack-latest tarball
>+wget -i "jetpack-location.txt" -O $JETPACK_TARBALL
>+tar -xvf $JETPACK_TARBALL
>+# Find the sdk dir to run tests in
>+SDK_DIR=$(ls . | grep 'jetpack-sdk*')

Likewise, we should error out if SDK_DIR is empty.

Comment 37

[cmtalbert]

Hey folks,  we've hit a snag doing mozmill + buildbot integration on windows.  We're going to hit the same problem with jetpack because it also uses mozrunner.  The issue is being tracked in bug 600274.

You'll know if you hit the issue because you'll get a traceback that says "Could not start firefox" due to Windows Error 13, access denied.

I'm currently working on a fix, but if anyone has information or experience with python ctypes on windows, help would be appreciated.

Comment 38

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #37)
> I'm currently working on a fix, but if anyone has information or experience
> with python ctypes on windows, help would be appreciated.

Brian or Atul: do either of you have relevant experience to contribute here?

Comment 39

[Brian Warner [:warner :bwarner]]

nope, sorry, my windows-fu is pretty weak.

although.. bug 600274 doesn't look like the right one.. if there's a different bug I should look at, I'll take a look and see if any of my limited linux-python-ctypes experience is applicable.

Comment 40

[cmtalbert]

Doh, These shiny new 600... numbers all look the same to me.  I meant bug 600321.

Thanks for the offer to help guys, but I think we've figured it out.

The issue is that the buildbot twistd on the slaves has been patched (by RelEng) to launch all processes in a windows job.  This is a good idea.  It ensures that buildbot has the power to close everything that it spawns.

However, when creating that job they do not give us the right to break away from that job and create our own jobs.

Unfortunately that's what Mozrunner wants to do.  We want to launch firefox in our own job object and use that for easy monitoring and to have an easy mechanism for closing it down. Windows does not allow jobs to be nested and without that extra creation flag in twistd, we get the "access denied" error when we attempt to create the job.

We are working on a fix to make Mozrunner more resilient and we also filed 600736 to track the necessary changes for our twistd patch.  

So I think we're on track to getting this problem addressed, and we now know more than you'd ever need to know about starting processes on windows.

Comment 41

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: [tested] run_jetpack.sh for tools/buildfarm/utils (now with more erroring out)

added erroring out for $APP_PATH and $SDK_DIR -- also able to pass in the platform in the calling of the test suite and handle the different paths to binary needed by the cfx testall

Comment 42

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: jetpack results on linux/linux64/macosx/macosx64

Attaching the most recent runs of the test suite on linux/linux64/macosx/macosx64 to show where they are at right now.

Comment 43

[Chris AtLee [:catlee]]

Comment on attachment 481380 [details] [diff] [review]
[tested] run_jetpack.sh for tools/buildfarm/utils (now with more erroring out)

>diff --git a/buildfarm/utils/run_jetpack.sh b/buildfarm/utils/run_jetpack.sh
>new file mode 100755
>--- /dev/null
>+++ b/buildfarm/utils/run_jetpack.sh
>@@ -0,0 +1,34 @@
>+#!/bin/bash
>+set -e
>+JETPACK_TARBALL="jetpack.tar.bz2"
>+BASE_PATH=$(pwd)
>+ # handling for each platform's executable path
>+if [ $1 == 'linux' -o $1 == 'linux64' ]; then
>+  APP_PATH=$BASE_PATH/firefox/firefox
>+elif [ $1 == 'macosx' -o $1 == 'macosx64' ]; then
>+  APP_PATH=$(find $BASE_PATH -maxdepth 1 -type d -name '*.app')
>+fi;

Nit: no need for a ';' here, and after other 'fi' statements later.

Also, please add some documentation that the script expects the platform to be passed in as a first argument.  Passing in an invalid platform should generate an error.

>+
>+if [ ! -e $APP_PATH ]; then

Quotes around $APP_PATH would be good, as well as below.

>+  echo "The location $APP_PATH does not exist"
>+  exit 1
>+fi;
>+
>+if [ ! -d "./jetpack" ]; then
>+  echo "No jetpack directory present!  Cannot run test suite."
>+  exit 1
>+fi;
>+
>+# Set up for running jetpack
>+cd "./jetpack"
>+# Download jetpack-latest tarball
>+wget -i "jetpack-location.txt" -O $JETPACK_TARBALL
>+tar -xvf $JETPACK_TARBALL
>+# Find the sdk dir to run tests in
>+SDK_DIR=$(ls . | grep 'jetpack-sdk*')
>+if [ -d $SDK_DIR ]; then
>+  cd $SDK_DIR
>+  # Start virtualenv and run test suites
>+  source bin/activate
>+  cfx testall -a firefox -b $APP_PATH
>+fi;

Need to error out if SDK_DIR can't be found.

Comment 44

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: [tested] run_jetpack.sh for tools/buildfarm/utils (fixed nits, usage, more erroring out)

Comment 45

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 483571 [details] [diff] [review]
[tested] run_jetpack.sh for tools/buildfarm/utils (fixed nits, usage, more erroring out)

landed: http://hg.mozilla.org/build/tools/rev/0d34ed0836bc