794627 - Run mochitests in Valgrind tbpl builds on test slaves

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P3)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

We should be striving to run more tests in Valgrind tbpl builds. This bug is about running mochitests.

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

ted mentions that we could probably just use makefile targets, e.g. `make xpcshell-tests`.

Comment 2

[[e-mail] (:bhearsum)]

Assuming you're OK running on the build machines. All of the other unit tests run on the test pool, though, and don't have an objdir.

Comment 3

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Ben Hearsum [:bhearsum] from comment #2)
> Assuming you're OK running on the build machines. All of the other unit
> tests run on the test pool, though, and don't have an objdir.

On second thoughts, I'm not sure if running on the build machines is the most ideal situation. (it might turn up oranges specific to build machines) I think we would want to share as much infrastructure as possible.

Comment 4

[Julian Seward [:jseward]]

Attachment: Adjust Mochitests timeouts more suitable for Valgrind runs

I've run Mochitests on Valgrind quite a lot.  Timeouts due to the
added slowness are a real problem, so I use this patch to allow extra
time.  Note that the numbers are not exactly scientifically derived.

Also, be sure to use x86_64-linux for this.  The performance of
Valgrind running Firefox on MacOS is very poor compared to running on
Linux.

Memory use can be a problem.  Running all the tests in a single
process takes 10+ CPU hours on a fast machine, and 5+ GB memory, and
that's if nothing goes wrong.  Better to divide the run into about 20
pieces; that also makes it parallelisable.  Here's the command I use,
more or less.  "v38BRANCH" is the 3.8.x branch.  I strongly recommend
you use 3.8.1 for this.

Here's the command I use to start it.  It's been evolved over many
runs, so might be a useful starting point.  In its raw form:


for ((qq = 1; qq <= 100; qq++)); do (DISPLAY=:3.0 make -C ff-opt mochitest-plain EXTRA_TEST_ARGS="--this-chunk=$qq --total-chunks=100 --close-when-done --debugger=/home/sewardj/VgTRUNK/hgde3/Inst/bin/valgrind --debugger-args='--smc-check=all --suppressions=/home/sewardj/MOZ/mochitest-mc.supp --suppressions=/home/sewardj/MOZ/SUPPS/mc-helgrind-not-our-problem.supp --error-limit=no --stats=no --trace-children=yes --child-silent-after-fork=yes "--trace-children-skip=/usr/bin/hg,/bin/rm,*/bin/certutil,*/bin/pk12util,*/bin/ssltunnel,*/bin/uname,*/bin/which,*/bin/ps,*/bin/grep" "--trace-children-skip-by-arg=*/npviewer*" --fullpath-after=-HG/ --tool=helgrind --history-level=none --profile-heap=no --read-var-info=no --num-callers=20 --free-is-write=yes'") 2>&1 | tee spew-hg-none--chunk-$qq-of-100 ; done


Formatted nicely, that is (I don't know if this line breaking is 
really valid, but anyway:)


for ((qq = 1; qq <= 20; qq++)); do \
  (DISPLAY=:3.0 \
     make -C ff-opt mochitest-plain \
     EXTRA_TEST_ARGS="--this-chunk=$qq --total-chunks=20 --close-when-done \
                     --debugger=v38BRANCH \
                     --debugger-args='--fair-sched=yes --smc-check=all-non-file \
                                      --suppressions=/home/sewardj/MOZ/SUPPS/mochitest-mc.supp \
                                      --error-limit=no --stats=no --trace-children=yes \
                                      --child-silent-after-fork=yes \
                                      "--trace-children-skip=/usr/bin/hg,/bin/rm,*/bin/certutil,*/bin/pk12util,*/bin/ssltunnel,*/bin/uname,*/bin/which,*/bin/ps,*/bin/grep" \
                                      --tool=memcheck --track-origins=no'") 
     2>&1 \
      | tee spew-mc-1--chunk-$qq-of-20 ; \
done


Which isn't pretty, but basically runs V on large process trees, but
uses --trace-children-skip to avoid following into a whole bunch of
processes (hg, rm, certutil, etc) which we're not interested in.


Maybe should be turned into a proper shell script and added to the
tree?


It traces also into Python and generates a lot of noise -- Python
confuses Memcheck.  Hence I find it essential to use the listed supp
file, /home/sewardj/MOZ/SUPPS/mochitest-mc.supp, to get sane
behaviour.  I'll attach that in the next comment.

I run it on a 16 bit deep, 800x600 VNC server, hence the
"(DISPLAY=:3.0" above.  It runs with no window manager, with input
disabled and the mouse pointer "frozen" pretty much in the middle, so
that Fx has mouse focus most of the time.  It still has problems
losing focus sometimes, though.  I don't know why.

Comment 5

[Julian Seward [:jseward]]

Attachment: /home/sewardj/MOZ/SUPPS/mochitest-mc.supp as mentioned in previous comment

These are baseline suppressions needed for runs on x86_64-linux,
Ubuntu 10.04.4.  You will need them or something very similar to get
sane behaviour.  They are partially for Firefox itself, but partially
for the Python process(es) that also wound up getting run on Valgrind
as part of the game.

Comment 6

[Julian Seward [:jseward]]

(In reply to Julian Seward from comment #4)
> Created attachment 667390 [details] [diff] [review]
> Adjust Mochitests timeouts more suitable for Valgrind runs

One further comment.  Probably obvious, but .. once this patch
is applied, you still need to do "make" at the top level to
get it to take effect, before starting the Mochitest run.