627847 - add multithreading support for linux-mips acceptance test runs

Status: VERIFIED FIXED

(Tamarin Graveyard :: Tools, defect)

Description

[Dan Schaffer]

We should add support in acceptance/runtests.py for multithreading in linux-mips and platforms using run-acceptance-generic-ssh.sh.  

Here are the steps:
- in environment.sh set: threads=n and set SSH_SHELL_REMOTE_HOST, SSH_SHELL_REMOTE_USER, SSH_SHELL_REMOTE_DIR each with followed by number 0...n-1
- add runtests.py --passthreads (passes --threadid=n with the thread id to the AVM)
- in ssh-shell-runner.sh load SSH_SHELL_REMOTE_HOST,USER,DIR matching the threadid
- in ssh-shell-deployer.sh deploy the shell to each SSH_SHELL_REMOTE_HOST,USER,DIR

When have new machines hardware we want to add just have to update environment.sh to add threads=n and set SSH_SHELL_REMOTE_HOST,DIR,USER with the new index.

Tests have reduced the release run in linux-mips from threads=1 on 1 host from 1h 20m to threads=4 (2 dual core machines) to 18m.  Also debug run in linux-mips goes from threads=1 x 1 host from 4h 10m to threads=4, 2 machines x 2 cores to 50m.

Comment 1

[Dan Schaffer]

Attachment: mips-threads support patch

Comment 2

[Dan Schaffer]

in addtion to the patch I update the file build/buildbot/slaves/linux-mips/scripts/environment.sh for the new thread # format.

Comment 3

[Dan Schaffer]

Attachment: mips threads support patch (take 2)

small fixes to this patch.  this one passed sandbox builds.

Comment 4

[Dan Schaffer]

based on feedback from Brent I renamed the runtests.py --passthreads to --passthreadid.  I tried to pass $threadid as an environment variable but could not get it working reliably.  I had to use os.environ (all existing environment variables) and add "threadid" into the dictionary and passing to new dictionary.  There were intermittent failures where the subprocess unexpectedly stopped.

I'd like to keep the existing patch with renaming passthreads to passthreadid

Comment 5

[Dan Schaffer]

the problem was os.environ was being shared by each thread.  the fix is to do:
newenv=copy.deepcopy(os.environ) 
newenv['threads']=threadid

so each thread has a unique copy of the environment.  I will post a new patch.

Comment 6

[Dan Schaffer]

Attachment: mips threads support patch (v3)

Comment 7

[Dan Schaffer]

Comment on attachment 506804 [details] [diff] [review]
mips threads support patch (v3)

in the above patch the master/tamarin-redux.py change to remove threads=1 will work without restarting the master since it will produce threads twice in runtests.   e.g. runtests.py threads=1 --passthreadid --threads=4
the change is to clarify.

Comment 8

[Tamarin Bot]

changeset: 5837:088a87d38be7
user:      Dan Schaffer <[e-mail]>
summary:   bug 627847: allow test platforms with ssh based test runs (like linux-mips) to use python threads across multiple devices (r=brbaker)

The mechanism gives us the flexibilty to add and remove test hardware slaves
without reconfiguring buildbot.  On slow hardware platforms we can drive the
tests from a desktop machine and each test thread is assigned a test machine.
In environment.sh the threads environment variable sets the number of threads
and for threads=n i=1..n-1 SSH_SHELL_REMOTE_{HOST,USER,DIR}i refers to a test
machine.  The speedup on linux-mips is from 1h20m to 18m for release using
2 dual core linux mips boxes x 2 threads per machine.  For debug the speedup
is from 4h to 50m.  We can investigate using for other slow platforms.

http://hg.mozilla.org/tamarin-redux/rev/088a87d38be7

Comment 9

[Dan Schaffer]

Attachment: environment updates for new mips hardware

Comment 10

[Tamarin Bot]

changeset: 5916:f27a553cd2c8
user:      Dan Schaffer <[e-mail]>
summary:   bug 627847: update linux-mips configurations to utilize new hardware (r=brbaker)

http://hg.mozilla.org/tamarin-redux/rev/f27a553cd2c8