add multithreading support for linux-mips acceptance test runs

VERIFIED FIXED

Status

Tamarin
Tools
VERIFIED FIXED
7 years ago
7 years ago

People

(Reporter: Dan Schaffer, Assigned: Dan Schaffer)

Tracking

Details

Attachments

(2 attachments, 2 obsolete attachments)

(Assignee)

Description

7 years ago
We should add support in acceptance/runtests.py for multithreading in linux-mips and platforms using run-acceptance-generic-ssh.sh.  

Here are the steps:
- in environment.sh set: threads=n and set SSH_SHELL_REMOTE_HOST, SSH_SHELL_REMOTE_USER, SSH_SHELL_REMOTE_DIR each with followed by number 0...n-1
- add runtests.py --passthreads (passes --threadid=n with the thread id to the AVM)
- in ssh-shell-runner.sh load SSH_SHELL_REMOTE_HOST,USER,DIR matching the threadid
- in ssh-shell-deployer.sh deploy the shell to each SSH_SHELL_REMOTE_HOST,USER,DIR

When have new machines hardware we want to add just have to update environment.sh to add threads=n and set SSH_SHELL_REMOTE_HOST,DIR,USER with the new index.

Tests have reduced the release run in linux-mips from threads=1 on 1 host from 1h 20m to threads=4 (2 dual core machines) to 18m.  Also debug run in linux-mips goes from threads=1 x 1 host from 4h 10m to threads=4, 2 machines x 2 cores to 50m.
(Assignee)

Comment 1

7 years ago
Created attachment 505923 [details] [diff] [review]
mips-threads support patch
Assignee: nobody → dschaffe
Status: NEW → ASSIGNED
Attachment #505923 - Flags: review?(cpeyer)
Attachment #505923 - Flags: feedback?(brbaker)
(Assignee)

Comment 2

7 years ago
in addtion to the patch I update the file build/buildbot/slaves/linux-mips/scripts/environment.sh for the new thread # format.
(Assignee)

Comment 3

7 years ago
Created attachment 505957 [details] [diff] [review]
mips threads support patch (take 2)

small fixes to this patch.  this one passed sandbox builds.
Attachment #505923 - Attachment is obsolete: true
Attachment #505957 - Flags: review?(cpeyer)
Attachment #505957 - Flags: feedback?(brbaker)
Attachment #505923 - Flags: review?(cpeyer)
Attachment #505923 - Flags: feedback?(brbaker)
(Assignee)

Updated

7 years ago
Attachment #505957 - Flags: review?(cpeyer)
Attachment #505957 - Flags: review?(brbaker)
Attachment #505957 - Flags: feedback?(cpeyer)
Attachment #505957 - Flags: feedback?(brbaker)
(Assignee)

Comment 4

7 years ago
based on feedback from Brent I renamed the runtests.py --passthreads to --passthreadid.  I tried to pass $threadid as an environment variable but could not get it working reliably.  I had to use os.environ (all existing environment variables) and add "threadid" into the dictionary and passing to new dictionary.  There were intermittent failures where the subprocess unexpectedly stopped.

I'd like to keep the existing patch with renaming passthreads to passthreadid
(Assignee)

Comment 5

7 years ago
the problem was os.environ was being shared by each thread.  the fix is to do:
newenv=copy.deepcopy(os.environ) 
newenv['threads']=threadid

so each thread has a unique copy of the environment.  I will post a new patch.
(Assignee)

Comment 6

7 years ago
Created attachment 506804 [details] [diff] [review]
mips threads support patch (v3)
Attachment #506804 - Flags: review?(brbaker)
(Assignee)

Updated

7 years ago
Attachment #505957 - Attachment is obsolete: true
Attachment #505957 - Flags: review?(brbaker)
Attachment #505957 - Flags: feedback?(cpeyer)
(Assignee)

Comment 7

7 years ago
Comment on attachment 506804 [details] [diff] [review]
mips threads support patch (v3)

in the above patch the master/tamarin-redux.py change to remove threads=1 will work without restarting the master since it will produce threads twice in runtests.   e.g. runtests.py threads=1 --passthreadid --threads=4
the change is to clarify.

Updated

7 years ago
Attachment #506804 - Flags: review?(brbaker) → review+

Comment 8

7 years ago
changeset: 5837:088a87d38be7
user:      Dan Schaffer <dschaffe@adobe.com>
summary:   bug 627847: allow test platforms with ssh based test runs (like linux-mips) to use python threads across multiple devices (r=brbaker)

The mechanism gives us the flexibilty to add and remove test hardware slaves
without reconfiguring buildbot.  On slow hardware platforms we can drive the
tests from a desktop machine and each test thread is assigned a test machine.
In environment.sh the threads environment variable sets the number of threads
and for threads=n i=1..n-1 SSH_SHELL_REMOTE_{HOST,USER,DIR}i refers to a test
machine.  The speedup on linux-mips is from 1h20m to 18m for release using
2 dual core linux mips boxes x 2 threads per machine.  For debug the speedup
is from 4h to 50m.  We can investigate using for other slow platforms.

http://hg.mozilla.org/tamarin-redux/rev/088a87d38be7
(Assignee)

Updated

7 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Assignee)

Updated

7 years ago
Status: RESOLVED → VERIFIED
(Assignee)

Comment 9

7 years ago
Created attachment 511432 [details] [diff] [review]
environment updates for new mips hardware
Attachment #511432 - Flags: review?(brbaker)

Updated

7 years ago
Attachment #511432 - Flags: review?(brbaker) → review+

Comment 10

7 years ago
changeset: 5916:f27a553cd2c8
user:      Dan Schaffer <dschaffe@adobe.com>
summary:   bug 627847: update linux-mips configurations to utilize new hardware (r=brbaker)

http://hg.mozilla.org/tamarin-redux/rev/f27a553cd2c8
You need to log in before you can comment on or make changes to this bug.