Last Comment Bug 706887 - Write mozharness LocalLogger, BaseHelper classes for in-script parallelization
: Write mozharness LocalLogger, BaseHelper classes for in-script parallelization
Status: NEW
[mozharness]
:
Product: Release Engineering
Classification: Other
Component: Mozharness (show other bugs)
: other
: All All
P5 normal (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
: Chris AtLee [:catlee]
:
Mentors:
Depends on:
Blocks: 682129 772959
  Show dependency treegraph
 
Reported: 2011-12-01 10:40 PST by Aki Sasaki [:aki]
Modified: 2014-07-09 10:10 PDT (History)
1 user (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description User image Aki Sasaki [:aki] 2011-12-01 10:40:09 PST
While writing sign_android.py, I noted that various tasks, like downloading+uploading apks, and signing apks, were serial operations that could be sped up by parallelizing.

We already have a parallelization solution: chunking across multiple machines.  I'd like to add chunking support to mozharness; this bug addresses the issue more like a -j4 compile flag.

LocalLogger will eschew the python logging module, since afaict any second logging instance will interfere with the main logging instance.  Instead, it will either buffer timestamps+loglines as a variable in memory, or in a tmpfile.

* possibly rename BaseLogger to BaseFileLogger
* possibly write an AbstractBaseLogger
* write a LocalLogger class that doesn't inherit the current BaseLogger, that defines a log_message() that buffers the timestamps+loglines as a self.variable or in a tmpfile.

BaseHelper can be used for a helper signing object, a helper transfer object, a helper VCS object, or whatever we want to do in parallel.  We can set its self.log_obj to the main log_obj and have it behave like it's part of the main serial flow.  However, if we set a flag, it should create a LocalLogger self.log_obj.  This will then be able to run some specified command(s) in parallel with other BaseHelper-derived objects, without intermingling the output in the main logs.  Once the command(s) finish, we can retrieve the timestamps+log output, the status/return code(s), and log to the main log appropriately, e.g.

12:00:00 INFO - Cloning 90 locale repos (parallelizing x10)
12:01:30 INFO - Clone of ar finished successfully:
12:01:30 INFO -  (12:00:01) Running command hg clone http://...
12:01:30 INFO -  (12:00:01)  destination directory: ar
12:01:30 INFO -  (12:00:01)  requesting all changes
...

We poll the 10 helper objects and as each one finishes, we log to the main log and then kick off the next one queued up.
Comment 1 User image Aki Sasaki [:aki] 2011-12-01 10:41:50 PST
(In reply to Aki Sasaki [:aki] from comment #0)
> We already have a parallelization solution: chunking across multiple
> machines.  I'd like to add chunking support to mozharness; this bug
> addresses the issue more like a -j4 compile flag.

Read: chunking-across-machines-in-mozharness is wanted, but should be a separate bug.

Note You need to log in before you can comment on or make changes to this bug.