706887 - Write mozharness LocalLogger, BaseHelper classes for in-script parallelization

Status: RESOLVED WONTFIX

(Release Engineering :: Applications: MozharnessCore, defect, P5)

Description

[Aki Sasaki (not active)]

While writing sign_android.py, I noted that various tasks, like downloading+uploading apks, and signing apks, were serial operations that could be sped up by parallelizing.

We already have a parallelization solution: chunking across multiple machines.  I'd like to add chunking support to mozharness; this bug addresses the issue more like a -j4 compile flag.

LocalLogger will eschew the python logging module, since afaict any second logging instance will interfere with the main logging instance.  Instead, it will either buffer timestamps+loglines as a variable in memory, or in a tmpfile.

* possibly rename BaseLogger to BaseFileLogger
* possibly write an AbstractBaseLogger
* write a LocalLogger class that doesn't inherit the current BaseLogger, that defines a log_message() that buffers the timestamps+loglines as a self.variable or in a tmpfile.

BaseHelper can be used for a helper signing object, a helper transfer object, a helper VCS object, or whatever we want to do in parallel.  We can set its self.log_obj to the main log_obj and have it behave like it's part of the main serial flow.  However, if we set a flag, it should create a LocalLogger self.log_obj.  This will then be able to run some specified command(s) in parallel with other BaseHelper-derived objects, without intermingling the output in the main logs.  Once the command(s) finish, we can retrieve the timestamps+log output, the status/return code(s), and log to the main log appropriately, e.g.

12:00:00 INFO - Cloning 90 locale repos (parallelizing x10)
12:01:30 INFO - Clone of ar finished successfully:
12:01:30 INFO -  (12:00:01) Running command hg clone http://...
12:01:30 INFO -  (12:00:01)  destination directory: ar
12:01:30 INFO -  (12:00:01)  requesting all changes
...

We poll the 10 helper objects and as each one finishes, we log to the main log and then kick off the next one queued up.

Comment 1

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #0)
> We already have a parallelization solution: chunking across multiple
> machines.  I'd like to add chunking support to mozharness; this bug
> addresses the issue more like a -j4 compile flag.

Read: chunking-across-machines-in-mozharness is wanted, but should be a separate bug.