Write mozharness LocalLogger, BaseHelper classes for in-script parallelization

RESOLVED WONTFIX

Status

Release Engineering
Mozharness
P5
normal
RESOLVED WONTFIX
6 years ago
22 days ago

People

(Reporter: aki, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [mozharness])

(Reporter)

Description

6 years ago
While writing sign_android.py, I noted that various tasks, like downloading+uploading apks, and signing apks, were serial operations that could be sped up by parallelizing.

We already have a parallelization solution: chunking across multiple machines.  I'd like to add chunking support to mozharness; this bug addresses the issue more like a -j4 compile flag.

LocalLogger will eschew the python logging module, since afaict any second logging instance will interfere with the main logging instance.  Instead, it will either buffer timestamps+loglines as a variable in memory, or in a tmpfile.

* possibly rename BaseLogger to BaseFileLogger
* possibly write an AbstractBaseLogger
* write a LocalLogger class that doesn't inherit the current BaseLogger, that defines a log_message() that buffers the timestamps+loglines as a self.variable or in a tmpfile.

BaseHelper can be used for a helper signing object, a helper transfer object, a helper VCS object, or whatever we want to do in parallel.  We can set its self.log_obj to the main log_obj and have it behave like it's part of the main serial flow.  However, if we set a flag, it should create a LocalLogger self.log_obj.  This will then be able to run some specified command(s) in parallel with other BaseHelper-derived objects, without intermingling the output in the main logs.  Once the command(s) finish, we can retrieve the timestamps+log output, the status/return code(s), and log to the main log appropriately, e.g.

12:00:00 INFO - Cloning 90 locale repos (parallelizing x10)
12:01:30 INFO - Clone of ar finished successfully:
12:01:30 INFO -  (12:00:01) Running command hg clone http://...
12:01:30 INFO -  (12:00:01)  destination directory: ar
12:01:30 INFO -  (12:00:01)  requesting all changes
...

We poll the 10 helper objects and as each one finishes, we log to the main log and then kick off the next one queued up.
(Reporter)

Updated

6 years ago
Summary: Write LocalLogger, BaseHelper classes for in-script parallelization → Write mozharness LocalLogger, BaseHelper classes for in-script parallelization
(Reporter)

Comment 1

6 years ago
(In reply to Aki Sasaki [:aki] from comment #0)
> We already have a parallelization solution: chunking across multiple
> machines.  I'd like to add chunking support to mozharness; this bug
> addresses the issue more like a -j4 compile flag.

Read: chunking-across-machines-in-mozharness is wanted, but should be a separate bug.
(Reporter)

Updated

6 years ago
Blocks: 682129
(Reporter)

Updated

5 years ago
Assignee: nobody → aki
(Reporter)

Updated

5 years ago
Assignee: aki → nobody
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: release → catlee
(Reporter)

Updated

5 years ago
Blocks: 772959
(Assignee)

Updated

4 years ago
Product: mozilla.org → Release Engineering

Updated

3 years ago
Component: General Automation → Mozharness
(Reporter)

Updated

22 days ago
Status: NEW → RESOLVED
Last Resolved: 22 days ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.