Closed Bug 1366731 Opened 3 years ago Closed 3 years ago

Autophone - move worker subprocess logging into the subprocess

Categories

(Testing :: Autophone, enhancement)

enhancement
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bc, Assigned: bc)

References

Details

Attachments

(1 file)

Bug 1304063 introduced the logging server and attempted to fix issues with the logging between the multiple processes by logging everything from the main process. This worked to regain the ability for the main process log to contain all log messages from all of the workers but it failed to fix the Ubuntu problem with losing test failure messages and introduced other issues such as:

Logs were sometimes truncated due to unexpected reopening of logs due to opening logs with mode='w'.

The synchronization of the logging server and the worker processes using Threading.Events was overly complicated and difficult to prove correct.

Rollover of the main log at midnight sometimes resulted in truncation of the rotated log.

I realized I had been approaching this all wrong and that with the new ability for the subprocess logs to be passed to the main process log, we could move the worker subprocess logging into the subprocess totally and remove any need for synchronization between the main process and the worker sub processes.
I've been running multiple try runs testing this. The latest is https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=52b207e50972b74546b89775b6dd7c182ff6ff6a&filter-searchStr=autophone

Some of the earlier tests suffered from the log url length issue. You can see this when the logview doesn't think it has a log but the link to the log in the job details panel works. This was fixed in later tests.

I also fixed some log bleed over from earlier tests or other autophone logging not directly related to the test by truncating the log just prior to the test setup step.

This *might* fix the Ubuntu issue, but I really don't hold out much hope but you never know. It does fix a number of other outstanding issues which are critical to sheriffing.

It does appear that the log steps may need additional attention in the event that the build fails due to too many attempts to test a build. I ran into this since I was stopping the workers to fix newly identified issues then restarting.
Attachment #8869993 - Flags: review?(jmaher)
Comment on attachment 8869993 [details] [diff] [review]
bug-1366731-v1.patch

Review of attachment 8869993 [details] [diff] [review]:
-----------------------------------------------------------------

thanks for all the comments
Attachment #8869993 - Flags: review?(jmaher) → review+
https://github.com/mozilla/autophone/commit/fd909fffd07941c067d9707fe8de849110d8ea2b
deploying shortly
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.