1179819 - runner tasks output on OSX goes to /dev/null

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Chris AtLee [:catlee]]

The output from tasks running under runner on OSX doesn't get logged anywhere - not in /var/log/system.log, nor in papertrail.

Comment 1

[Vlad Ciobancai [:vladC] [:vciobancai]]

I checked runner application from git and I found that for Darwin operation system is used /var/run/syslog for output (more details : https://github.com/mozilla/build-runner/blob/master/runner/__init__.py#L188 )

The only message that I see in syslog log is "com.apple.xpc.launchd[1] (com.mozilla.runner): This key does not do anything: OnDemand"

Comment 2

[Vlad Ciobancai [:vladC] [:vciobancai]]

I made some research and I found that OnDemand is deprecated since 10.5 and needs to be removed from the init script used for Darwin servers. I will create a patch and I will test

Comment 3

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1179819_runner.plist.erb.patch

Attached you can find the patch to remove the "OnDemand" key from the service script because is deprecated. 

I tried to start the script by using the following command : "launchctl start /Library/LaunchAgents/com.mozilla.runner.plist" but I was not able 

Kim can you please help me to test the change ?

Comment 4

[Vlad Ciobancai [:vladC] [:vciobancai]]

After some research I was able to start the application by running the following commands:

- launchctl load /Library/LaunchAgents/com.mozilla.runner.plist
- launchctl start com.mozilla.runner

After the application started the output from runner is logged in /var/log/system.log . Before removing the "OnDemand" line in the com.mozilla.runner.plist, the messages from system.log was like in comment #1.

Also the output can be view on papertrail for t-yosemite-r7-0021 (the slave have been loaned to me)

Comment 5

[Kim Moir [:kmoir] ET]

merged to production

Comment 6

[Vlad Ciobancai [:vladC] [:vciobancai]]

I checked randomly some OS X servers and the logs from runner appears in papertrail.

Comment 7

[Chris Cooper [:coop] (he/him)]

catlee mentioned today that while the command is now being logged on OS X, the output of the command is not. Can we verify that the output is being logged too, please?

Comment 8

[Kim Moir [:kmoir] ET]

Andrei: what machine are you testing this command and as what user?  If you used a loaner I think runner is disabled.

Comment 9

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

I did not yet begun any test but I checked the server used by Vlad, t-yosemite-r7-0021 and I saw that he used these commands :
- launchctl load /Library/LaunchAgents/com.mozilla.runner.plist
- launchctl start com.mozilla.runner

This slave however does not have LaunchAgents folder under root or cltbld accounts to start the runner.

Comment 10

[Kim Moir [:kmoir] ET]

Maybe this machine is in a weird state since Vlad had it for a long time?  Perhaps reimage it so all the runner bits are in the places that you expect.

Comment 11

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Made some tests on t-yosemite-r7-0021,after runnig ( launchctl load /Library/LaunchAgents/com.mozilla.runner.plist) we will have this job(/opt/runner/bin/python2.7 /opt/runner/bin/runner -v --syslog -H -n 15 -c /opt/runner/runner.cfg /opt/runner/tasks.d) which will send messages to system.log that will appear in Papertrail.
Also com.mozilla.runner.plist is configured to output the logs in runner.out and runner.err which are located in /var/tmp.

If the machine will be disable the logs from runner.err will be also present on Papertrail but the logs from runner.out are never showed even when the machine is enabled.

I suppose that the runner job should  also send the logs from runner.out to system.log,but as this action never happens and the scripts doesn't indicate any restrictions, the solution is to redirect the logs from runner.com to system.log? Or is there any other way?
I think that the logs that Chris is looking for are in runner.out

Comment 12

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: runner tasks.PNG

After making some investigation and comparing the logs from Papertrail on some talos-linux64-ix machines and the logs from some t-yosemite-r7 machines the conclusion is that similar logs are shown in Papertrail.

After runner is enabled the following common tasks will be run : 
['0-checkout_tools', '1-cleanslate', '1-cleanup', '1-mig_agent', '1-purge_builds', '3-update_shared_repos', '4-buildbot.py', '99-post_flight', '99-restart_services']

These tasks have the same logs on Papertrail for CentOS,Ubuntu or Darwin.
So there is no need for redirecting the logs from runner.out to system.log.

I have also attached a screenshot from Papertrail for a talos-linux64 machine and a yosemite-r7 machine with the logs from these tasks. 

If more informations will need to be displayed on Papertrail for runner's tasks then the configuration will need to be done for all machines not just OS X.

Comment 13

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: runner tasks talos.PNG

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Andrei Obreja [:aobreja][:buildduty] from comment #11) 
> I suppose that the runner job should  also send the logs from runner.out to
> system.log,but as this action never happens and the scripts doesn't indicate
> any restrictions, the solution is to redirect the logs from runner.com to
> system.log? Or is there any other way?
> I think that the logs that Chris is looking for are in runner.out

Your papertrail screenshot indicates to me that this is working you've found a valid solution here. 

Verify that there's no missing data between the two logs (runner.out & syslog) for the events, and then proceed.

Comment 15

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

These logs from Papertrail were shown without runner.out been redirected to system.log,this is the current situation without any change.
Also for OS X the tasks that are running under runner (0-checkout_tools , 1-cleanslate,etc) are not logged in system.log ,but are present  in Papertrail(not sure yet how Papertrail got these messages).

Also runner.out have different logs from papaertrail or system.log but after checking them I think that these logs are not needed in Papertrail,here(runner.out) we will only find this sequence in a loop:


0-darwin_clean_buildbot: finished buildslave cleanup Fri Jul 29 04:15:48 PDT 2016
216.15 GB of space available
Updating hg repos in /builds/hg-shared
pulling from https://hg.mozilla.org/build/tools
searching for changes
no changes found
pulling from https://hg.mozilla.org/build/mozharness
searching for changes
no changes found
skipping /builds/slave/twistd.pid
skipping /builds/slave/talos-slave/twistd.pid
Starting buildbot via runslave.py.
Buildbot started successfully.
Buildbot finished.


So in conclusion I think there is no need for any action,the commands and the output of the tasks under runner are already in Papertrail for OS X.

Comment 16

[Chris Cooper [:coop] (he/him)]

I see output in papertrail for bld-lion-r5, t-yosemite-r5, and t-yosemite-r7 machines. Thanks, Andrei!