783815 - Tegra Verification steps should not buffer stdout/stderror

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Justin Wood (:Callek)]

With buildbot jobs, we have verify.py and reboot.py run for every job.

This can frequently fail or timeout with little to no output printed to terminal. If we set the environment var |PYTHONUNBUFFERED| for at least these steps we should be able to avoid that problem.

The reason it happens, is these steps, on a failure, set an "error flag" file, which clientproxy on the foopy watches for and hard-kills buildbot when it is present. Buildbot however runs these steps without a tty, and when python is run without a tty it, by default, buffers stdout. So when buildbot dies we don't properly flush the stdout/stderr to the log files that we use, thus the tree sheriffs don't see things.

This should be a simple fix, and (in theory) could be used for all steps without regressing ANY [or negligable] disk i/o.

Self contained and not an emergency priority, but good gains to tree-sheriffs in terms of diagnosing issues, that coop said we might consider this as something for kim to get her feet wet with.

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: patch

Comment 2

[Justin Wood (:Callek)]

Comment on attachment 667168 [details] [diff] [review]
patch

yay!

Comment 3

[Ed Morley [:emorley]]

:-D

Comment 4

[Chris AtLee [:catlee]]

deployed in production