bugzilla.mozilla.org will be intermittently unavailable on Saturday, March 24th, from 16:00 until 20:00 UTC.

Mozharness scripts can lose output



Release Engineering
6 years ago
3 years ago


(Reporter: jgriffin, Unassigned)


Firefox Tracking Flags

(Not tracked)


(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2582] [mozharness])



6 years ago
It appears that when mozharness scripts hang and get killed by buildbot, we can lose output in the log, probably due to stdout buffering.

This appears to be happening with the mozharness Marionette scripts on win32 debug.  The logs all end at random points, making it hard to see what's going on.  See e.g., https://tbpl.mozilla.org/php/getParsedLog.php?id=15230913&tree=Try&full=1

However, running the script locally (sans buildbot) you can see that the tests end but Firefox isn't getting shut down correctly by the Marionette test runner.

I wonder if http://hg.mozilla.org/build/mozharness/file/87c0d9079e81/mozharness/base/log.py#l50 should be changed to write all log statements to stderr, instead of some going to stdout, to avoid stdout buffering?

Comment 1

6 years ago
I've tried various solutions a la http://stackoverflow.com/questions/107705/python-output-buffering, though I've always come out of it feeling like maybe the buffering wasn't a problem to begin with.  I think twistd also caches until it has a certain amount of output as well.

stderr is certainly a solution I haven't thought of, but I tend to dislike other scripts that write non-errors to stderr, so my preference would be a different solution.  If it's the best of a bad lot, we may need to go with it, though.

Comment 2

6 years ago
One solution is to make sure that the marionette/mozharness timeout is lower than the buildbot timeout.  If we notice the hung test and try to kill it and start outputting info about that, those buffers should clear.


6 years ago
Priority: -- → P3
Whiteboard: [mozharness]

Comment 3

6 years ago
Catlee thinks this should solve the buffering:
This is noticeable in logs such as: https://tbpl.mozilla.org/php/getParsedLog.php?id=19055414&tree=Firefox&full=1

Where we're running "make upload" and getting no output from it at all, because buildbot times it out before it completes.

Comment 5

5 years ago
Seems like we can call ScriptFactory with an interpreter of [python, '-u']


5 years ago
Product: mozilla.org → Release Engineering


4 years ago
Component: General Automation → Mozharness


3 years ago
Whiteboard: [mozharness] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2582] [mozharness]
You need to log in before you can comment on or make changes to this bug.