650065 - I/O on windows slaves gets messed up sometimes

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P5)

Description

[[e-mail] (:bhearsum)]

Particularly with stdout, and maybe stderr (not sure about that one though).

A good example of this is Clobberer during l10n release repacks. In this scenario, Buildbot calls bash to run a script, which then calls python to run clobberer.py. In clobberer.py, it loops over the directories needing a clobberer, printing a single line to stdout on each iteration. On Linux and Mac, we see this output as it goes, on Windows, we don't. I don't know the exact cause of this, but the root problem is definitely below the Buildbot level (because Buildbot sends any log data it has every N seconds or X bytes). This means that the buildslave process, which launches things with win32process.CreateProcess has not received any output from the process.

I tried to create a minimal test case by doing the following:
test.py:
import sys, time
print "abc"
time.sleep(3)

run.sh:
#!/bin/bash
d:/mozilla-build/python25/python test.py

runtest.py:
import win32process, time
win32process.CreateProcess(d:/mozilla-build/msys/bin/bash.exe", "d:/mozilla-build/msys/bin/bash.exe run.sh", None, None, 1, 0, {}, ".", win32process.STARTUPINFO())
time.sleep(5)

Unfortunately, I couldn't reproduce the process this way.


Also of note may be the way processes get launched by Buildbot. Here's the way, all the way from Windows startup, that clobberer gets launched for release repacks
- buildbot.bat launched by Startup folder
- start-buildbot.bat launched by buildbot.bat
- start-buildbot.sh launched by start-buildbot.bat
- buildbot runner script (python) launched by start-buildbot.sh
- release_repacks.sh launched by buildbot (through twisted, through win32process.CreateProcess)
- clobberer.py is launched by release_repacks.sh

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Try a test case that uses Twisted?

Sadly, we never got the GSoC application to fix Windows startup that a potential student was floating around.

There's a Buildbot bug where output from one step ended up in another step.  I disbelieved it, but who knows - maybe one of the horrible hacks that comprise msys or cygwin was responsible.