1137228 - [mozprocess] mochitest debugger output not getting flushed to stderr

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Andrew Halberstadt [:ahal]]

This is a regression from bug 794984.

(In reply to Julian Seward [:jseward] from comment #40)
> This seems to break "mach mochitest" with the --debugger argument.  At least
> the way I am using it.  At 230777:c49df0e00b59, this worked the same as it
> always did.  At 230778:9c40ffbcf6b3 (this commit), Valgrind output -- that
> is, output from the --debugger= program -- does not appear incrementally
> as it used to.  It does appear when I do control-C, which suggests some
> change
> in buffering or some such (change in streams).
> 
> FWIW, the command I am running is
> 
> DISPLAY=:1.0 G_SLICE=always-malloc MOZ_DISABLE_NONLOCAL_CONNECTIONS=0 /
>   ./mach mochitest-plain
> --debugger=/home/sewardj/VgTRUNK/mozhx/Inst/bin/valgrind /
>   --debugger-args="--fair-sched=yes --smc-check=all-non-file /
>     --suppressions=/home/sewardj/MOZ/SUPPS/mochitest-mc.supp
> --error-limit=no /
>     --trace-children=yes --child-silent-after-fork=yes  /
>    
> --trace-children-skip=/usr/bin/hg,/bin/rm,*/bin/certutil,*/bin/pk12util,*/
> bin/ssltunnel,*/bin/uname,*/bin/which,*/bin/ps,*/bin/grep,*/bin/java /
>     --num-transtab-sectors=24 --tool=memcheck --freelist-vol=500000000 /
>     --redzone-size=256 --gen-suppressions=no
> --px-default=allregs-at-mem-access /
>     --px-file-backed=unwindregs-at-mem-access --stats=yes
> --partial-loads-ok=yes /
>     --show-mismatched-frees=no --read-inline-info=yes
> --fullpath-after=-CURR/ /
>     --num-callers=32 --track-origins=no --show-leak-kinds=definite" /
>   dom/canvas/test/webgl-mochitest/test_hidden_alpha.html

I was unable to reproduce this (valgrind output showed up fine) on linux with the command:
./mach mochitest-plain --debugger=valgrind

Comment 1

[Andrew Halberstadt [:ahal]]

Confirmed that mochitest debugger output goes to stderr. Julian, what platform are you using? What happens if you add:
def write_stderr(line):
    sys.stderr.write(line)
    sys.stderr.flush()
kp_kwargs['processStderrLine'] = [write_stderr]

Here:
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#1761 ?

Comment 2

[(not currently active) Ted Mielczarek]

Attachment: MozReview Request: bz://1137228/ted

/r/4379 - bug 1137228 - use readline instead of iterator for reading mozprocess output

Pull down this commit:

hg pull review -r e88cac0fbd3f98b7fdd59c7b144a6b654b5216d7

Comment 3

[(not currently active) Ted Mielczarek]

I tested this with these simple scripts:
https://gist.github.com/luser/6d827d3e0464fc6fed55

If you grab those and run "./mach python /tmp/test.py" without this patch you won't see output for a while. With this patch you'll see output immediately.

Comment 4

[Andrew Halberstadt [:ahal]]

Comment on attachment 8570079 [details]
MozReview Request: bz://1137228/ted

https://reviewboard.mozilla.org/r/4377/#review3581

Ship It!

Comment 5

[Andrew Halberstadt [:ahal]]

Also without this patch running ./mach build buffers ~4k bytes of data or so at a time. With it, ./mach build streams output as it happens.

Comment 6

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/93707c1b4edb

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/93707c1b4edb

Comment 8

[Julian Seward [:jseward]]

That seems to fix it for me; thanks both!

Comment 10

[(not currently active) Ted Mielczarek]

Attachment: MozReview Request: bug 1137228 - use readline instead of iterator for reading mozprocess output